In their recent comments on Paul Collier’s book The Bottom Billion Paul Hoebink and Erwin Bulte have questioned the use of cross-country regression analysis in the study of economic growth. Their discussion may leave many outsiders confused. While most economists (ourselves included) have reservations about cross-country regressions, some of the points raised by Hoebink and Bulte are not well taken.

First, what Churchill said about democracy also applies here: cross-country regressions are quite imperfect, until you consider the alternatives. How would you investigate, say, the effect of trade policy and institutional quality on economic growth, other than by running a cross-country regression? (1) An experimental set-up (with countries randomly assigned to treatment and control groups) is in this case impossible. A case studies approach cannot resolve the issue either: the sample will always be special and there is no known way for such an approach to deal with the endogeneity of the two determinants, trade policy and institutional quality.(2) In addition, such methods are not replicable by independent research. If you are not prepared to adopt a regression approach you simply cannot investigate some important macro questions empirically.

Second, regressions are not ‘easy to do’ and something for ‘a rainy afternoon’. On the contrary, researchers are acutely aware of problems such as endogeneity (measurement error, reverse causality, omitted variable bias) and non-linearities. They spend most of their time struggling with these issues; the referees of the leading economic journals increasingly focus on these issues.

Third, the fact that some regression studies do not deal well with these issues does not mean that the regression method as such is wrong. Bulte and Hoebink referred to an evaluation of World Bank research that was critical of Collier’s regressions. But that critique came from Acemoglu, famous for using cross-country regressions to estimate the effect of institutions. Note that Bulte himself has used the method to investigate the effects of resource abundance.

Fourth, it is true that small changes in the data can sometimes destroy a conclusion, as happened with the (in)famous Burnside-Dollar paper on the effectiveness of aid. That does not disqualify the regression approach. Testing whether a result is robust to such changes is an essential part of academic research. If authors ignore the wise advice of Bulte and Hoebink to be careful and modest in conclusions then they will be corrected quickly. Any striking conclusion by a well-known academic will attract the same scrutiny as the Burnside-Dollar paper: if it’s wrong it won’t stand long.

Fifth, Hoebink complains that ‘Regression analysis doesn’t allow for complexity: if you have to allow for all the factors that are of influence on a social phenomenon you can not run your computer programmes anymore.’ The ancient Greeks already knew that simplification of real world complexity is essential for any analysis.

Finally, if you are interested in, say, the effects of the new aid architecture you can, of course, as Hoebink suggest, interview donors, civil servants and politicians. If the policy change is recent such interviews may be one of the few sources of information. But this can only tell you how the effect of changes in aid on corruption is perceived by stakeholders. Does Hoebink seriously suggest that a final conclusion can be based on such perceptions, without using more objective analytical tools?

Chris Elbers and Jan Willem Gunning, VU University Amsterdam

Notes:

(1) Dani Rodrik, ‘Institutions Rule’, Journal of Economic Growth, vol. 9, 2004, pp. 131-165 uses a regression approach for this issue.

(2) In a regression endogeneity means that the explanatory variables are correlated with the ‘error term’, resulting in biased estimates. In the case considered this can arise because both determinants are influenced by geographical country characteristics which also affect the dependent variable, growth. Rodrik deals very carefully with the endogeneity issue.