Jan 29th 2004
From The Economist print edition
Too many economists misuse statistics
FIGURES lie, as everyone knows, and liars figure. That should make economists especially
suspect, since they rely heavily on statistics to try and resolve a wide range of controversies. For
example, does a rise in the minimum wage put people out of work? Are stockmarket returns
predictable? Do taxes influence whether a company pays dividends? In recent years, helped by
cheaper, more powerful computers, and egged on by policy-makers anxious for their views,
economists have analysed reams of statistics to answer such questions. Unfortunately, their
guidance may be deeply flawed.
Two economists, Deirdre McCloskey of the University of Illinois, and Stephen Ziliak of Roosevelt
University, think their colleagues do a lousy job of making sense of figures, often falling prey to
elementary errors. But their biggest gripe is that, blinded by statistical wizardry, many economists
fail to think about the way in which the world really works.
To be fair, statistics can be deceptive, especially when explaining human behaviour, which is
necessarily complicated, and to which iron laws do not apply. Moreover, even if a relationship
exists, the wrong conclusions can be drawn. In medieval Holland, it was noted that there was a
correlation between the number of storks living on the roof of a house and the number of children
born within it. The relationship was so striking that, according to the rules of maths that govern
such things, you could say with great confidence that the results were very unlikely to be merely
random. Such a relationship is said to be “statistically significant”. But the Dutch folklore of the
time that storks somehow increased human fertility was clearly wrong.
Examples of similar errors abound. W.S. Jevons, an English economist of the mid-19th century,
thought that sunspots influenced crop yields. More recently and tragically, British mothers have
felt the harsh effects of statistical abuse. An expert witness frequently called to give evidence in
the trials of mothers accused of murdering their children argued that the odds of more than one
cot death in a family were statistically so slim that three such deaths amounted to murder. On this
erroneous evidence, hundreds of parents have been separated from their children and many
others have been sent to prison.
A failure to separate statistical significance from plausible explanation is all too common in
economics, often with harmful consequences. In a past paper* Professors McCloskey and Ziliak
attacked other economists' over-reliance on statistical rather than economic reasoning, and
focused on one case in particular.
In the 1980s, the American state of Illinois launched a programme to keep people off the dole.
Economists asked whether its costs outweighed its benefits. One study estimated that the
programme produced benefits that were more than four times as large as the costs. Although this
seemed a good deal for taxpayers—and other tests seem to support this conclusion—the authors
of the study rejected such a finding because they found that their estimate was not statistically
significant. In other words, their results fell just short of 90% certainty—the usual, though ad hoc,
rule of thumb for most economic work—of not being random.
But far from this being an unusual case, Ms McCloskey and Mr Ziliak found that 70% of the papers
published during the 1980s in the American Economic Review (AER), one of the most respected
journals of the dismal science, failed to distinguish between “economic” and “statistical”
significance. They relied too much on numbers, and too little on economic reasoning.
The two had hoped things might be getting better in recent years. The reverse seems to be the
case. In their latest work**, Ms McCloskey and Mr Ziliak looked at all the AER articles in the
1990s, and found that more than four-fifths of them are guilty of the same sin. Indeed, so
pervasive is the cult of statistical significance, say the authors, that ever more economists
dispense altogether with the awkward question of whether the patterns they uncover have
anything meaningful to say about the real world.
Examples are legion, and can be found in the work of very distinguished economists. In a widely
quoted study of the minimum wage two Princeton University professors, Alan Krueger and David
Card, claimed to show that, contrary to what you might expect, a rise in minimum wages caused
less unemployment, not more. Though their statistics looked compelling, professors McCloskey
and Ziliak say, they seemed to indicate, at best, a rise in employment so small as to be
economically insignificant. Moreover, the paper did not address why this surprising result might be
true (although the authors have discussed that question elsewhere).
Another paper criticised by Ms McCloskey and Mr Ziliak is one co-written by Gary Becker, a Nobel-
winning economist. This claims to show that addiction is rational, mainly on the basis that people's
response to changes in price is statistically significant. This is interesting, but does not really
explain much. The three authors offered little account of why people become addicted—an odd life
choice for a rational person to make.
Most fundamentally, argue Ms McCloskey and Mr Ziliak, the focus on statistical significance often
means that they fail to ask whether their findings matter. They look, in other words, at things that
are statistically but not economically insignificant. Most people would prefer their conclusions to be
significant in both senses. Failing that, economic significance is presumably the more important.
* “The Standard Error of Regressions”. By Deirdre McCloskey and Stephen Ziliak. Journal of Economic Literature, March 1996
**”Size Matters: The Standard Error of Regressions in The American Economic Review”. (Forthcoming in the Journal of Socio-
Copyright © 2004 The Economist Newspaper and The Economist Group