Signifying nothing?


Jan 29th 2004

From The Economist print edition


Too many economists misuse statistics


FIGURES lie, as everyone knows, and liars figure. That should make economists especially

suspect, since they rely heavily on statistics to try and resolve a wide range of controversies. For

example, does a rise in the minimum wage put people out of work? Are stockmarket returns

predictable? Do taxes influence whether a company pays dividends? In recent years, helped by

cheaper, more powerful computers, and egged on by policy-makers anxious for their views,

economists have analysed reams of statistics to answer such questions. Unfortunately, their

guidance may be deeply flawed.


Two economists, Deirdre McCloskey of the University of Illinois, and Stephen Ziliak of Roosevelt

University, think their colleagues do a lousy job of making sense of figures, often falling prey to

elementary errors. But their biggest gripe is that, blinded by statistical wizardry, many economists

fail to think about the way in which the world really works.


To be fair, statistics can be deceptive, especially when explaining human behaviour, which is

necessarily complicated, and to which iron laws do not apply. Moreover, even if a relationship

exists, the wrong conclusions can be drawn. In medieval Holland, it was noted that there was a

correlation between the number of storks living on the roof of a house and the number of children

born within it. The relationship was so striking that, according to the rules of maths that govern

such things, you could say with great confidence that the results were very unlikely to be merely

random. Such a relationship is said to be “statistically significant”. But the Dutch folklore of the

time that storks somehow increased human fertility was clearly wrong.


Examples of similar errors abound. W.S. Jevons, an English economist of the mid-19th century,

thought that sunspots influenced crop yields. More recently and tragically, British mothers have

felt the harsh effects of statistical abuse. An expert witness frequently called to give evidence in

the trials of mothers accused of murdering their children argued that the odds of more than one

cot death in a family were statistically so slim that three such deaths amounted to murder. On this

erroneous evidence, hundreds of parents have been separated from their children and many

others have been sent to prison.


A failure to separate statistical significance from plausible explanation is all too common in

economics, often with harmful consequences. In a past paper* Professors McCloskey and Ziliak

attacked other economists' over-reliance on statistical rather than economic reasoning, and

focused on one case in particular.


In the 1980s, the American state of Illinois launched a programme to keep people off the dole.

Economists asked whether its costs outweighed its benefits. One study estimated that the

programme produced benefits that were more than four times as large as the costs. Although this

seemed a good deal for taxpayers—and other tests seem to support this conclusion—the authors

of the study rejected such a finding because they found that their estimate was not statistically

significant. In other words, their results fell just short of 90% certainty—the usual, though ad hoc,

rule of thumb for most economic work—of not being random.


But far from this being an unusual case, Ms McCloskey and Mr Ziliak found that 70% of the papers

published during the 1980s in the American Economic Review (AER), one of the most respected

journals of the dismal science, failed to distinguish between “economic” and “statistical”

significance. They relied too much on numbers, and too little on economic reasoning.


Increasingly insignificant


The two had hoped things might be getting better in recent years. The reverse seems to be the

case. In their latest work**, Ms McCloskey and Mr Ziliak looked at all the AER articles in the

1990s, and found that more than four-fifths of them are guilty of the same sin. Indeed, so

pervasive is the cult of statistical significance, say the authors, that ever more economists

dispense altogether with the awkward question of whether the patterns they uncover have

anything meaningful to say about the real world.


Examples are legion, and can be found in the work of very distinguished economists. In a widely

quoted study of the minimum wage two Princeton University professors, Alan Krueger and David

Card, claimed to show that, contrary to what you might expect, a rise in minimum wages caused

less unemployment, not more. Though their statistics looked compelling, professors McCloskey

and Ziliak say, they seemed to indicate, at best, a rise in employment so small as to be

economically insignificant. Moreover, the paper did not address why this surprising result might be

true (although the authors have discussed that question elsewhere).


Another paper criticised by Ms McCloskey and Mr Ziliak is one co-written by Gary Becker, a Nobel-

winning economist. This claims to show that addiction is rational, mainly on the basis that people's

response to changes in price is statistically significant. This is interesting, but does not really

explain much. The three authors offered little account of why people become addicted—an odd life

choice for a rational person to make.


Most fundamentally, argue Ms McCloskey and Mr Ziliak, the focus on statistical significance often

means that they fail to ask whether their findings matter. They look, in other words, at things that

are statistically but not economically insignificant. Most people would prefer their conclusions to be

significant in both senses. Failing that, economic significance is presumably the more important.


* “The Standard Error of Regressions”. By Deirdre McCloskey and Stephen Ziliak. Journal of Economic Literature, March 1996

**”Size Matters: The Standard Error of Regressions in The American Economic Review”. (Forthcoming in the Journal of Socio-



Copyright © 2004 The Economist Newspaper and The Economist Group



return to home