Review of Stat-Spotting: A Field Guide to Identifying Dubious Data
This is a fine little—not including the acknowledgements, footnotes, and index, it’s just 114 pages—book. It would be useful as a supplementary text in any statistics or quantitative methods course. It also could be enjoyed by anyone with an interest in data, especially when data are used to formulate public policy. The author, Joel Best, succinctly states his theme as follows (p. 5):
This book is guided by the assumption that we are exposed to many statistics that have serious flaws. This is important, because most of us have a tendency to equate numbers with facts, to presume that statistical information is probably pretty accurate information. If that’s wrong—if lots of the figures—that we encounter are in fact flawed—then we need ways of assessing the data we’re given.
Best makes his case through a set of well-chosen examples. Some are of numbers that are inaccurately high. He warns (p. 11), “. . . keep in mind one rule of thumb: in general, the worse things are the less common they are. . . . Most social problems display this pattern: there are lots of less serious cases, and relatively few very serious ones. This point is important because media coverage and other claims about social problems often feature disturbing typifying examples: that is, they use dramatic cases to illustrate the problem.”
Here are three examples.
Page 10: a claim that “more than four million women [in the U.S.] are battered to death by their husbands or boyfriends each year”. Best notes that four million is far more than the annual deaths of women from all causes. I found this claim repeated in other places such as here and here. How did this claim come about? Best doesn't speculate but I found sites claiming four million batterings but not deaths: http://www.clarkprosecutor.org/html/domviol/facts.htm and http://www.davidicke.com/forum/showthread.php?t=232778 I speculate "batterings" were transformed into "deaths".
Page 19, a classic example. Claim: “Today, a young person, age 14—26, kills herself or himself every 13 minutes in the United States.” But that’s more than 40,000 per year. The total number of suicides by people of all ages was about 32,000 per year. The correct number, Best finds, turns out to be about 4000 (in 2002), or about one every 131 minutes. Best concludes the claimed number was the product of a slipped decimal point.
Pages 44 and 45: What about the trend toward obesity in America? “In 1998, the federal government redefined the category ‘overweight’ . . . The redefinition meant that 29 million Americans whose weight had been considered normal suddenly were classified as overweight?” How many journalists know this? How many adjust the numbers indicating a rapid rise in obesity for the change in definition?
Best offers another set of examples that follows from the analysis on p. 51: “When hearing a number, it is always a good idea to pause for a second and ask yourself: How could they know that? How could they measure that?” This is, of course, very important. Milton Friedman used to ask frequently, “How do you know?”
Here are examples Best presents of this.
Pages 52 to 54: “In recent years, it has become very common to hear that this or that social problem costs America so much each year in lost productivity.” One instance: a claimed $75 billion annually is lost due to “hidden grief”? How do they know?
Pages 67 and 68: In late 2005 and early 2006 the press was concerned about Avian flu. The press was worried about a pandemic occurring because of the flu’s supposed virulence: half of the people hospitalized had died. But less than 1000 people, worldwide, died eventually. Why wasn’t there a pandemic? Best argues that hospitalization, especially in rural Asia where most of the deaths occurred, would have been mainly for the very sickest. Many other people probably never entered hospitals and lived, so the 50% death rate was significantly overstated.
Finally, as always in the social sciences, establising causality is tricky:
Page 83 presents a claim that families that eat dinner together have fewer problems. But maybe having fewer problems makes it easier to eat dinner together.
Pages 89 to 91: why did the crime rate in the U.S. trend down unexpectedly in the 1990s? There are lots of possibilities.
I also recommend Best’s Damned Lies and Statistics, especially the first chapter about the “worst social science statistic” ever: “Every year since 1950, the number of American children gunned down has doubled.” (This example seemed to impress my students.)