The law of small numbers

In 1971 Amos Tversky and Daniel Kahneman gave a questionnaire about research design to professional psychologists, people trained in statistics, and found that most of them expected small samples to look just like the larger population they came from. Small samples rarely do. The pair called it belief in the law of small numbers, a play on the real statistical law of large numbers, which says that averages become reliable only as samples grow large.

Flip a coin six times and four heads in a row can feel meaningful, but short runs like that turn up by chance all the time. The same thing happens with anything measured on a small group, and it can be expensive. The statistician Howard Wainer looked at test results from 1,662 schools in Pennsylvania and found small schools in the top 50 about four times as often as you would expect. Kahneman tells how the Gates Foundation invested $1.7 billion in creating small schools, on the strength of findings like that. But nobody had looked at the bottom of the list, where small schools were over-represented too. In Wainer's data, small schools were not better on average. They were more variable, because a small school's results swing further from year to year, in both directions.

The same arithmetic applies to a career. Someone who hires might make a few dozen hiring decisions over many years. That feels like a lot of experience, but for telling a real pattern from coincidence it is a small sample, and it is spread across different companies, roles and markets. The confident rules people draw from their own hiring history can be small-school patterns: extremes that happened to be noticed and then turned into lessons. Later research on New York City's small high schools, using admission lotteries, did find them raising graduation rates, so the lesson here is about how the original rankings were read, not a final verdict on small schools.

The lessons you draw from your own experience rest on a smaller sample than they feel like, which is why two experienced people can hold opposite rules and both feel proven by their own history. Before you trust a rule from your own experience, write two numbers next to it: how many cases it is based on, and how many went the other way. If you cannot fill in either one, treat the rule as a hunch, and check it against larger evidence before you act on it. What that larger evidence says about hiring is in what actually predicts job performance.

This does not make experience worthless. How many cases you need depends on how big the effect is: something that fails nine times out of ten shows itself quickly. It is the subtler differences, which is where most hiring and business questions sit, that need far more cases than one career provides. And extreme results tend to drift back towards the average the next time they are measured, which is regression to the mean.

Source: Tversky and Kahneman, Belief in the Law of Small Numbers, Psychological Bulletin, 1971. The schools case: Howard Wainer, The Most Dangerous Equation, American Scientist, 2007, as retold by Daniel Kahneman in Thinking, Fast and Slow, 2011, which gives the $1.7 billion figure.

Thinking, Fast and Slow

Daniel Kahneman, 2011

On how readily the mind builds confident stories out of far too little data.