The law of small numbers
Researchers have tested this many times, in different places, over many years, and they keep finding the same result. This one is safe to trust.
People are strangely willing to trust a pattern drawn from almost no data. Two researchers showed this with a series of simple questions and found that people expect small samples to look just like large ones, which they never do. Flip a coin six times and a run of four heads feels meaningful, but it is just noise. The researchers called it belief in the law of small numbers, a play on the real statistical law of large numbers, which only holds for big samples. Here is where it bites in real life. Over an entire career, a founder might make thirty to fifty hires. That feels like a lot of experience. But statistically, thirty to fifty is a tiny sample, far too small to separate a real pattern from coincidence. The confident rules people extract from their own hiring history, this type always works out, that background never does, are usually patterns in noise.
The lessons you draw from your own experience are built on a smaller sample than they feel like, and small samples are mostly noise dressed as wisdom. No single person's career generates enough decisions to reliably know what predicts success, which is why two experienced people can hold opposite confident rules and both feel proven by their own history. The answer is to hold your rules loosely rather than to ignore experience, and to check them against larger evidence when you can, because the pattern you are sure you see may not survive the next thirty cases.
The law of small numbers says you draw confident rules from thirty or fifty cases, and a sample that size cannot separate signal from coincidence. Regression to the mean says any result that contains luck contains the most luck at its extremes, and extremes are followed by something more ordinary whether or not you do anything. The hire who interviewed brilliantly and then performed like everyone else is both at once. One is about how few cases you have. The other is about which cases you happen to be looking at, which is always the extreme ones, because extremes are what get your attention. The question that defuses both is the same: compared to what. Against your own history, and against the cases you did not act on.
Source: Tversky and Kahneman, 1971, Psychological Bulletin.
Daniel Kahneman, 2011
On how readily the mind builds confident stories out of far too little data.
Draw your own card. It does not take long, and it rewards taking your time.