Survivorship bias

The oldest version is two thousand years old. Cicero tells of a visitor, Diagoras, being shown the painted tablets that sailors had dedicated to the gods after praying and surviving storms at sea, offered as proof that prayer works. He asked where the paintings were of the sailors who prayed and drowned. Nassim Taleb later gave the missing paintings a name, silent evidence.

The version everyone knows comes from the Second World War. The Statistical Research Group at Columbia University worked on problems for the American military, and in 1943 the statistician Abraham Wald wrote a series of eight memoranda on a practical question: how to estimate which parts of a bomber were most vulnerable, when the only planes you can examine are the ones that came back. His reasoning was that damage on a returning plane is, by definition, damage a plane can absorb. If the returning planes showed little damage in some section, that was not because those sections were rarely hit. It was because a hit there tended to bring the plane down, so it never came back to be counted. In his comments he suggested the analysis could guide where to put protective armour. The memoranda were never published externally during the war, and the Center for Naval Analyses reprinted them in 1980, noting that the method had been used on data from the Korean and Vietnam wars.

The story as usually told compresses those eight technical memoranda into a single remark in a room full of officers, and ends with the advice being taken and lives being saved. The memoranda are about method. They do not record what the military did with them, so the happy ending belongs to the retelling rather than the record. The reasoning in the middle is the part that is right, and it is the part worth keeping.

The business version sits on most management shelves. In Search of Excellence, published in 1982, studied companies that were doing well and drew out what they had in common. What it did not have was a group of companies that shared the same habits and failed, so there was no way to tell whether the habits explained the success. Within a couple of years BusinessWeek ran a cover asking who was excellent now, because a number of the companies on the list had stumbled. The first problem is survivorship. The second is closer to regression to the mean, and it is worth keeping the two apart.

And in finance it has been measured rather than argued. Funds that perform badly tend to close or merge, and when they drop out of the databases researchers use, the average past performance of the funds that remain looks better than the funds as a group ever did. Brown, Goetzmann, Ibbotson and Ross set out how much this distorts performance studies in 1992, and correcting for it has been standard practice in the field since.

Every success story you study is a plane that came back. The founder explaining how they did it, the playbook everybody copies, the career path you admire: what those people went through tells you what can be survived. It tells you nothing about what kills, because the people it killed are not in the room to say so, and their absence does not show up anywhere. So the plan always looks safer than it is.

The same thing is built into ordinary business data. Your customer satisfaction survey goes to the customers who stayed. Your exit interviews reach the people who left politely. And in hiring, you know how the people you hired turned out, and you will never know how the ones you rejected would have done, so everything your instinct has learned about hiring it learned from the planes that came back. That is part of why feedback matters so much and why it is so hard to get.

The correction is simple to state. Before you copy anything that worked, name three people or companies who did the same thing and did not make it. If you cannot, what you have is a list of survivors. And before you trust any number from the past, ask what never made it into the data it was drawn from. How much of the winning was luck to begin with is the music lab. The case where survival really does carry information, about how long something will last rather than why it lasted, is the Lindy effect.

Source: Abraham Wald, A Method of Estimating Plane Vulnerability Based on Damage of Survivors, Statistical Research Group, Columbia University, 1943, reprinted by the Center for Naval Analyses, 1980. On the memoranda and what they contain: Mangel and Samaniego, Abraham Wald's Work on Aircraft Survivability, Journal of the American Statistical Association, 1984. The oldest version: Cicero, On the Nature of the Gods, on Diagoras. The business case: Peters and Waterman, In Search of Excellence, 1982, and BusinessWeek's follow-up cover, Who's Excellent Now. The measured case: Brown, Goetzmann, Ibbotson and Ross, Survivorship Bias in Performance Studies, Review of Financial Studies, 1992.

Fooled by Randomness

Nassim Nicholas Taleb, 2001

The people who vanished are the main character. It will change how you read every success story after it.