The wisdom of crowds
Researchers have tested this many times, in different places, over many years, and they keep finding the same result. This one is safe to trust.
At a livestock show in Plymouth in 1906, there was a competition that caught the attention of the scientist Francis Galton. A live ox was on display, and hundreds of people, some farmers, many not, paid to guess its weight after it was slaughtered and dressed, with a prize for the closest. Galton borrowed all 787 usable entry cards to see how trustworthy popular judgement really was. Then he did the maths. The individual guesses were all over the place, as he expected. But the average of all of them was 1,197 pounds. The true weight was 1,198. The crowd, taken as a whole, was off by a single pound. Galton himself preferred the middle guess, 1,207 pounds, which was still within one per cent. There was one condition that made it work: the guesses were independent, made privately, with no one influencing anyone else.
A crowd of independent guesses can be startlingly accurate, but only when the guesses are truly independent. Once people hear each other first, they can stop thinking for themselves and start echoing, and the errors line up instead of cancelling out. This is the hidden flaw in many group decisions and hiring panels: everyone talks before deciding, so four opinions can collapse into one loud one. To get the wisdom, gather judgements privately first, then discuss. Ask people to score before they hear each other, or you get conformity wearing the costume of consensus. The mechanism underneath all of this is information cascades: when people decide in sequence and can see what came before, the chain can form on an answer few people independently held.
Eight hundred independent guesses at a country fair landed within a pound of an ox's weight. Anderson and Holt's subjects, deciding in sequence and seeing each other's choices, formed chains on the wrong answer about a third of the time while each reasoning correctly. Same kind of crowd, opposite result, and one condition explains it. The wisdom of crowds needs the guesses not to touch. A cascade is what happens when they do. A market price, a hiring panel or a room converging on a number can be either, and from inside the only way to tell is to ask how many people checked and how many were counting the nods.
Thiel says the important truths are the ones very few people agree with you on. Galton's ox says the average of eight hundred ordinary guesses was within a pound of the truth. They are not about the same crowd. The wisdom of crowds only holds when the guesses are independent, and Thiel's question is really a test of whether they are. A crowd that has talked first is one opinion wearing many badges, and the contrarian answer is what that crowd filters out. Interviewers who score before they confer get the first; interviewers who reward the candidate who reflects the company back get the second.
Source: Galton, Vox Populi, Nature, 7 March 1907, with the mean reported in his reply the following week. On the numbers and how the story changed in retelling: Kenneth Wallis, Revisiting Francis Galton's Forecasting Competition, 2014.
James Surowiecki, 2004
When many independent judgements beat the experts, and the specific conditions that make it work or fail.
The Saturday letters
Every Saturday, one idea worth your time, with an honest account of how far the evidence really goes and where it stops. Free, a few minutes to read, and it leaves you with questions rather than instructions.
Draw your own card. It does not take long, and it rewards taking your time.