The wisdom of crowds
Researchers have tested this many times, in different places, over many years, and they keep finding the same result. This one is safe to trust.
At a country fair in 1907, the scientist Francis Galton came across a competition. A live ox was on display, and hundreds of people, some farmers, many not, paid to guess its weight after it was slaughtered and dressed, with a prize for the closest. Galton, who had a low opinion of the average person's judgment, collected all 787 guesses to prove how wrong the crowd would be. Then he did the math. The individual guesses were all over the place, as he expected. But the average of all of them was 1,197 pounds. The true weight was 1,198. The crowd, as a whole, was off by a single pound, more accurate than any individual expert in the room, including the cattle experts. There was one condition that made it work: the guesses were independent, made privately, with no one influencing anyone else.
A crowd of independent guesses can be startlingly accurate, more accurate than the experts, but only when the guesses are truly independent. The instant people hear each other first, they stop thinking for themselves and start echoing, the errors line up instead of cancelling out, and the crowd becomes worse than its smartest member. This is the hidden flaw in most group decisions and hiring panels: everyone talks before deciding, so four opinions collapse into one loud one. To get the wisdom, gather judgments privately first, then discuss. Ask people to score before they hear each other, or you get conformity wearing the costume of consensus.
Eight hundred independent guesses at a country fair landed within a pound of an ox's weight. Anderson and Holt's subjects, deciding in sequence and seeing each other's choices, formed chains on the wrong answer about a third of the time while each reasoning correctly. Same kind of crowd, opposite result, and one condition explains it. The wisdom of crowds needs the guesses not to touch. A cascade is what happens when they do. A market price, a hiring panel or a room converging on a number can be either, and from inside the only way to tell is to ask how many people checked and how many were counting the nods.
Thiel says the important truths are the ones very few people agree with you on. Galton's ox says the average of eight hundred ordinary guesses was within a pound of the truth. They are not about the same crowd. The wisdom of crowds only holds when the guesses are independent, and Thiel's question is really a test of whether they are. A crowd that has talked first is one opinion wearing many badges, and the contrarian answer is what that crowd filters out. Interviewers who score before they confer get the first; interviewers who reward the candidate who reflects the company back get the second.
Source: Galton, 1907, Nature.
James Surowiecki, 2004
When many independent judgments beat the experts, and the specific conditions that make it work or fail.
Draw your own card. It does not take long, and it rewards taking your time.