The replication crisis
Researchers have tested this many times, in different places, over many years, and they keep finding the same result. This one is safe to trust.
In 2015, a large group of researchers did something psychology had rarely done at scale: they tried to repeat one hundred published studies, faithfully, to see if the original results held up. The results were a shock to the field. Only about a third reproduced the original effect at a statistically significant level, and the effects that did appear were on average half the size of the originals. Famous results that had been taught for years, cited thousands of times, built into textbooks and business advice, simply did not appear when the experiments were run again. Separate large replication efforts in the same period hit several of the field's best-known ideas, including power posing, some priming effects and willpower as a depletable fuel, which failed or weakened badly. It became known as the replication crisis, and it forced a hard question: how much of what the field believed was ever true?
A finding being famous, even being taught everywhere, does not mean it is true, and the last decade proved it with a whole field's greatest hits. The rule this leaves you with is simple and worth keeping for life. Results that have replicated for fifty or a hundred years, and across many places, deserve real trust. Results from a single recent study deserve interest but not belief, because a surprising one-off result is the kind of thing that later fails to reproduce. This is the whole reason the entries here carry an honest verdict rather than just a claim. Three that did not survive the checking: the ten thousand hour rule, the Hawthorne effect, and the Stanford prison experiment.
Lindy says the longer something has lasted, the longer it can be expected to last. The replication crisis says a third of psychology's most famous findings failed when re-run, and Hawthorne, the prison experiment and the ten thousand hour rule each lasted decades before anyone checked. They look like the same rule and they are not. Lindy measures persistence, which is evidence about durability and silent about truth. Replication measures whether a finding survives being tested again. A finding can persist for eighty years in textbooks because nobody went back to the data, which is QWERTY, not evidence. The version of Lindy that holds for knowledge is the replication entry's own rule: trust what has been re-tested for fifty years, not what has merely been repeated for fifty years.
Source: Open Science Collaboration, Estimating the reproducibility of psychological science, Science, 2015: 36 per cent of the replications were statistically significant, and replication effects were on average half the size of the originals.
Stuart Ritchie, 2020
From a psychologist whose own failed replication was part of what set the crisis off. On fraud, bias, negligence and hype, how each one produces findings that do not hold, and what the field is doing about it.
The Saturday letters
Every Saturday, one idea worth your time, with an honest account of how far the evidence really goes and where it stops. Free, a few minutes to read, and it leaves you with questions rather than instructions.
Draw your own card. It does not take long, and it rewards taking your time.