Tetlock and the limits of expert prediction

A political scientist ran a study of expert judgement that lasted about twenty years. Over about twenty years, he collected tens of thousands of concrete predictions from nearly three hundred experts, people paid to forecast politics and economics, and then waited to see what actually happened. The headline result was humbling: on average, the experts barely beat chance, and the most famous, most confident, most media-friendly ones were often the worst. But a smaller group did far better than the rest, and what separated them was how they thought rather than what they knew. The ones who held several partial explanations and updated as evidence arrived beat the ones working from a single confident framework. Underneath that sits a second question, which is whether the field allows anyone to get good in the first place. Weather forecasters learn within days whether they were right, and they become genuinely well calibrated. Pundits are rarely held to their past predictions, and can stay wrong for decades while feeling more certain. The conditions that separate those two cases are set out in when expert intuition works.

To describe that style, Tetlock borrowed a line the philosopher Isaiah Berlin had taken from an ancient Greek poet: the fox knows many things, but the hedgehog knows one big thing. Hedgehogs, in his sorting, explained the world through one favourite theory and bent new facts until they fitted it. Foxes held several partial explanations at once, were comfortable with ambiguity, and moved their forecasts as the evidence moved. The foxes did better, often by a wide margin. The hedgehogs did worst of all on long-range forecasts inside their own specialty, which is where you would expect expertise to help most. Neither group, it should be said, beat the simple statistical rules.

The line that made the study famous, that the average expert did about as well as a dart-throwing chimpanzee, is Tetlock's own joke, and it flattens what he found. The average was poor, of course, but the spread was the finding: a minority were measurably better, and the next stage of his work set out to learn who they were and whether their skill could be taught. That stage was a later forecasting tournament, told in Superforecasting, and two of its results bear directly on this entry. Even simple training exercises improved people's accuracy, measured with a scoring rule that punishes confident mistakes hardest. And the best forecasters learned to tell apart far more degrees of uncertainty than the seven-step verbal scale US intelligence analysts were using. Forecasting turned out to be a skill that improves with practice, on one condition, which is that someone keeps the score.

That condition is the one the whole field keeps arriving at, and the clearest demonstration of it is the weather forecast. Forecasters make the same type of call every day and find out within a day whether it rained, and studies of US National Weather Service forecasts going back to the 1970s found their probability forecasts reliable, which means, roughly, that on the days they say seventy per cent, it rains on about seventy per cent of them. Political punditry is the opposite case. Its forecasts resolve years later, if anyone checks at all, and a failed call can always be explained away afterwards, so the confidence grows and the accuracy stays where it was. The two conditions that separate those worlds are the subject of when expert intuition works.

Most of the forecasts that shape your working life are made by people who will never be scored on them. The annual plan is a forecast, and so is the sales target, the headcount number and the launch date. Someone looked at next year, made a call, and handed it down as a goal. When the year ends, the target is reset, the story is rewritten around whatever happened, and nobody goes back to ask how good the original call was. That is a wicked environment in Hogarth's sense, and it produces what Tetlock found in the pundits: people who have been forecasting for twenty years with complete confidence and no measured skill.

Interviewing is the same trap, closer to home. You meet a candidate, form a strong view within the hour, and then rarely learn, clearly and soon, whether you were right. The result arrives a year later, tangled up with the manager they got, the market that year and the project they landed on, so you can explain most outcomes in a way that leaves your judgement intact. Twenty years of interviewing can produce twenty years of confidence and very little calibration, and from the inside the confidence feels identical to skill. That is why the research on hiring keeps finding that structure beats the friendly conversation, and why hindsight makes every past call look more obvious than it was.

The correction is cheap, and it is the one nobody bothers with. Write your forecasts down: the number, the date it resolves, and how sure you are, as a percentage. Then go back and score them. It will be uncomfortable the first time, and that discomfort is the feedback your environment was never going to give you. The same applies to the forecasts handed to you. Ask what this year's target assumes about next year, write the assumption down, and check it when the year closes. None of this says experience is worthless. It says experience turns into skill only where the results come back.

Source: Philip Tetlock, Expert Political Judgment: How Good Is It? How Can We Know?, Princeton University Press, 2005, new edition 2017. Counts differ between sources: the book reports 82,361 probability estimates, and many summaries give the number of forecasts as about 28,000; this entry says tens of thousands, which holds either way. The later tournament and its training findings: Tetlock and Gardner, Superforecasting, 2015. On weather forecasters: Murphy and Winkler, Reliability of Subjective Probability Forecasts of Precipitation and Temperature, Journal of the Royal Statistical Society, Series C, 1977. On kind and wicked learning environments: Robin Hogarth, Educating Intuition, 2001.

Superforecasting

Tetlock and Gardner, 2015

Tetlock's follow-up to his long-running forecasting tournaments, on the specific habits that separate the accurate few from the confident many.