Researchers have tested this many times, in different places, over many years, and they keep finding the same result. This one is safe to trust.
A political scientist ran one of the longest studies of expert judgment ever attempted. Over about twenty years, he collected tens of thousands of concrete predictions from nearly three hundred experts, people paid to forecast politics and economics, and then waited to see what actually happened. The headline result was humbling: on average, the experts barely beat chance, and the most famous, most confident, most media-friendly ones were often the worst. But a smaller group did far better than the rest, and what separated them was how they thought rather than what they knew. The ones who held several partial explanations and updated as evidence arrived beat the ones working from a single confident framework. Underneath that sits a second question, which is whether the field allows anyone to get good in the first place. Weather forecasters learn within days whether they were right, and they become genuinely well calibrated. Pundits are rarely held to their past predictions, and stay confidently wrong for decades. The conditions that separate those two cases are set out in when expert intuition works.
Being an expert does not make your predictions good, unless your field tells you quickly and clearly when you were wrong. Without that feedback loop, years of experience just deepen your confidence without improving your accuracy. This is the quiet problem with interviewing: you meet a candidate, form a strong judgment, and then almost never learn the truth about how that hire turned out, at least not soon or clearly enough to learn from. So twenty years of interviewing can produce twenty years of unchallenged confidence and very little real skill. Feedback, not experience, is what makes judgment better.
Source: Philip Tetlock, Expert Political Judgment, 2005, on forecasting accuracy and cognitive style. On when expert intuition can be trusted: Kahneman and Klein, Conditions for Intuitive Expertise, American Psychologist (2009), and Hogarth on kind versus wicked learning environments.
Tetlock and Gardner, 2015
Tetlock's follow-up to his long-running forecasting tournaments, on the specific habits that separate the accurate few from the confident many.
Draw your own card. It does not take long, and it rewards taking your time.