Natural frequencies

The demonstration is a screening test, and the numbers are Gigerenzer's own. One percent of women in a particular age group have breast cancer. If a woman has it, there is a ninety percent chance the mammogram is positive. If she does not have it, there is still a nine percent chance the mammogram is positive. A woman tests positive. What is the chance she actually has cancer?

Gigerenzer put that question to 160 gynaecologists at the start of a course on risk literacy. Sixty percent answered that the chance was eighty to ninety percent. Nineteen percent answered that it was one percent. The spread is the finding as much as the error is: given identical information, the answers ranged from one in a hundred to nine in ten. His own comment was that if patients knew about this variability, they would rightly be scared.

The correct answer is about nine percent, and here is the same information counted rather than expressed as rates. Ten in every thousand women have breast cancer. Of those ten, nine will test positive. Of the 990 who do not have cancer, about 89 will test positive anyway. So 98 women test positive in total, and nine of them have cancer. Nine out of 98, which is roughly one in ten. Nothing has been simplified and no information has been dropped. The arithmetic that required Bayes's rule in the first version requires only addition and a comparison in the second.

The effect is large and it has held up. A meta-analysis of these tasks puts typical performance at around four percent correct in probability format and around twenty-four percent in natural frequency format. In Gigerenzer's own session, after a single teaching block, eighty-seven percent of the gynaecologists could do it. And in work published in Cognition in 2006, ten-year-old children solved these problems when given natural frequencies while being helpless with conditional probabilities.

Which brings the argument. Kahneman and Tversky's tradition treated results like the doctors' answers as evidence of a systematic mental flaw, base rates being neglected in favour of how well the evidence seems to fit. Gigerenzer's response is that the flaw is in the representation rather than the mind, and that the same brain performs competently once the numbers are put in a form it evolved to handle. He has been openly sharp about the other side's explanation, describing the appeal to a representativeness heuristic as a one-word explanation that was never properly defined.

The honest limits cut both ways, which is why this stays interesting rather than settling. Natural frequencies produce a six-fold improvement and still leave roughly three quarters of people getting the answer wrong, so the format is clearly not the whole story. Later work found that many people handed natural frequencies convert them back into probabilities before calculating, defeating the advantage. And Gigerenzer is careful that natural frequencies are joint frequencies rather than simple ones, a distinction that has caused studies to report no benefit while testing something else entirely.

Every number that reaches a decision in your company arrives as a rate. Conversion rate, churn probability, win rate, confidence interval, forecast accuracy. Rates are compact, they compare cleanly across periods, and they are the format in which almost nobody can reason well. The doctors in that study were not careless and they were not innumerate. They were given the information in the form that hides the structure.

The practical move is one question, and it works in any meeting. Ask for the count. Out of how many, how many did what. A twenty percent close rate becomes forty deals out of two hundred conversations, and the moment it is counted rather than rated, everything sitting underneath it becomes visible: how many were repeat buyers, how many came from a single source, and how thin the denominator actually is. That last one matters more than people expect, because a rate calculated on a handful of cases looks exactly like a rate calculated on thousands, which is the law of small numbers hiding inside a percentage sign.

The uncomfortable version is what happens when someone cannot answer that. A person presenting a rate they cannot decompose is presenting a number they did not build and cannot interrogate. That is not usually dishonesty. It is a spreadsheet cell that has been forwarded so many times its origin is gone. But the decision resting on it is being made blind, and the question that exposes it is free.

There is a second use, in the other direction. If you are trying to make someone understand a risk, a cost or a trade-off, the count is the format that lands. Not two percent of customers churn, but of the last five hundred customers, ten left, and eight of those ten came from the same channel. The information is identical, and one version can be argued with while the other is nodded at. The related mechanism, where the same fact stated two ways produces different decisions, is the framing effect.

Source: Gigerenzer, What are natural frequencies? Doctors need to find better ways to communicate risk to patients, BMJ, volume 343, 2011, which contains the gynaecologist survey and the worked example above. The original demonstration: Gigerenzer and Hoffrage, How to improve Bayesian reasoning without instruction: frequency formats, Psychological Review, volume 102, 1995, pages 684 to 704. The children's study: Zhu and Gigerenzer, Children can solve Bayesian problems: the role of representation in mental computation, Cognition, volume 98, 2006. The meta-analytic figures of roughly four percent against twenty-four percent are from McDowell and Jacobs, 2017.

Risk Savvy

Gerd Gigerenzer, 2014

Gigerenzer's own book for a general reader, and it is this entry at book length. His argument is that doctors, lawyers and financial advisers misunderstand statistics far more often than anyone assumes, that this leaves the rest of us exposed, and that the fix is teachable rather than innate.