The good face, Moneyball
This is documented history rather than a laboratory result. The events are well recorded. What they teach is an argument, and the entry makes it openly.
For about a hundred years, baseball scouts selected players partly on something they openly called the good face, how a player looked, how he carried himself, whether he seemed like a ballplayer. It felt like expertise, the trained eye of men who had watched the game their whole lives. In 2002, the Oakland Athletics, one of the poorest teams in baseball, could not afford to compete on gut and glamour. So they did something radical: they handed player selection to measurement, buying up players who were statistically undervalued, often unimpressive-looking men who did dull, crucial things like getting on base. Other teams overlooked them precisely because they did not fit the picture of a star. Oakland won twenty consecutive games that season, a record, against teams spending three times as much. The good face had been measuring almost nothing. The boring numbers held the truth.
The way someone looks and carries themselves feels like strong evidence of ability, and it is often measuring nothing at all. Baseball spent a century drafting on the good face, confident that experienced eyes could see talent, while a team that ignored appearance and trusted the numbers beat them badly. This is worth remembering every time a candidate simply looks the part, or does not. Appearance, presence, and the sense that someone seems right are exactly the signals most likely to mislead you, because they are easy to read and only loosely connected to performance. The unglamorous evidence of what a person actually produces is duller to look at and far more honest.
Goodhart says that once a measure becomes a target it stops being a good measure, because people optimise the number instead of the goal. Moneyball says a broke team that trusted the numbers over a century of scouts' eyes won twenty straight. Both are right, and the difference is what the number is for. The Oakland statistics were a diagnostic nobody was being paid to hit, so they still described the thing. The rat tails were a bounty, so they described only the bounty. The test before trusting any number is whether anyone's reward depends on it. If nobody's does, it can beat the eye. If somebody's does, the eye is looking at the wrong thing already.
Sutherland says the spreadsheet only holds what can be counted, and what wins markets often cannot, so a decision made purely on the numbers will confidently reject the thing that would have worked. Moneyball says the scouts' feel for who looked right was measuring nothing, and the numbers beat it badly. What separates them is what the uncounted thing is actually doing. The good face felt like evidence and produced no results, which is the case for the numbers. Red Bull failed every rational test and produced a category, which is the case against them. The question is whether the thing the spreadsheet cannot see is showing up in outcomes, or only in how confident people feel.
Source: Michael Lewis, Moneyball, 2003. Documented sports history, 2002 season.
Michael Lewis, 2003
How measurement beat a century of gut instinct, and what happened when the numbers won.
Draw your own card. It does not take long, and it rewards taking your time.