Review of Expert Political Judgement (Philip E. Tetlock)
[Philip_E._Tetlock]_Expert_Political_Judgment_How(Bookos.org) Very interesting book! ---- Game Theorists. The rivalry between Sherlock Holmes and the evil genius Professor Moriarty illustrates how indeterminacy can arise as a natural by-product of rational agents second-guessing each other. When the two first met, Moriarty was eager, too eager, to display his capacity for interactive thinking by announcing: “All I have to say has already crossed your mind.” Holmes replied: “Then possibly my answer has crossed yours.” As the plot unfolds, Holmes uses his superior “interac- tive knowledge” to outmaneuver Moriarty by unexpectedly getting off the train at Canterbury, thwarting Moriarty who had calculated that Paris was Holmes’s rational destination. Convoluted though it is, Moriarty failed to recognize that Holmes had already recognized that Moriarty would deduce what a rational Holmes would do under the circum- stances, and the odds now favored Holmes getting off the train earlier than once planned.23 Indeterminacy problems of this sort are the bread and butter of behav- ioral game theory. In the “guess the number” game, for example, con- testants pick a number between 0 and 100, with the goal of making their guess come as close as possible to two-thirds of the average guess of all the contestants. 24 In a world of only rational players—who base their guesses on the maximum number of levels of deduction—the equilib- rium is 0. However, in a contest run at Richard Thaler’s prompting by the Financial Times, 25 the most popular guesses were 33 (the right guess if everyone else chooses a number at random, producing an average guess of 50) and 22 (the right guess if everyone thinks through the preceding argument and picks 33). Dwindling numbers of respondents carried the deductive logic to the third stage (picking two-thirds of 22) or higher, with a tiny hypereducated group recognizing the logically correct answer to be 0. The average guess was 18.91 and the winning guess, 13, which suggests that, for this newspaper’s readership, a third order of sophisti- cation was roughly optimal. interesting - Our reluctance to acknowledge unpredictability keeps us looking for predictive cues well beyond the point of diminishing returns. 39 I witnessed a demonstration thirty years ago that pitted the predictive abilities of a classroom of Yale undergraduates against those of a single Norwegian rat. The task was predicting on which side of a T-maze food would ap- pear, with appearances determined—unbeknownst to both the humans and the rat—by a random binomial process (60 percent left and 40 per- cent right). The demonstration replicated the classic studies by Edwards and by Estes: the rat went for the more frequently rewarded side (getting it right roughly 60 percent of the time), whereas the humans looked hard for patterns and wound up choosing the left or the right side in roughly the proportion they were rewarded (getting it right roughly 52 percent of the time). Human performance suffers because we are, deep down, de- terministic thinkers with an aversion to probabilistic strategies that ac- cept the inevitability of error. We insist on looking for order in random sequences. Confronted by the T-maze, we look for subtle patterns like “food appears in alternating two left/one right sequences, except after the third cycle when food pops up on the right.” This determination to ferret out order from chaos has served our species well. We are all bene- ficiaries of our great collective successes in the pursuit of deterministic reg- ularities in messy phenomena: agriculture, antibiotics, and countless other inventions that make our comfortable lives possible. But there are occa- sions when the refusal to accept the inevitability of error—to acknowledge that some phenomena are irreducibly probabilistic—can be harmful. indeed, but generally it is wise to not accept the unpredictability hypothesis about some fenomena. many things that were thought unpredictable for centures turned out to be predictable after all, or at least to some degree. i have confidence we will see the same for earthquakes, weather systems and the like in the future as well. predictability (and the related determinism) hypothesis are good working hypotheses, even if they turn out to be wrong some times. this is what i wrote about years ago on my danish blog here. basically, its a 2x2 table: What we think/what is true Determinism Indeterminism Determinism We keep looking for explanations for fenomena and in over time, we find regularities and explanations. We waste time looking for patterns that arent there. Indeterminism We dont spend time looking for patterns, but there actually are patterns we that cud use to predict the future, and hence we lose out on possible advances in science. We dont waste time looking for patterns that arent there. The above is assuming that indeterminism implies total unpredictability. This isnt true, but in the simplified case where were dealing with completely random fenomena and completely predictable fenomena, this is a reasonable way of looking at it. IMO, it is much better to waste time looking for explanations for things that are not orderly (after all), than risk not spotting real patterns in nature. Finally, regardless of whether it is rash to abandon the meliorist search for the Holy Grail of good judgment, most of us feel it is. When we weigh the perils of Type I errors (seeking correlates of good judgment that will prove ephemeral) against those of Type II errors (failing to discover durable correlates with lasting value), it does not feel like a close call. We would rather risk anointing lucky fools over ignoring wise counsel. Radi- cal skepticism is too bitter a doctrinal pill for most of us to swallow. exactly -
But betting is one thing, paying up another. Focusing just on reactions
to losing reputational bets, figure 4.1 shows that neither hedgehogs nor
foxes changed their minds as much as Reverend Bayes says they should
have. But foxes move more in the Bayesian direction than do hybrids and
hedgehogs. And this greater movement is all the more impressive in light
of the fact that the Bayesian updating formula demanded less movement
from foxes than from other groups. Foxes move 59 percent of the pre-
scribed amount, whereas hedgehogs move only 19 percent of the pre-
scribed amount. Indeed, in two regional forecasting exercises, hedgehogs
move their opinions in the opposite direction to that prescribed by Bayes’s
theorem, and nudged up their confidence in their prior point of view after
the unexpected happens. This latter pattern is not just contra-Bayesian; it
is incompatible with all normative theories of belief adjustment.8
https://en.wikipedia.org/wiki/Backfire_effect#Backfire_effect
-