Expert Political Judgment: How Good is it? How can We Know?

"Expert political judgment" -- it sounds like an oxymoron, but only because it is. Philip E. Tetlock's groundbreaking research shows that experts are no better than the rest of us when it comes to political prognostication. But then again, you probably had a sneaking hunch that that was so. You need rely on hunches no more. Tetlock is Professor of Leadership at the Haas Management of Organizations Group, U.C. Berkeley. A Yale graduate with his Ph.D. in Psychology, Expert Political Judgment is the result of his 20 year statistical study of nearly 300 impeccably credentialed political pundits responding to more than 80,000 questions in total. The results are sobering. In most cases political pundits did no better than dart throwing chimps in prediciting political futures. Of course, Tetlock did not actually hire dart throwing chimps -- he simulated their responses with the statistical average. If the computer was programmed to use more sophisticated statistical forecasting techniques (e.g., autoregressive distributed lag models), it beat the experts even more resoundingly.Were the experts better at anything? Well, they were pretty good at making excuses. Here are a few: 1. I made the right mistake. 2. I'm not right yet, but you'll see. 3. I was almost right. 4. Your scoring system is flawed. 5. Your questions aren't real world. 6. I never said that. 7. Things happen. Of course, experts applied their excuses only when they got it wrong... er... I mean almost right... that is, about to be right, or right if you looked at it in the right way, or what would have been right if the question were asked properly, or right if you applied the right scoring system, or... well... that was a dumb question anyway, or....Not only did experts get it wrong, but they were so wedded to their opinions that they failed to update their forecasts even in the face of building evidence to the contrary. And then a curious thing happened -- after they got it wrong and exhausted all their excuses, they forgot they were wrong in the first place. When Tetlock did follow-up questions at later dates, experts routinely misremembered their predictions. When the expert's models failed, they merely updated their models post hoc, giving them the comforting illusion that their expert judgment and simplified model of social behavior remained intact. Compare this with another very complex system -- predicting the weather. In this latter case, there is a very big difference in the predictive abilities of experts and lay persons. Meteorologists do not use over-simplified models like "red in the morning, sailor's warning." They use complex modeling, statistical forecasting, computer simulations, etc. When they are wrong, weathermen do not say, well, it almost rained; or, it just hasn't rained yet; or, it didn't rain, but predicting rain was the right mistake to make; or, there's something wrong with the rain guage; or, I didn't say it was going to rain; or, what kind of a question is that?Political experts, unlike weathermen, live in an infinite variety of counterfactual worlds; or as Tetlock writes, "Counterfactual history becomes a convenient graveyard for burying embarrassing conditional forecasts." That is: sure, given x, y, and z, the former Soviet Union collapsed; but if z had not occurred, the former Soviet Union would have remained intact. Really? Considering the expert got it wrong in the first place, how could they possibly know the outcome in a hypothetical counterfactual world? At best, this is intellectual dishonesty. At worst, it is fraud.But some experts did better than others. In particular, those who were less dogmatic and frequently updated their predictions in response to countervailing evidence (Tetlock's "foxes") did much better than the opposing camp (termed "hedgehogs"). The problem is that hedgehogs climb the ladder faster and have positions of greater prominence. My Machiavellian take? You might as well make dogmatic pronouncements because all the hedgehogs you work for aren't any better at predicting the future than you are -- they're just more sure of themselves. So, work on your self-confidence. It is apparently the only thing anyone pays any attention to.

This is a critical book for anyone one who depends on professional forecasters of "social" variables, and even more for anyone whose livelihood rests on making such forecasts. "Social" because Tetlock's book is focussed on political forecasting, but I'm convinced that it applies to economic and social forecasting as well. (Having spent a professional career forecasting economic variables, I have some insight here). Tetlock is not discussing forecasting in the hard sciences, where forecasting is based on much harder data.His first critical conclusion is that, in forecasting complex political events, "we could do as well by tossing coins as by consulting experts". This is based on a massive set of surveys of expert opinion that were compared to outcomes in the real world over many years. The task was enormously complex to set up; defining an experiment in the social sciences presents the problems that constantly arise in making judgements in these sciences (what does one measure, and how? How can bias be measured and eliminated? etc. etc.) Much of the book is devoted to the problems in constructing the study, and how they were resolved.His second key conclusion is that, while that may be true of experts as an undifferentiated group, some experts do significantly better than other experts. This does not reflect the level of expertise involved, nor does it reflect political orientation. Rather, it reflects the way the experts think. Poorer performers tend to be what Tetlock characterizes as "hedgehogs" -- people who apply theoretical frameworks, who stick with a line of argument, and who believe strongly in their own forecasts. The better performers tend to be what he calls "foxes" -- those with an eclectic approach, who examine many hypotheses, and who are more inclined to think probabilistically, by grading the likelihood of their forecasts.But, as he notes, the forecasters who get the most media exposure tend to be the hedgehogs, those with a strong point of view that can be clearly expressed. This makes all the sense in the world; someone with a clear cut and compelling story is much more fun to listen to (and much more memorable than) someone who presents a range of possible outcomes (as a former many-handed economist, I know this all too well).What does that mean for those of us who use forecasts? We use them in making political decisions, personal financial decisions, and investment decisions. This book tells us that WHAT THE EXPERTS SAY IS NOT LIKELY TO ADD MUCH TO THE QUALITY OF YOUR OWN DECISION MAKING. And that says be careful how much you pay for expert advice, and how much you rely on it. That of course applies to experts in the social sciences, NOT to experts in the hard (aka real) sciences. Generally, it is a good idea to regard your doctor as a real expert.Because it makes it impossible to avoid these conclusions, I gave this book five stars; this is very important stuff. I would not have given it five stars for the way in which it is written. For me, it read as if it had been written for other academics, rather than for the general reader. This is hard to avoid, but some other works in the field do manage -- for example, "Thinking Fast and Slow". Don't skip the book because it is not exactly an enjoyable read, however: its merit far outweighs its manner.

This book presents some amazing results. Essentially, what it proves is that, in the area of political prediction, experience, eminence and in-depth knowledge are of essentially no value. This parallels in a way the now hardly disputed result that highly-paid investment managers almost never, over the long run, beat the market.Most academics with an interesting, new theory like this write two books. A magnum opus stuffed with references and experimental detail for the academic community, and another, shorter, more easily digestible work for the lay reader. This is what Bruce Bueno de Mesquita did for "The Logic of Political Survival" when he wrote "The Dictator's Handbook". Sadly Tetlock has failed to come out with the second type of book, so we are left to wade through a fairly dry academic tome which, while having its moments, is a bit too footnote heavy for me.I urge you to skim through this, maybe using a library copy, but I would counsel against buying it, unless you are of an academic cast of mind.

Der Fuchs weiß viele verschiedene Sachen, der Igel aber nur eine große .(Archilochos, griechischer Lyriker, 680 bis 645 v. Chr.)Philip Tetlock ist gelernter Psychologe und Prof. für Leadership an der Univ. Kalifornien. Er forscht darüber, welche Faktoren für menschliche Weitsicht und Blindheit verantwortlich sind. Von 1985 bis 2003 befragte er 284 ausgewählte amerikanische Experten über den Lauf der Welt. Der Prognosehorizont war meistens 2 bis 5 Jahre, in Einzelfällen aber auch bis zu einer Dekade. Die Teilnehmer mussten z.B. beantworten, ob in einem bestimmten Land die jetzige Regierung nach den nächsten bzw. übernächsten Wahlen noch immer am Ruder ist (in autoritären Regimen, ob sie weg geputscht wird). Andere Fragen waren etwa, ob sich die Provinz Quebec von Kanada loslösen wird, ob zwischen Indien und Pakistan ein Krieg ausbricht. Ob das Wachstum des Bruttonationalproduktes, die Staatsverschuldung oder der Zinssatz der Notenbank höher, niedriger oder gleich ausfallen wird, die Preise für wichtige Rohstoffe nach oben, unten gehen oder gleich bleiben, ob die Internet-Börsenblase innerhalb des Prognosehorizontes platzt. Die Experten mussten nicht nur die Richtung angeben, sondern auch, für wie wahrscheinlich sie die einzelnen Szenarien – z.B. die Abspaltung Quebecs – hielten.Ein roter Faden in Tetlock's Experimenten ist: Die Experten sind sich – egal auf welchem Gebiet – viel zu sicher. Wenn sie etwas als „praktisch sicher“ einstufen, dann kommt es höchstens mit 70% Wahrscheinlichkeit vor. Es sind auch Wunder gar nicht so selten. Es treten Ereignisse ein, die die Experten für (denk-)unmöglich gehalten haben. Generell ist die Prognose „kräht der Hahn am Mist bleibt das Wetter so wie es ist“ ziemlich gut. Sie hätte gegen die Experten klar gewonnen. Die Experten überschätzen krass die Wahrscheinlichkeit von Wendungen zum Guten oder Schlechten. Dies gilt insbesondere für die Igel.Die Unterscheidung zwischen Igel (Engl. Hedgehog) und Fuchs geht auf einen – im englischen Sprachraum – allgemein bekannten Aufsatz des Philosophen Sir Isaiah Berlin (1909 bis 1996) zurück. Nach Berlins Auffassung versuchten die Igel, ein allumfassendes System menschlicher Handlungen, der Geschichte und von moralischen Werten zu entwickeln. Die Füchse hingegen tendierten eher dazu, überall eine Vielfalt zu sehen. Ein Fuchs verfolgt viele Ziele, oft ohne inneren Zusammenhang oder sogar widersprüchlich. Typische Igel sind für Berlin Plato, Pascal, Hegel, Dostojewski, Nietzsche oder Proust, Füchse hingegen Shakespeare, Aristoteles, Erasmus von Rotterdam, Goethe, Puschkin oder Joyce. Es gibt natürlich auch Mischformen. Der von Berlin verehrte Tolstoi war seiner Auffassung nach ein Fuchs, der gerne ein Igel gewesen wäre. Für die Welt der Experten könnte man den Typus des Hedgehogs auch etwas vereinfachend mit „Fachidiot“ beschreiben. Der Fuchs ist der umtriebige bunte Hund, der alles und nichts kann.Der oben geschilderte Zusammenhang ist die zentrale Botschaft des Buches. Tetlock kann sich auch überzeugend belegen. Daneben gibt es eine Reihe von anderen Zusammenhängen. Z.B. Stars die besonders oft in den Medien zitiert werden liefern sehr schlechte Prognosen ab. Das hängt mit der Vorliebe der Medien für Igel zusammen.Inhaltlich verdient das Buch 6 Sterne. Allerdings ist es ziemlich akademisch-öd geschrieben. Tetlock wollte damit offensichtlich einen akademischen Klassiker verfassen. Das ist ihm gelungen. Für das breite Publikum ist seine Darstellung aber eher ermüdend und wenn man sich nicht in Statistik auskennt teilweise auch unverständlich. Ich kenn mich in Statistik aus, habe mich aber durch manche Passagen eher durchgekämpft. Vieles könnte man ohne inhaltliche Abstriche einfacher, flotter sagen. Aber dann wärs kein akademischer Klassiker geworden.Wer an einer gut lesbaren Zusammenfassung interessiert ist wohl mit dem Artikel "Trau keinem Igel" in der Dezember 2014 Ausgabe von Chrilly's Monatlicher Goldreport besser bedient (Nach "Chrilly's Monatlicher Goldreport" googeln).P.S.: Das Wort Akademie bezieht sich auf den Treffpunkt von Platos Gelehrtenkreis. Gerade Plato hat aber grossen Wert auf die Lesbarkeit seiner Werke gelegt und viele seiner Gedanken in Dialogform präsentiert.

I've read various accounts of Tetlock's research and figured I should read the original. It has been a bit of a chore. If you aren't comfortable reading the results of factor analyses, you may be better off reading some of the more accessible authors who summarize Tetlock's work. The research is fascinating, but its presentation isn't aimed at the casual reader.

Even for those who're not interested in politics this is a very well written book. The structure of the argumentation in itself is very interesting and can be applied to other arguments. This is one of the books I'd buy for others if I knew their interests aligned enough.

To specific for an ordinary reader to get the hang of it. Utterly hard to read, not interesting for the layman.