Why Media "Experts" Don't Make Useful Predictions

Famous pundits virtually never make falsifiable forecasts.

Read more at BigThink.com: http://bigthink.com/videos/philip-tetlock-on-superforecasters-and-tv-pundits

Follow Big Think here:
YouTube: http://goo.gl/CPTsV5
Facebook: https://www.facebook.com/BigThinkdotcom
Twitter: https://twitter.com/bigthink

Transcript - We know virtually nothing about the forecasting track records of famous pundits because famous pundits virtually never make falsifiable forecasts. They say something might happen or could happen or may very well happen, but when I say something could happen that doesn't mean a lot. I mean it could be that it could happen that we're all going to be vaporized by an asteroid in the next 24 hours, or it could be that the sun will rise tomorrow. It could subsume an enormous range of possible probabilities. So if I say something could happen and it does happen, I can say to my readers well, you know, I told you it could. And if I say something could happen and it doesn't happen, I can come back to my readers and say, I just said it could. One of the interesting things about superforecasters is how opportunistic they are. Superforecasters think quite strategically about when it makes sense to invest effort in thinking. So if you wanted to predict the outcome of the presidential election in early October 2015, 2016 presidential election, how would you go about it if you were a superforecaster? A superforecaster wouldn't look carefully at the presidential debates, look into the eyes of the candidates and see which one looks more presidential. At some point in the process a superforecaster might do that, but a superforecaster would tend to start with more the outside view and gradually work in rather than start at the inside and work out. So they would ask very general questions initially like let's look at elections, all presidential elections since World War II, how likely is a democrat or a republican to win? Or they might say after one party has held the presidency for two terms, how likely is there to be a transition? Or they might say if economic growth is less than two percent three quarters before the presidential election, does that bode ill for the party that controls the presidency?

So they would start off with these more general estimates, these more outside view estimates and then they would gradually adjust in response to estimates about popularity polls of candidates, which are notoriously volatile at this stage of the process. But they would take them into account but they would discount them considerably because they are so volatile and then they would adjust incrementally. So the best forecasters I think, over the course of 2015, have somewhat lowered their probability estimate of Hillary Clinton being the next president of the United States. I think they started off significantly above 50 percent and there's been some significant hemorrhaging of those estimates, but nothing all that dramatic. I mean the interesting thing about superforecasters is they're very patience and they make granular belief updates. So they don't suddenly say oh my God the latest email scandal or is Biden going to come into the race or this or that, there are these little clues and it's not that they ignore them but they tend to respond very incrementally to them. So it may be the probability of Hillary Clinton becoming the next president of the United States moves from the 45 percent to 43 percent in response to an Inspector General Report in the State Department, that sort of thing. The best forecasters tend not to make it on to television. They’re not very attracted to the TV producers because they’re much more likely to say on the one hand, on the other hand.