How to Be Bad at Forecasting

In his wonderful book Expert Political Judgment, psychologist Philip Tetlock (following the lead of Isaiah Berlin), divided the world of political forecasters into hedgehogs and foxes:

The intellectually aggressive hedgehogs knew one big thing and sought, under the banner of parsimony, to expand the explanatory power of that big thing to "cover" new cases; the more eclectic foxes knew many little things and were content to improvise ad hoc solutions to keep pace with a rapidly changing world.

These hedgehogs are of course the people more likely to be asked onto cable TV news shows to comment on world affairs. They're more likely to be famous, as measured by Google mentions. They are also much more likely, Tetlock found, to be wrong. Foxes see the world more clearly than hedgehogs.

By Tetlock and Berlin's taxonomy (not to mention by surname), I am a fox. So last year, when Tetlock and several other scholars announced that they were looking for participants in a forecasting tournament sponsored by the Intelligence Advanced Research Projects Activity (IARPA, or "Like DARPA, But for Spies" as Wired put it a few years ago), I signed right up. I read through all the lessons meant to make me a better, less cognitively biased forecaster, and started placing bets (using play money) on whether, say Muqtada al-Sadr would formally withdraw support for the current Iraqi government of Nouri al-Maliki by 30 September 2011.

I paid pretty close attention in the first few months, and did okay. That is, I got the overwhelming majority of the answers right, but I didn't place big enough bets to keep up with the leaders. I also ended up losing out on a few questions that I had tried to game. When IARPA asked about something that could happen at any time before a particular date (example: Will the Tunisian Ennahda party officially announce the formation of an interim coalition government by 15 November 2011?) I automatically picked yes, with plans to revisit closer to the deadline. But sometimes I forgot about the deadline.

As of January 19, I was ranked 123rd of 207 forecasters in my group. After that, though, things only got worse. I just didn't care enough to check the news and update my forecasts every day, or even every week. There wasn't a significant monetary reward, I was pretty busy with other stuff, and I clearly wasn't going to win. I ended up in 199th place out of 203 (there were dropouts).

Overall, Tetlock has reported that my group "collectively blew the lid off the performance expectations that IARPA had for the first year." Tetlock and crew are now rounding up the best forecasters to create teams of "super forecasters," with the aim of learning more about what makes someone good at predicting the future, and are looking to recruit new forecasters to replace losers like me. His provisional view, as expressed in an e-mail to economist/blogger Tyler Cowen, is that the best forecasters:

are distinguished by three characteristics: (1) an intense curiosity about the workings of the political-economic world; (2) an intense curiosity about the workings of the human mind; (3) cognitive crunching power ("fluid intelligence" and a capacity for "timely self correction").

So what distinguishes a bad forecaster? In my case, two things: (1) a discomfort with expressing my level of confidence with the size of my bets — this is a real flaw, perhaps traceable to the fact that I had never played a game of poker until two weeks ago; and (2) an almost complete lack of interest in the events being forecast. I think I'm pretty curious about the workings of the political-economic world. I just wasn't interested in whether the IMF would officially announce before 1 April 2012 that an agreement had been reached to lend Hungary an additional 15+ Billion Euros.

I'm not convinced that my lack of interest in such petty predictions (well, not petty for Hungarians. But I'm not Hungarian) is really a bad thing. In 2007, futurist Paul Saffo wrote in HBR that:

Prediction is concerned with future certainty; forecasting looks at how hidden currents in the present signal possible changes in direction for companies, societies, or the world at large. Thus, the primary goal of forecasting is to identify the full range of possibilities, not a limited set of illusory certainties.

Yeah, take that, you "super forecasters"! This can of course be a self-serving argument: When Saffo says to judge him not on whether his forecasts come true but on his logic, he's effectively removing a hard performance metric and replacing it with a soft one. But he's right that for many purposes in business and other fields, having a sense of the range of possibilities is more valuable than an explicit prediction.

Still, I don't think I cared about even the range of possible outcomes in the IMF's negotiations with Hungary. Which raises an important point. Hedgehogs who are obsessively focused on a particular theory of how the world works aren't very good at forecasting. But foxes who don't care aren't very good at it either. The best forecasters would appear to be foxes who really really want to win the game of forecasting. To quote Saffo again, the key is to "hold strong opinions weakly." Don't be stuck in your views; be willing to revise them quickly when new information comes in. But have bold views, or don't bother trying to make forecasts.