Memo to Reinhart and Rogoff: I think it’s best to admit your errors and go on from there

Jeff Ratto points me to this news article by Dean Baker reporting the work of three economists, Thomas Herndon, Michael Ash, and Robert Pollin, who found errors in a much-cited article by Carmen Reinhart and Kenneth Rogoff analyzing historical statistics of economic growth and public debt. Mike Konczal provides a clear summary; that’s where I got the above image.

Errors in data processing and data analysis

It turns out that Reinhart and Rogoff flubbed it. Herndon et al. write of “spreadsheet errors, omission of available data, weighting, and transcription.” The spreadsheet errors are the most embarrassing, but the other choices in data analysis seem pretty bad too. It can be tough to work with small datasets, so I have sympathy for Reinhart and Rogoff, but it does look like they were jumping to conclusions in their paper. Perhaps the urgency of the topic moved them to publish as fast as possible rather than carefully considering the impact of their data-analytic choices.

A disappointing response from Reinhart and Rogoff

Reinhart and Rogoff posted a response to the critique.

On the methodology, they do not deny any of the data and methodological errors pointed out by Herndon et al. I don’t know if that means they’ve known for awhile about these problems, or if they quickly checked and realized that, yeah, they’d screwed up the spreadsheets and, yeah, they did some funky averaging.

But they don’t admit anything either. As always, I find that sort of behavior annoying. It’s a step forward that they’re not denying it, but to not admit it—that’s just tacky.

And I speak as someone who’s made serious data errors myself, most notably in this retracted paper (background here).

In another case, I dodged a bullet by detecting a data error shortly before publication. (The story: “the problem was in the name of one party (the Popular Party)—it had an extra comma in its name and when we read in the data, we mistakenly counted it as a different party.”)

In yet another case, I posted (just on the blog, not in a publication) a set of maps that had problems, not from bad data but from poor assumptions in my statistical analysis (which, for me, is even more embarrassing). The criticisms came from an angry and somewhat uninformed blogger—but many of his points were correct! It took me some months to redo the analysis in a more reasonable way. But, before I did that, I engaged with the critics. I wrote things like:

I appreciate the careful scrutiny. One of my pet peeves is people assuming a number or graph is correct, just because it has been asserted. BorisG and Kos and others are doing a useful service by subjecting my maps to criticism.

Kos was not polite to me. But that’s not the point. Science isn’t about being polite. Herndon, Ash, and Pollin weren’t polite to Reinhart and Rogoff. So what.

I also wrote:

Because of the small sample size, I couldn’t just take the raw numbers. But I’m wiling to believe there are problems with the model, and I’m amenable to working to improve it.

and

I agree that the sharp color change of the map can make things look more dramatic than they really are.

and

Kos is right––there’s something wrong with my New Hamphire numbers. McCain only won 45% of the vote in New Hampshire, and the state is nearly 100% white, so clearly I am mistaken in giving him 50% of the vote in each income category. My guess as to what is happening here is that, with such a small sample size in the state, the model shifted the estimates over to what was happening in other states. This wasn’t a problem in my map of all voters, because i adjusted the total to the actual vote, but for the map of just whites it was a problem because my model didn’t “know” that New Hampshire was nearly all white. In the fuller model currently being fit by Yair [i.e., our recent AJPS paper!], this problem will be solved, because we’ll be averaging over population categories within each state.

In the meantime, though, yeah, I should’ve realized New Hampshire had a problem.

So, yeah, I think Reinhart and Rogoff should admit they have a problem too.

On to the substance

I don’t know anything about macroeconomics so all I can report here is what was written in the recent exchange. Here’s Reinhart and Rogoff defending themselves:

It is hard to see how one can interpret these [corrected] tables and individual country results as showing that public debt overhang over 90% is clearly benign.

But nobody claimed that! The words “clearly benign” are from Reinhart and Rogoff. What Herndon et al. actually wrote was:

The full extent of those errors transforms the reality of modestly diminished average GDP growth rates for countries carrying high levels of public debt into a false image that high public debt ratios inevitably entail sharp declines in GDP growth. Moreover, as we show, there is a wide range of GDP growth performances at every level of public debt among the 20 advanced economies that RR survey.

Herndon et al. don’t say that high debt is “clearly benign,” they say that it’s not inevitably bad. This is quite different. Reinhart and Rogoff have a real quantitative point to make—the point is that “modestly diminished average GDP growth rates” can be a big deal. “Modest” sounds like no big whoop but it can matter a lot when you’re talking about a big economy. But, hey, don’t make a straw man and say that they’re talking about “clearly benign.” What’s the point of that??

Uh oh

Between this and the notorious “Out of Africa” paper, it hasn’t been such a good year for the American Economic Review. And it’s still only April! This is almost as embarrassing as when Statistical Science published that Bible Code paper back in 1994.

Here’s more from Tyler Cowen and Paul Krugman. I haven’t looked at the data myself so I can’t offer any informed statistical commentary, but not a single person in the discussion seems to be denying that Reinhart and Rogoff messed up with their data, so I assume this actually happened. Which makes it even seem even more awkward that they didn’t directly admit it.

In summary

Reinhart and Rogoff may have a point that the corrections to their analyses have little practical import, merely reducing the magnitude of their claim without changing its sign. Maybe Konczal is overstating it when he writes, “one of the core empirical points providing the intellectual foundation for the global move to austerity in the early 2010s was based on someone accidentally not updating a row formula in Excel.” I have no informed opinion on that at all. Really. I’m not being polite and circumspect, I know nothing about this stuff.

But unless they want to enter the competition for the lamest, grudgingest, non-retraction retraction ever, I recommend they start by admitting their error and then going on from there. I think they should also thank Herndon, Ash, and Pollin for finding multiple errors in their paper. Admit it and move forward. (That’s something we’ve been saying a lot here lately.)

P.S. But I don’t want to be too hard on Reinhart and Rogoff. As a statistician, I’m acutely aware of the potential for error in myself as well as others, but I don’t think researchers in other fields get this sort of training. And of course I have decades of experience making research errors of different degrees of seriousness. Reinhart and Rogoff may well have been closer to error-free in the past. Also, this is a high-profile case and their immediate response was defensive. Had this happened to me, I might have reacted the same way (especially if I didn’t already have many previous experiences of admitting error). I hope that, tomorrow or sometime soon, Reinhart and Rogoff will admit their mistakes, profusely thank their critics, and move on.