Your Favorite DSGE Sucks

Attention conservation notice: 1800+ words of academic self-promotion, boosting a paper in which statisticians say mean things about some economists' favored toys. They're not even peer-reviewed mean things (yet). Contains abundant unexplained jargon, and cringe-worthy humor on the level of using a decades-old reference for a title. Entirely seriously: Daniel is in no way responsible for this post.

I am very happy that after many years, this preprint is loosed upon the world:

Daniel J. McDonald and CRS, "Empirical Macroeconomics and DSGE Modeling in Statistical Perspective", arxiv:2210.16224

Abstract: Dynamic stochastic general equilibrium (DSGE) models have been an ubiquitous, and controversial, part of macroeconomics for decades. In this paper, we approach DSGEs purely as statstical models. We do this by applying two common model validation checks to the canonical Smets and Wouters 2007 DSGE: (1) we simulate the model and see how well it can be estimated from its own simulation output, and (2) we see how well it can seem to fit nonsense data. We find that (1) even with centuries' worth of data, the model remains poorly estimated, and (2) when we swap series at random, so that (e.g.) what the model gets as the inflation rate is really hours worked, what it gets as hours worked is really investment, etc., the fit is often only slightly impaired, and in a large percentage of cases actually improves (even out of sample). Taken together, these findings cast serious doubt on the meaningfulness of parameter estimates for this DSGE, and on whether this specification represents anything structural about the economy. Constructively, our approaches can be used for model validation by anyone working with macroeconomic time series.

To expand a little: DSGE models are models of macroeconomic aggregate quantities, like levels of unemployment and production in a national economy. As economic models, they're a sort of origin story for where the data comes from. Some people find DSGE-style origin stories completely compelling, others think they reach truly mythic levels of absurdity, with very little in between. While settling that is something I will leave to the professional economists (cough obviously they're absurd myths cough), we can also view them as statistical models, specifically multivariate time series models, and ask about their properties as such.

Now, long enough ago that blogging was still a thing and Daniel was doing his dissertation on statistical learning for time series with Mark Schervish and myself, he convinced us that DSGEs were an interesting and important target for the theory we were working on. One important question within that was trying to figure out just how flexible these models really were. The standard learning-theoretic principle is that the more flexible model classes learn slower than less flexible ones. (If you are willing and able to reproduce really complicated patterns, it's hard for you to distinguish between signal and noise in limited data. There are important qualifications to this idea, but it's a good start.) We thus began by thinking about trying to get the DSGEs to fit random binary noise, because that'd tell us about their Rademacher complexity, but that seemed unlikely to go well. That led to thinking about trying to get the models to fit the original time series, but with the series randomly scrambled, a sort of permutation test of just how flexible the models were.

At some point, one of us had the idea of leaving the internal order of each time series alone, but swapping the labels on the series. If you have a merely-statistical multivariate model, like a vector autoregression, the different variables are so to speak exchangeable --- if you swap series 1 and series 2, you'll get a different coefficient matrix out, but it'll be a permutation of the original. (The parameters will be "covariant" with the permutations.) It'll fit as well as the original order of the variables. But if you have a properly scientific, structural model, each variable will have its own meaning and its own role in the model, and swapping variables around should lead to nonsense, and grossly degraded fits. (Good luck telling the Lotka-Volterra model that hares are predators and lynxes are prey.) There might be a few weird symmetries of some models which leave the fit alone (*), but for the most part, randomly swapping variables around should lead to drastically worse fits, if your models really are stru