“Why the New Pollution Literature is Credible” . . . but I’m still guessing that the effects are being overestimated:

In a post entitled, “Why the New Pollution Literature is Credible,” Alex Tabarrok writes:

My recent post, Air Pollution Reduces Health and Wealth drew some pushback in the comments, some justified, some not, on whether the results of these studies are not subject to p-hacking, forking gardens and the replication crisis. Sure, of course, some of them are. . . . Nevertheless, I don’t think that skepticism about the general thrust of the results is justified. Why not?

First . . . my rule is trust literatures not papers and the new pollution literature is showing consistent and significant negative effects of pollution on health and wealth. . . . It’s not just that the literature is large, however, it’s that the literature is consistent in a way that many studies in say social psychology were not. In social psychology, for example, there were many tests of entirely different hypotheses—power posing, priming, stereotype threat—and most of these failed to replicate. But in the pollution literature we have many tests of the same hypotheses. We have, for example, studies showing that pollution reduces the quality of chess moves in high-stakes matches, that it reduces worker productivity in Chinese call-centers, and that it reduces test scores in American and in British schools. . . . from different researchers studying different times and places using different methods but they are all testing the same hypothesis, namely that pollution reduces cognitive ability. . . .

Another feature in favor of the air pollution literature is that the hypothesis that pollution can have negative effects on health and cognition wasn’t invented yesterday . . . The Romans, for example, noted the negative effect of air pollution on health. There’s a reason why people with lung disease move to the countryside and always have.

I also noted in Why Most Published Research Findings are False that multiple sources and types of evidence are desirable. The pollution literature satisfies this desideratum. Aside from multiple empirical studies, the pollution hypothesis is also consistent with plausible mechanisms . . .

Moreover, there is a clear dose-response effect–so much so that when it comes to “extreme” pollution few people doubt the hypothesis. Does anyone doubt, for example, that an infant born in Delhi, India–one of the most polluted cities in the world–is more likely to die young than if the same infant grew up (all else equal) in Wellington, New Zealand–one of the least polluted cities in the world? . . .

What is new about the new pollution literature is more credible methods and bigger data and what the literature shows is that the effects of pollution are larger than we thought at lower levels than we thought. But we should expect to find smaller effects with better methods and bigger data. . . . this isn’t guaranteed, there could be positive effects of pollution at lower levels, but it isn’t surprising that what we are seeing so far is negative effects at levels previously considered acceptable.

Thus, while I have no doubt that some of the papers in the new pollution literature are in error, I also think that the large number of high quality papers from different times and places which are broadly consistent with one another and also consistent with what we know about human physiology and particulate matter and also consistent with the literature on the effects of pollution on animals and plants and also consistent with a dose-response relationship suggest that we take this literature and its conclusion that air pollution has significant negative effects on health and wealth very seriously.

This all makes a lot of sense—enough so that I quoted large chunks of Tabarrok’s post.

Still, I think actual effects will be quite a bit lower than claimed in the literature. Yes, it’s appropriate to look at the literature, not just individual studies. But if each individual study is biased, that will bias the literature. You can think of Alex’s two posts on the effects of air pollution as a sort of informal meta-analysis, and it’s a meta-analysis that does not correct for selection bias within each published study. Again, his general points (both methodologically and with regard to air pollution in particular) make sense; I just think there’s a bias he’s not correcting for. When we talk about forking paths etc. it’s typically in settings where there’s essentially zero signal (more precisely, settings where any signal is overwhelmed by noise) and people are finding patterns out of nothing—but the same general errors can lead to real effects being greatly overestimated.

P.S. The comments to Tabarrok’s post are pretty wack. Some reasonable points but lots and lots of people just overwhelmed by political ideology.