The AAA Tranche of Subprime Science

In our new ethics column for Chance, Eric Loken and I write about our current favorite topic:

One of our ongoing themes when discussing scientific ethics is the central role of statistics in recognizing and communicating uncer- tainty. Unfortunately, statistics—and the scientific process more generally—often seems to be used more as a way of laundering uncertainty, processing data until researchers and consumers of research can feel safe acting as if various scientific hypotheses are unquestionably true. . . .

We have in mind an analogy with the notorious AAA-class bonds created during the mid-2000s that led to the subprime mortgage crisis. Lower-quality mortgages—that is, mortgages with high probability of default and, thus, high uncertainty—were packaged and transformed into financial instruments that were (in retrospect, falsely) characterized as low risk. There was a tremendous interest in these securities, not just among the most unscrupulous market manipulators, but in a world where a lot of money was looking for safe investments and investors were willing to believe the ratings agencies and brokers.

Similarly, the concerns about reliability and validity of published results come after years of rapid expansion in the world of scientific output. In published research studies, data of varying quality are thrown together, processed, and analyzed and formed into statistically significant aggregates that are combined into research papers. . . .

How Is a Research Paper Like a Mortgage?

The analogy is anything but exact, but we see two equivalents in the modern scientific process to the aggregation and skimming that led to tranches of mortgages being declared AAA (high-quality) bonds. The first step is statistical significance. Out of the primordial soup of all possible data analyses, the statistically significant comparisons float to the top. . . . he second step is publication in a scientific journal, ideally a high-prestige outlet . . . but, if not at a top journal, any outlet will do. The convention is to treat published claims as true unless demonstrated otherwise.

By analogizing to the mortgage crisis, are we saying all research is over-valued? Not at all. Nor were all subprime mortgages destined to fail. The problem happened when reliable and less reliable components were bundled into a AAA tranche. The analogy might be to believe all papers published in Nature just because they are published in a top journal, or to believe the results of all published medical trials that have randomization in their designs.

Lots more at the link. We were motivated to write this article after reflecting on the defensive reactions of various scientists regarding issues of scientific plausibility and replication; see, for example, here, here and here (the last of which gave rise to the Freshman Principle).

So, yes, we do think that this is a big issue. Here’s how Eric and I conclude our article:

We would be troubled to see a generalized skepticism of science take hold, in which even well-established findings are up for grabs. The reality is that we have to make personal and political decisions about health care, the environment, and economics—to name only a few areas—in the face of uncertainty and variation. It’s exactly because we have a tendency to think more categorically about things as being true or false, there or not there, that we need statistics. Quantitative research is our central tool for understanding variance and uncertainty and should not be used as a way to overstate confidence.

The post The AAA Tranche of Subprime Science appeared first on Statistical Modeling, Causal Inference, and Social Science.