Using simulation from the null hypothesis to study statistical artifacts (ivermectin edition)

Robin Mills, Ana Carolina Peçanha Antonio, and Greg Tucker-Kellogg write:

Background

Two recent publications by Kerr et al. reported dramatic effects of prophylactic ivermectin use for both prevention of COVID-19 and reduction of COVID-19-related hospitalisation and mortality, including a dose-dependent effect of ivermectin prophylaxis. These papers have gained an unusually large public influence: they were incorporated into debates around COVID-19 policies and may have contributed to decreased trust in vaccine efficacy and public health authorities more broadly. . . .

Methods

Starting with initially identified sources of error, we conducted a revised statistical analysis of available data, including data made available with the original papers and public data from the Brazil Ministry of Health. We identified additional uncorrected sources of bias and errors from the original analysis, including incorrect subject exclusion and missing subjects, an enrolment time bias, and multiple sources of immortal time bias. . . .

Conclusions

The inference of ivermectin efficacy reported in both papers is unsupported, as the observed effects are entirely explained by untreated statistical artefacts and methodological errors.

I guess that at this point ivermectin is over (see also here and here); still, just as it’s always good to do good science, even if the results are not surprising, it’s also always good to do good science criticism. From a methods point of view, this new paper by Mills et al. has a pleasant discussion of the value of simulation from the null hypothesis as a way to learn the extent of statistical artifacts; see discussion on page 12 of the above-linked paper.

I say all this in general terms, as I have not read the article in detail. The authors thank me in their acknowledgments so I must have helped them out at some point, but it’s been awhile and now I don’t remember what I actually did!

P.S. “Using simulation from the null hypothesis to study statistical artifacts” is another way of saying “hypothesis testing.”