Proving a null hypothesis?

Steve Stigler points to an unusual example of research being used to provide evidence that the null hypothesis is true.

Here’s the research article (“No evidence for magnetic field effects on the behaviour of Drosophila,” by Marco Bassetto, Thomas Reichl, Dmitry Kobylkov, Daniel Kattnig, Michael Winklhofer, P. J. Hore, and Henrik Mouritsen), and here’s the quick summary (“Doubt cast on magnetic sensing in flies,” by Eric Warrant).

From the article:

Under meticulously controlled conditions and with vast sample sizes, we have been unable to find evidence for magnetically sensitive behaviour in Drosophila. Moreover, after reassessment of the statistical approaches and sample sizes used in the studies that we tried to replicate, we suggest that many—if not all—of the original results were false positives. Our findings therefore cast considerable doubt on the existence of magnetic sensing in Drosophila and thus strongly suggest that night-migratory songbirds remain the organism of choice for elucidating the mechanism of light-dependent magnetoreception.

From the summary:

But do the authors definitively debunk the existence of a magnetic sense in Drosophila? Possibly, although there are now at least 15 publications reporting that this sense does exist, with many indicating a Cry-based mechanism. Can all of them be wrong? Again, possibly — and for similar reasons — but this is a serious call to make. . . . Nonetheless, Bassetto et al. have raised a major red flag over the likelihood of Drosophila having the capacity for magnetic sensing. . . .

Here are some interesting things:

1. As can be seen from the abstract of the article, the scientific models here are not just speculations (if we try X, perhaps Y will happen!); they are attempts to explain an empirical puzzle, in this case regarding animals’ navigation.

2. As is typically the case, the null hypothesis of zero effect includes a fuzzy zone of effects that are not exactly zero but show no predictable pattern.

3. With enough data, it should still be possible to reject the statistical null hypothesis, as real data will never match any “specific random number generator”.

4. The summary asks, “do the authors definitively debunk the existence . . .” You can never prove a negative (a concept that seems to have been beyond the capacity of the U.S. judiciary to understand), so I think this question is pretty much pointless.

5. The summary asks whether “at least 15 publications” can be wrong. Hey—check out out the literature on embodied cognition. Or nudging. Hey, Brian Wansink alone had more than 15 publications! I don’t really like the phrase, “this is a serious call to make,” but, sure, future work is warranted etc.

Anyway, this is a good example to point to when we want to talk about a point null hypothesis.