“I work in a biology lab . . . My PI proposed a statistical test that I think is nonsense. . .”

This came in the email:

I work in a biology lab . . . My PI proposed a statistical test that I think is nonsense. If you have spare time I was wondering if you could look it over and tell me if my concerns are well founded.

I describe the test in an attached pdf, I wrote it up in LaTeX for readability. I wish it was shorter, I’m sorry, I did my best not to ramble.

To be honest, I am having a lot of arguments in the lab about statistical rigor, and I am wondering if you have advice about when it’s time to move on. But, this email is already long enough, there is not enough space to discuss my other concerns / vent about my frustrations.

For obv reasons I’ve anonymized the details, which is kinda too bad because the actual work looks super interesting; it has to do with modeling deformations of shapes, which is one of my favorite topics. In any case, I did not have the energy to try to understand everything that was going on, so I’ll just offer three general comments on hypothesis testing, checking the properties of a statistical method, and communicating to your supervisor:

1. The question at hand involved a permutation test. I generally don’t think permutation tests make much sense, as they are testing a null hypothesis of zero effect that is not particularly interesting, and the permutation typically corresponds to an unrealistic design; see section 3.3 of this paper.

2. A question arose about how a particular statistical method would perform: can certain effects of interest be separately estimated from available experimental data? Advice from the literature weren’t clear. My general recommendation here is that if you are skeptical of a statistical method and you do not think it should work well, one way to demonstrate or check this is to simulate fake data from the model and various alternatives and see how the procedure works. I think this should be more convincing than a theoretical argument.

3. Finally, what to do if your supervisor suggests a method that you think is nonsense? If you don’t have a great relationship to your supervisor, so that you can’t just say, “Hey, I think this method is nonsense!”, then my recommendation is to throw the burden of proof back in the other direction. If your lab publishes a paper using a particular statistical method, it’s the job of the authors of that paper to make a convincing case for why this method is appropriate for the job it is being asked to do. So when talking with your supervisor, you can ask why the method is being used, and you can express concern that you’re not sure the paper will be accepted by the journal, given various objections the reviewers might raise.