“My quick answer is that I don’t care much about permutation tests because they are testing a null null hypothesis that is of typically no interest”

Riley DeHaan writes:

I’m a psych PhD student and I have a statistical question that’s been bothering me for some time and wondered if you’d have any thoughts you might be willing to share.

I’ve come across some papers employing z-scores of permutation null distributions as a primary metric in neuroscience (for an example, see here).

The authors computed a coefficient of interest in a multiple linear regression and then permuted the order of the predictors to obtain a permutation null distribution of that coefficient. “The true coefficient for functional connectivity is compared to the distribution of null coefficients to obtain a z-score and P-value.” The authors employed this permutation testing approach to avoid the need to model potentially complicated autocorrelations between the observations in their sample and then wanted a statistic that provided a measure of effect size rather than relying solely on p-values.

Is there any meaningful interpretation of a z-score of a permutation null distribution under the alternative hypothesis? Is this a commonly used approach? This approach would not appear to find meaningfully normalized estimates of effect size given the variability of the permutation null distribution may not have anything to do with the variance of the statistic of interest under its own distribution. In this case, I’m not sure a z-score based on the permutation null provides much information beyond significance. The variability of the permutation null distribution will also be a function of the sample size in this case. Could we argue that permutation null distributions would in many cases (I’m thinking about simple differences in means rather than regression coefficients) tend to overestimate the variability of the true statistic given permutation tests are conservative compared to tests based on known distributions of the statistic of interest? This z-score approach would then tend to produce conservative effect sizes. I’m not finding references to this approach online beyond this R package.

My reply: My quick answer is that I don’t care much about permutation tests because they are testing a null null hypothesis that is of typically no interest. Related thoughts are here.

P.S. If you, the reader of this blog, care about permutation tests, that’s fine! Permutation tests have a direct mathematical interpretation. They just don’t interest me, that’s all.