Home Statistics and Visualization Statistical Modeling, Causal Inference, and Social Science Signal-to-noise ratio and statistical tests

Signal-to-noise ratio and statistical tests

Statistical Modeling, Causal Inference, and Social Science 2022-12-19

Thomas Langkamp writes:

I posted a question on the binomial test on the cross-validated forum. Surprisingly nombody seems to have an answer or even a suggestion. Maybe you can share your thoughts?

Here’s Langkamp’s question:

How to calculate the Signal-to-Noise-Ratio (SNR) for the classic / frequentist binomial test? For example for the z-Test of proportions the empirical z-value or SNR is:

SNR = z = Signal / Noise = Cohens g / Standard error of proportions = (p – p0) / Square Root of (p · (1 – p) / n) with p as the observed proportion and with p0 as the reference value, sometimes called Pi. with n as the number of observations

So can I use this SNR also for the Binomial Test if n is small and or if p does not equal 0.5? If so, why do e.g. SPSS or JASP do not report this value? At least for big n it would make sense to report it, as the binomial distribution aproaches the z-distribution.

And here’s my reply:

There’s no precise definition of signal-to-noise ratio, but the way that I think about it, the numerator (the size of the “signal”) should represent some population of effects, not the effect in a single study. Trying to make design calculations using the effect in a single study creates all sorts of problems of the sort discussed here. So instead of using p-p0 in the numerator, I’d use an average absolute value of effect size, or root mean squared effect size, or something like that, in a hypothetical population of studies. You won’t know this value, but I’ll just say that it’s probably much smaller than you think it is.

And that brings us to the answer to the question, why doesn’t statistical software report the signal-to-noise ratio? It’s because this ratio depends on the numerator, which you don’t know—and for studies where statistics is relevant, you can’t estimate it well from the available data either. So best to just report the estimate and standard error, and if you want to take an estimated “signal” from somewhere and make a signal-to-noise ratio calculation, you can do so.