“My basic question is do we really need data to be analysed by both methods?”

Ram Bajpai writes:

I’m an early career researcher in medical statistics with keen interest in meta-analysis (including Bayesian meta-analysis) and prognostic modeling. I’m conducting a methodological systematic review of Bayesian meta-analysis in the biomedical research. After reading these studies, many authors presented both Bayesian and classical results together and comparing them and usually say both methods provide similar results (trying to validate). However, being a statistician, I don’t see any point of analysing data from both techniques as these are two different philosophies and either one is sufficient if well planned and executed. Consider me no Bayesian expert, I seek your guidance on this issue. My basic question is do we really need data to be analysed by both methods?

My quick answer is that I think more in terms of methods than philosophies. Often a classical method is interpretable as a Bayesian method with a certain prior. This sort of insight can be useful. From the other direction, the frequency properties of a Bayesian method can be evaluated as if it were a classical procedure.

This also reminds me of a discussion I had yesterday with Aaditya Ramdas at CMU. Ramdas has done a lot of theoretical work on null hypothesis significance testing; I’ve done lots of applied and methodological work using Bayesian inference. Ramdas expressed the view that it is a bad thing that there are deep philosophical divisions in statistical regarding how to approach even the simplest problems. I replied that I didn’t see a deep philosophical divide between Bayesian inference and classical significance testing. To me, the differences are in what assumptions we are willing to swallow.

My take on statistical philosophy is that all statistical methods require assumptions that are almost always clearly false. Hypothesis testing is all about assumptions that something is exactly zero, which does not make sense in any problem I’ve studied. If you bring this up with people who work on or use hypothesis testing, they’ll say something along the lines of, Yeah, yeah, sure, I know, but it’s a reasonable approximation and we can alter the assumption when we need to. Bayesian inference relies assumptions such as normality and logistic curves. If you bring this up with people who work on or use Bayesian inference, they’ll say something along the lines of, Yeah, yeah, sure, I know, but it’s a reasonable approximation and we can alter the assumption when we need to. To me, what appears to be different philosophies are more like different sorts of assumptions that people are comfortable with. It’s not just a “matter of taste”—different methods work better for different problems, and, as Rob Kass says, the methods that you use will, and should, be influenced by the problems you work on—I just think it makes more sense to focus on differences in methods and assumptions rather than frame as incommensurable philosophies. I do think philosophical understanding, and misunderstanding, can make a difference in applied work—see section 7 of my paper with Shalizi.