Informative priors for treatment effects

Biostatistician Garnett McMillan writes:

A PI recently completed a randomized trial where the experimental treatment showed a large, but not quite statistically significant (p=0.08) improvement over placebo. The investigators wanted to know how many additional subjects would be needed to achieve significance. This is a common question, which is very hard to answer for non-statistical audiences. Basically, I said we would need to conduct a new study.

I took the opportunity to demonstrate how a Bayesian analysis of these data using skeptical and enthusiastic priors on the treatment effect. I also showed how the posterior is conditional on accumulated data, and naturally lends itself to sequential analysis with additional recruitment. The investigators, not surprisingly, loved the Bayesian analysis because it gave them ‘hope’ that the experimental treatment might really help their patients.

Here is the problem: The investigators want to report BOTH the standard frequentist analysis AND the Bayesian analysis. In their mind the two analyses are simply two sides of the same coin. I have never seen this (outside of statistics journals), and have a hard time explaining how one reconciles statistical results where the definition of probability is so different. Do you have any help for me in explaining this problem to non-statisticians? Any useful metaphors or analogies?

My reply: I think it’s fine to consider the classical analysis as a special case of the Bayesian analysis under a uniform prior distribution. So in that way all the analyses can be presented on the same scale.

But I think what’s really important here is to think seriously about plausible effect sizes. It is not in general a good idea to take a noisy point estimate and use it as a prior. For example, suppose the study so far gives an estimated odds ratio of 2.0, with a (classical) 95% interval of (0.9, 4.4). I would not recommend a prior centered around 2. Indeed, the same sort of problem—or even worse—comes from taking previous published results as a prior. Published results are typically statistically significant and thus can grossly overestimate effect sizes.

Indeed, my usual problem with the classical estimate, or its Bayesian interpretation, is with the uniform prior, which includes all sorts of unrealistically large treatment effects. Real treatment effects are usually small. So I’m guessing that with a realistic prior, estimates will be pulled toward zero.

On the other hand, I don’t see the need for requiring 95% confidence. We have to make our decisions in the meantime.

Regarding the question of whether the treatment helps the patients: I can’t say more without context, but in many settings we can suppose that the treatment is helping some people and hurting others, so I think it makes sense to consider these tradeoffs.

The post Informative priors for treatment effects appeared first on Statistical Modeling, Causal Inference, and Social Science.