Validation of Software for Bayesian Models Using Posterior Quantiles

Every once in awhile I get a question that I can directly answer from my published research. When that happens it makes me so happy.

Here’s an example. Patrick Lam wrote,

Suppose one develops a Bayesian model to estimate a parameter theta. Now suppose one wants to evaluate the model via simulation by generating fake data where you know the value of theta and see how well you recover theta with your model, assuming that you use the posterior mean as the estimate. The traditional frequentist way of evaluating it might be to generate many datasets and see how well your estimator performs each time in terms of unbiasedness or mean squared error or something. But given that unbiasedness means nothing to a Bayesian and there is no repeated sampling interpretation in a Bayesian model, how would you suggest one would evaluate a Bayesian model?

My reply:

I actually have a paper on this! It is by Cook, Gelman, and Rubin. The idea is to draw theta from the prior distribution. You can find the paper in the published papers section on my website.

P.S. Although unbiasedness doesn’t mean much to a Bayesian, calibration does.

We’re planning on implementing this in Stan at some point.

The post Validation of Software for Bayesian Models Using Posterior Quantiles appeared first on Statistical Modeling, Causal Inference, and Social Science.