Posterior SBC: Simulation-Based Calibration Checking Conditional on Data

If you know simulation based calibration checking (SBC), you will enjoy our new paper Posterior SBC: Simulation-Based Calibration Checking Conditional on Data with Teemu Säilynoja, Marvin Schmitt, Paul Bürkner, and Aki Vehtari

The original SBC checks whether the inference works for all possible data sets generated using the model and parameter draws from the prior. Priors are usually wider than posteriors and may contain regions where the computation fails.

For example, for hierarchical models, MCMC can have problems either with centered or non-centered parameterization depending on the data. Given one of the parameterizations, prior SBC observes both failing and non-failing inference. Posterior SBC focuses on the posterior conditional on the data, and can assess which parameterization works better for that specific data.

We illustrate with a hierarchical normal and a Lotka-Volterra models using MCMC, and a drift diffusion model using amortized Bayesian inference. Posterior SBC is specifically useful for amortized inference, as the repeated inference has negligible cost.