“Heterogeneity of variance in experimental studies: A challenge to conventional interpretations”

Avi sent along this old paper from Bryk and Raudenbush, who write:

The presence of heterogeneity of variance across groups indicates that the standard statistical model for treatment effects no longer applies. Specifically, the assumption that treatments add a constant to each subject’s development fails. An alternative model is required to represent how treatment effects are distributed across individuals. We develop in this article a simple statistical model to demonstrate the link between heterogeneity of variance and random treatment effects. Next, we illustrate with results from two previously published studies how a failure to recognize the substantive importance of heterogeneity of variance obscured significant results present in these data. The article concludes with a review and synthesis of techniques for modeling variances. Although these methods have been well established in the statistical literature, they are not widely known by social and behavioral scientists.

This is really important. I’ll have to think about whether we can do more on this, following the lead of my article on treatment effects in before-after data, which made a very similar point but in a less systematic way. Connections between interactions, varying treatment effects, and multilevel models. A really big deal if we can put it together in the right way.

The post “Heterogeneity of variance in experimental studies: A challenge to conventional interpretations” appeared first on Statistical Modeling, Causal Inference, and Social Science.