“Accounting for Nonresponse in Election Polls: Total Margin of Error”
Statistical Modeling, Causal Inference, and Social Science 2024-12-20
Jeff Dominitz and Chuck Manski write:
The potential impact of nonresponse on election polls is well known and frequently acknowledged. Yet measurement and reporting of polling error has focused solely on sampling error, represented by the margin of error of a poll. Survey statisticians have long recommended measurement of the total survey error of a sample estimate by its mean square error (MSE), which jointly measures sampling and non-sampling errors. Extending the conventional language of polling, we think it reasonable to use the square root of maximum MSE to measure the total margin of error. This paper demonstrates how to measure the potential impact of nonresponse using the concept of the total margin of error, which we argue should be a standard feature in the reporting of election poll results. We first show how to jointly measure statistical imprecision and response bias when a pollster lacks any knowledge of the candidate preferences of non-responders. We then extend the analysis to settings where the pollster has partial knowledge that bounds the preferences of non-responders.
Good stuff. This relates to two of my papers on nonsampling errors and differential nonresponse:
[2018] Disentangling bias and variance in election polls. Journal of the American Statistical Association 113, 607-614. (Houshmand Shirani-Mehr, David Rothschild, Sharad Goel, and Andrew Gelman)
[1998] Modeling differential nonresponse in sample surveys. Sankhya 60, 101-126. (Thomas C. Little and Andrew Gelman)
It’s funny about that last paper because Tom Little and I did that project nearly 30 years ago and I forgot about it even while doing applied research on differential nonresponse. I think there’s more to be done here, integrating the different perspectives here. I like the idea of modeling the nonsampling error rather than just treating it as another error term.
On the general point of reporting surveys and margins of errors, I recommend Ansolabehere and Belin’s paper from 1993.