No, an election forecast that’s 50-50 is not “giving up.” No, the election forecasters in 2024 did not say, “Whatever happened, it was supposed to be razor thin.”

tl;dr. In 2024, pre-election polls were off by about 2 percentage points. Election forecasters were aware of this and incorporated this into their uncertainties. But if you were not following the forecasts carefully, you might’ve not realized this. You might have mistakenly thought that, when predictions happened to be close to 50-50, the forecasters “gave up.” You might have mistakenly thought that “Whatever happened, it was supposed to be razor thin.” I guess news reports should be emphasizing forecast uncertainties even more! None of this means that polls are “useless” or that “the survey industry needs a tear-down.”

tl;dr of the tl;dr. Surveys are pretty good. They’re not perfect. The imperfections—nonsampling error—represent real difficulties. The lack of easy answers doesn’t mean that pollsters are chumps.

OK, this is just annoying. Ben Recht and Leif Weatherby write:

2024 was the year the election forecasters gave up. On the Monday before the election, the New York Times polling average showed Donald Trump and Kamala Harris within one point of each other in 6 critical swing states. They put the final election popular vote prediction at 49-48 in favor of Harris. Effectively, a tie. Poll aggregator Real Clear Politics split the difference even finer, predicting the result 48.5-48.5. Poll forecaster Nate Silver put the probability of either candidate winning at exactly 50-50.

Yes, the forecast was highly uncertain. No, that’s not “giving up.” Let me introduce you to some sports bookies in Vegas. They give lines on every game. Sometimes the two teams are, to the best of all information, evenly matched, and the betting line will be even. That doesn’t mean the bookies “gave up”; it means that, their best estimate is that the two teams are equally likely to win. (OK, not quite, it’s really their best estimate of what it would take for equal amount of bets to come in each direction. The point is, they’re not “giving up”; they’re doing their best.

So, yeah, that’s wrong, for the same reason that it’s wrong to label the National Weather Service as “giving up” on days where they announce a 50% chance of rain.

Recht and Weatherby continue:

Whatever happened, it was supposed to be razor thin.

Again, no. Here’s Nate Silver, the best-known forecaster, a couple weeks before the election:

I [Nate] have a guest essay up at the New York Times with a fun headline: “Here’s What My Gut Says About the Election. But Don’t Trust Anyone’s Gut, Even Mine.” . . . Most of the column is about how Kamala Harris could beat her polls — or Donald Trump could beat his again. One thing that might be counterintuitive is that even a normal-sized polling error — polls are typically off by around 3 points in one direction or the other — could lead to one candidate sweeping all 7 key battleground states. . . . the baseline assumption of the Silver Bulletin model is that while the polls could be wrong again — and in fact, they probably will be wrong to some degree — it’s extremely hard to predict the direction of the error.

I’m not saying Nate’s always right. We’ve had our disagreements. I’m just saying that, no, he was not saying the election “was supposed to be razor thin.” The forecast electoral vote outcome was a distribution with expected value 50-50 but with a substantial variance.

And here’s Elliott Morris, who runs Fivethirtyeight.com, the best-known forecasting site:

Trump and Harris are both a normal polling error away from a blowout

The race is uncertain, but that doesn’t mean the outcome will be close.

As of Oct. 30 at 11:30 a.m. Eastern, the margin between Vice President Kamala Harris and Trump in 538’s polling averages is smaller than 4 points in seven states: the familiar septet of Arizona, Georgia, Michigan, Nevada, North Carolina, Pennsylvania and Wisconsin. That means that, if the polling error from 2020 repeats itself, Trump would win all seven swing states and 312 Electoral College votes. . . . In a scenario where the polls overestimate Trump’s margin by 4 points in every state, Harris would win all seven swing states and 319 electoral votes. . . .

Both of these outcomes — and everything in between — are very much on the table next week. . . . the model is expecting a roughly 2020-sized polling error — although not necessarily in the same direction as 2020. (In 50 percent of the model’s simulations, Trump beats his polls, and 50 percent of the time, Harris does.)

This point is worth dwelling on. Because our average expectation is for there to be a decently large polling error at least half of the time, there is actually a very low probability that the polls are perfect and the election plays out exactly how the polls suggest. . . . Polls are inherently uncertain. This is why we model. . . . this is the big, fundamental problem with preelection polling: We don’t know what the demographic and political composition of the actual electorate will be, so pollsters are just making the best guesses they can. Those guesses have always, and will always, come with error attached to them.

Here’s some dude who worked on the Economist election forecast:

Why forecast an election that’s too close to call? . . . I think the main value of forecasts is not in the predictions themselves, but in how they portray uncertainty and the stability of the race over time. . . . In the end, elections will always be uncertain, because it is up to the individual to decide how to vote, and whether to vote at all.

And here’s Harry Crane, a forecaster who did better than the name brands in 2024 by incorporating additional information on party registration and early voting:

The forecast makes Trump about a 2-to-1 favorite . . . based on an analysis of fundamental data, polls, and early voting data. This is more or less in line with other opinions out there, such as the betting markets and other forecasters. But because this forecast likes Trump a bit more than markets and a bit more than the other forecasters (who are favorable to Harris), it is inevitable that I will be called an idiot, or worse, should Harris pull it out. The same people will still call me an idiot if Trump wins, so what’s the difference.

The point is that every serious forecaster in 2024 understood—and vocally communicated to the world—that their forecasts were uncertain. Nobody thought Harris or Trump had much of a chance of getting 400 electoral votes, but everybody was saying that 300+ was a possibility. Even the forecasters who were loudly disagreeing with each other on social media agreed on this point, that the forecasts had a lot of uncertainty in the electoral vote.

Recht and Weatherby continue:

The result was that, in a “close” race, he won every swing state. That stark truth seems like precisely the sort of thing the prognosticators should have been able to tell us, at least in the aggregate. Instead, the Republicans defeated the pollsters.

OK, 2 things. First, you can put “close” in scare quotes if you want, but the election really was close! The popular-vote margin was less than 2 percentage points, and a swing of 2 percentage points also would’ve swung the electoral college. That’s a close election.

Second, “that stark truth” that Trump (or, for that matter, Harris) could’ve won every swing state was explicitly stated by the forecasters.

Recht and Weatherby continue:

Polls attempt to divine big-picture answers about the sentiment of millions of people from the responses of a vastly smaller group, many of whom aren’t especially eager to tell the truth about their opinions. The internet, ubiquitous cell phones, and the widely varying use of technology among different age demographics have all contributed to the problem, thwarting techniques honed at a time when landlines were in every American household.

With response rates in the single digits, pollsters are now forced to apply “statistical corrections,” backed by a series of guesses about the pre-existing beliefs and tendencies of the populace.

This is not new. Polls have never been random samples. Old-time Gallup polls used quota sampling, which is just a statistical correction implemented in the design, and it requires all the same assumptions that any adjustment would. Modern polls have nonsampling errors—enough to roughly double the stated margin of error—but that’s always been the case. Final polls were way off in 1980 and 1948 as well.

They continue:

Low response rates and statistical corrections make polling into a special kind of obfuscated punditry, undermining its claim to neutral objectivity and rendering it useless.

Well, no. As I discussed in my article, Failure and success in political polling and election forecasting, yeah, we’d prefer if polls had no nonsampling error—but nonsampling error of 3 percentage points is not so bad. It just happens that when the forecast is very close, the potential nonsampling error is consequential.

There’s a big difference between imperfect and “useless.”

They conclude that “the survey industry needs a tear-down.” I can’t argue one way or another on that claim. It’s a matter of opinion.

In the comments section, Recht adds, “If you can’t randomly sample, you shouldn’t survey.”

Again, that’s a statement of opinion, so nothing to argue about. There are essentially no random samples of humans, whether you’re talking about political surveys, marketing surveys, public health surveys, or anything else.

I think that organizations will keep doing surveys, and I think that applied survey researchers will continue to recognize problems of measurement, nonresponse, coverage, and generalization, and they will use statistical models to estimate uncertainty.

I get it that many people are frustrated when news reports focus on point estimates. But in 2024 I think the news media were pretty good about recognizing uncertainty! Even on election day, when forecasts were at 50-50, there were lots of news reports accurately stating that anything could happen. It wasn’t like 1992, 1996, or 2008, when on election day there was a clear expectation of who would win.

So, yeah, it’s too bad that polling isn’t random sampling—it never was!—and a 3 percentage point error isn’t nothing. I just don’t think it’s appropriate to say “the election forecasters gave up” or that “Whatever happened, it was supposed to be razor thin,” given that the forecasters did not say that; indeed they took pains to emphasize their uncertainty.

Why go into all this detail?

Why bother writing the above post? Is it a notorious case of a blogger being upset because, in the immortal words of Randall Munroe, “someone is

wrong

on the internet”? I guess so!

As a statistician who works on probabilistic inference sampling, I hate to see credentialed academic experts get things so wrong. To think that a 50-50 forecast is “giving up” or to think that the pollsters were “defeated” by being off by 2 percentage points is just so naive, especially given that this is no worse a polling error than was typical in the “time when landlines were in every American household.” Naive mistakes by non-experts are ok and, in some sense, necessary steps in building our understanding. I make naive mistakes all the time, and sometimes these even find their way into this blog. What bothers me more is that air of assurance which I associate with a certain style of writing on the internet. In the meantime, yeah, polls and forecasts are imperfect–indeed, this imperfection is built into forecasts, hence the wide uncertainties and the completely reasonable pre-election statements by Nate Silver and other forecasts.