Nate Silver: Sam Wang's model showing Democratic Senate advantage 'is wrong'

Today, Nate Silver of FiveThirtyEight posted a long explanation of how his Senate forecasting model works this year — check out Vox's new forecasting hub to see how it compares to several other models. But in today's post, Silver mentions that he has particular objections to the design of one competing model. That's Princeton neuroscientist Sam Wang's model, which in recent weeks has given Democrats more rosy projections than any others, showing them with a clear Senate advantage. Silver writes:

That model is wrong - not necessarily because it shows Democrats ahead (ours barely shows any Republican advantage), but because it substantially underestimates the uncertainty associated with polling averages and thereby overestimates the win probabilities for candidates with small leads in the polls. This is because instead of estimating the uncertainty empirically - that is, by looking at how accurate polls or polling averages have been in the past - Wang makes several assumptions about how polls behave that don't check out against the data.

Silver specifically argues that, by building so little uncertainty into his model, Wang's projections don't present an accurate estimate of the chances that the polls may be wrong — either in certain races or overall. He argues that Wang's 2010 model estimated absurdly low probabilities for events that actually ended up happening — Sharron Angle losing to Harry Reid and the GOP picking up 63 seats in the House.

One of Silver's broader points is that his model makes extensive use of historical data about how much polls have been off by in the past — because it's not at all infrequent for polls to miss the mark. Wang's model, though, doesn't take this into account.  "There are many elections in which all or almost all polls are biased in the same direction," Silver writes, adding that Wang "underestimates the chance for the underdog to win because of systematic errors in the polls, better-than-expected turnout, and so forth."

Update: Check out a response from Wang here, courtesy of the Washington Post's Greg Sargent.

Will you support Vox’s explanatory journalism?

Most news outlets make their money through advertising or subscriptions. But when it comes to what we’re trying to do at Vox, there are a couple reasons that we can't rely only on ads and subscriptions to keep the lights on.

First, advertising dollars go up and down with the economy. We often only know a few months out what our advertising revenue will be, which makes it hard to plan ahead.

Second, we’re not in the subscriptions business. Vox is here to help everyone understand the complex issues shaping the world — not just the people who can afford to pay for a subscription. We believe that’s an important part of building a more equal society. We can’t do that if we have a paywall.

That’s why we also turn to you, our readers, to help us keep Vox free. If you also believe that everyone deserves access to trusted high-quality information, will you make a gift to Vox today?

One-Time Monthly Annual

$5/month

$10/month

$25/month

$50/month

Other

$

Yes, I'll give $5/month

Yes, I'll give $5/month

We accept credit card, Apple Pay, and Google Pay. You can also contribute via