Nate Silver: Sam Wang's model showing Democratic Senate advantage 'is wrong'

Today, Nate Silver of FiveThirtyEight posted a long explanation of how his Senate forecasting model works this year — check out Vox's new forecasting hub to see how it compares to several other models. But in today's post, Silver mentions that he has particular objections to the design of one competing model. That's Princeton neuroscientist Sam Wang's model, which in recent weeks has given Democrats more rosy projections than any others, showing them with a clear Senate advantage. Silver writes:

That model is wrong - not necessarily because it shows Democrats ahead (ours barely shows any Republican advantage), but because it substantially underestimates the uncertainty associated with polling averages and thereby overestimates the win probabilities for candidates with small leads in the polls. This is because instead of estimating the uncertainty empirically - that is, by looking at how accurate polls or polling averages have been in the past - Wang makes several assumptions about how polls behave that don't check out against the data.

Silver specifically argues that, by building so little uncertainty into his model, Wang's projections don't present an accurate estimate of the chances that the polls may be wrong — either in certain races or overall. He argues that Wang's 2010 model estimated absurdly low probabilities for events that actually ended up happening — Sharron Angle losing to Harry Reid and the GOP picking up 63 seats in the House.

One of Silver's broader points is that his model makes extensive use of historical data about how much polls have been off by in the past — because it's not at all infrequent for polls to miss the mark. Wang's model, though, doesn't take this into account.  "There are many elections in which all or almost all polls are biased in the same direction," Silver writes, adding that Wang "underestimates the chance for the underdog to win because of systematic errors in the polls, better-than-expected turnout, and so forth."

Update: Check out a response from Wang here, courtesy of the Washington Post's Greg Sargent.

Explanatory journalism is a public good

At Vox, we believe that everyone deserves access to information that helps them understand and shape the world they live in. That's why we keep our work free. Support our mission and help keep Vox free for all by making a financial contribution to Vox today.

One-Time Monthly Annual

$95/year

$120/year

$250/year

$350/year

Other

$

Yes, I'll give $120/year

Yes, I'll give $120/year

We accept credit card, Apple Pay, and Google Pay. You can also contribute via