Survey House Effects and The 2009 NJ Gubernatorial Election

So from last night’s NJ gubernatorial election, I want to take a moment to analyze “house effects” among polling firms who conducted polls during this race. The importance of analyzing house effects is because there are systematic effects between polling firms. As such, survey firms can drive different results.

So what is a house effect? Examples include differences in the methods used by pollsters in how they frame questions, deal with undecided voters, who they interview (e.g. adults, registered voters or likely voters), or how they conduct interviews; for instance, this could be either IVR (interactive voice response) or live interviewers. To estimate House effects, I first calculate the overall trends for Corzine, Christie, and Daggett using a lowess (local weighted regression) model. Once this is computed, I then subtract each poll separately from each candidate’s overall trend. Lastly to estimate differences between candidates by polls, I subtract the House differences per candidate and then take the average differences per polling firm.

To visual “House Effects” I have created three charts which show the effects of polling firms that participated in the 2009 NJ gubernatorial election. The dots presented in each chart are the estimated house effects by pollster. Because of the nature of a three-way race in this year’s election, I have created three charts to compare effects across the three leading candidates: First, Corzine compared to Christie; second, Corzine to Daggett; and third, Daggett to Christie.

For the first chart, the largest pro-Corzine house effect is Suffolk University at just over 10 points. The poll most favorable to Christie was Basswood (R)/Republican Governors Association at about -6 points. All other pollsters fall between these two extremes.

What does this mean? Simply that the effects on the differences between candidates from Suffolk University is equivalent to 5 points higher for Corzine and five points lower for Christie. So, a simply rule of thumb is to take half the estimated house effect per firm and this will give you the above and below average effects per candidate. In this case, the average effects are approximately 5 points per candidate. These differences are large enough to show significant variability between pollsters. This is a likely reason behind the inconsistency in polls throughout the election cycle.

For the second chart, a similar pattern emerges when comparing Corzine to Daggett. When comparing the house effects between these two candidates, the largest pro-Daggett firm was Rutgers/Eagleton at -10 points and the most favorable poll to Corzine was Fairleigh Dickinson at over 10 points. The difference between candidates for the Rutgers/Eagleton poll is roughly 5 points higher for Daggett and 5 points lower for Corzine. Again the average difference among pollsters was 5 points.

Lastly the third chart, which analyzes house effects between Daggett and Christie, shows similar patterns to the Daggett-Corzine match-up. Namely an average difference of about 5 points between these two candidates.

To sum up, the analysis demonstrates that house effects in this year’s election were not only real and measurable, but the magnitude of these effects helped explain why it was so difficult in assessing which candidate would win the election based on the polls.