Rate the Polls – Lessons in Market Research

Looking back at our election preview it seems that the bulk of our contributors did foresee Barack Obama’s win.  Junta42’s Joe Pulizzi, Brian Bucky and our own Kimba Jackson probably came in the closest to the final result.  For their efforts they win a year supply of Rice-a-Roni and brand spanking new President.  A look at the efforts of professional pollsters is a good lesson in market research technique and how to interpret data.

The subject of surveying was a big one this year as pollsters had the very tough task of trying to estimate the proportion of voters by party identification.  That is the key variable for weighting the results.  It is usually based on the last national election – Presidential or mid term congressional.  In 2008 pollsters tried to grapple with the unknown factor of how many new voters Obama would be bringing into the process.  It was tempting to jack up the percentage of democratic voters based on the response to Obama’s appearance and fund raising.  The results are in and Fordham University did a study comparing the results of all of the polls.  The winners were Rasmussen and Pew which nailed the final margin.  See the full list here.

All during the campaign I was reading and relying on Rasmussen because they were the most conservative in their estimates, and always had it as a pretty tight race.  I had a gut feel that Gallup was over weighting to Democratic Party identification and as a result they came in near the bottom for accuracy.  A couple of lessons learned:

The best poll methodology is the tracking polls like Rasmussen because they survey every day and keep reporting a three day rolling average.  They also screen on likely voters rather than registered voters which tend to favor democratic candidates over republican.

Because of accelerated news cycles and proliferation of news and information sources opinions move and change quickly.  That makes the tracking polls superior to the polls that are just a snapshot.  The “snap shot” polls only capture a moment in time vs. a rolling average.

Put your trust in pure research companies over polls sponsored or done by media companies.  The bottom half of the list is dominated by media outlets such as WSJ, NBC, ABC and FOX.  And the two overall worst were the New York Times/CBS poll and Newsweek’s.  All of these snapshot polls consistently over rated Obama’s support and put him much further ahead.  They gave the appearance of a much bigger margin between Obama and McCain.  Media companies do these polls for marketing and PR purposes as much as to provide content to their viewer and readers.  Going forward, they should be somewhat discounted after this very poor performance.  Newsweek’s poll was particularly galling – they usually had Obama up by 12-15 points.  And, since they promoted the heck out of their survey it made news because it was showing such a wide spread.

I’d like to give them the benefit of the doubt that their poll was just a poorly done marketing gimmick.  I would be shocked, shocked to learn that objective, unbiased news organizations such as Newsweek and the NY Times would try to manipulate the election by showing Obama ahead by huge margins that no research company was finding. 

Anyway, it’s time to move on and wish President-elect Obama good luck.  And one final message to the media.  Please give Mr. Obama some space and let him pick his cabinet without the need to vet (a.k.a. try to find information to ruin their lives) the people he may be considering.  We will be a better country if you show some restraint and class.