June 21st, 2007
Wednesday, December 01, 2004
By Brad Bannon
The good news for the polling industry was that the national telephone surveys conducted over the days leading up to the presidential election reflected the actual election result with most of them showing President Bush with a slim lead over his Democratic challenger John Kerry. The bad news is that the post-election exit polls conducted after the Americans had voted were way off.
Even in an imperfect world, a post-election survey should reflect the actual results better than a pre election survey.
In a pre-election survey, the pollster faces several problems. First, you have to find people to talk to which is a major problem when you’re trying to call people at home on land lines. Once you reach the person at the phone number from your sample, you have to figure out if the person you’re talking to (a.k.a., the respondent) is actually someone who is going to vote which in itself is a matter of some difficulty. People also might change their minds during the time period leading up to the election.
More after the jump…
In theory, at least, you don’t have any of these problems with an exit poll and therefore the in person surveys conducted at polling places should be a better reflection of reality than the pre election telephone surveys.
In effect, the exit pollster has the luxury of dealing with a captive audience which means the researcher has a better chance of capturing public opinion accurately. You don’t have to worry about identifying likely voters because all you have to do is to talk to every fourth, eighth or tenth person coming out of randomly chosen polling place and ask them how they just voted.
Despite the methodological advantages that exit pollsters have, they managed to find a way to screw up the numbers. The national exit poll indicated that Kerry won 51% of the vote and beat the President by 3%. The truth is of course that the President won 51% of the vote and beat Senator Kerry by 3%. And when you consider that the national sample size is more than 13,000 voters and the margin of error is so small, it is clear that someone was asleep at the switch. When you interview so many voters and come up so far short, something is seriously wrong.
The picture for the exit poll results in the battleground states is not any prettier. Steven Freeman of the University of Pennsylvania wrote an interesting analysis of the state by state exit results and he found that exit polls reported a significantly higher Kerry vote than the actual vote in 10 of the 11 battleground states that he examined.
In Ohio where the researchers had the chance to interview almost 2000 real live voters, they came up with Kerry leading by 4 points and Bush won by more than 2 points. I would like to think that a class of political science majors at Ohio State could have interviewed half that many voters on Election Day and come up with a result that better reflected the reality of the outcome in the Buckeye State. In New Hampshire, which is small, homogeneous and fairly easy to poll in, the exit poll had Kerry up by 11% and he ended up winning the state by only 2%.
The word snafu is an old U.S. Army acronym for Situation Normal All F*#@ed Up and it pretty much describes the state of the exit polls. In 2000, the exit polls in Florida played a role in the Sunshine State election crisis. In 2002, the entire system crashed and the networks were not able to report the results of the exit surveys until several days after the election, which might have been a blessing instead of a curse. And during the afternoon of Election Day this year, potential Kerry administration appointees were measuring the drapes in their new White House offices after they looked at the exit poll data which was circulating pretty freely on various web sites.
The folks at Edison Media Research and Mitofsky International have not yet come up with a plausible explanation for the discrepancy between the exit poll numbers and the actual results. But they had better, because the media is spending $10 million a cycle on the exits and there are a lot of unhappy reporters who spent the afternoon of Election Day working on Kerry in the White House stories that will never see the light of day.
The folks at Edison and Mitosky argue that the polls are designed to explain why people voted the way they do and are not supposed to be predictive of the result. But if they don’t do a better job getting the basic numbers right, no one is going to take the valuable data on voting behavior in the exits very seriously.
Brad Bannon is president of Bannon Communications Research which has designed poll driven messages for the last 20 years for labor unions, Democrats and progressive issue groups. You can reach Brad at Brad@BannonCR.com.
