Getting Seasick With Gallup
Posted by wpcomimportuser1 | Email This | Permalink | Email Author
Republicans in the last few days have found themselves buoyant -- thanks in large measure, it seems, to what about 2,000 respondents had to say to Fox News and Gallup about which party they prefer in November. Suddenly, in the "generic ballot," the GOP is even with the Democrats.
Unfortunately, some of these buoyant Republicans seem to be the same people who took great heart in my column about the perils of the generic ballot. I am not sure how the two positions are reconcilable -- how you can join me in rejecting the generic in August and now embrace it in September. But I would note that I saw myself quoted by many bloggers as saying, "The generic ballot has a Democratic skew...it is just not trustworthy!" The "..." eliminated a very important point, which was that the skew is just the begining of what are very serious problems. So, I am not entirely surprised by the positive reception the recent Gallup generic has enjoyed on the right.
Republicans might be buoyant, but personally I'm getting seasick. In the last month, the Gallup generic has gone back and forth. In late August, it had the Democrats up only 2% among registered voters in the generic ballot. Then the Dems sprang ahead to a 12% lead in the course of about 15 days. And then, another 15 days later, the "likely voters" are driving another tie.
Gallup has done this to us before. As I have mentioned, there is a large and sustained Democratic skew to the generic ballot. But the skew is not omnipresent. A few times it offered up outlying results in the summer such that, when you take the average of their summer polls together, you actually get a result that underestimates Democratic strength. I found this to be the case for a few election years when I analyzed the June/July data. 1986 was a major problem for the model because of its underestimation of Democratic strength. Why? In any estimate where you use one variable (generic ballot) to predict another (final House vote), the error has to be distributed in a bell-shaped fashion. When it is not, often the problem is due to an outlying incidence. Your model is predicting one value that is much, much farther from the actual result than any of the other predictions. This was the case with 1986. In this situation, standard operating procedure is to toss the outlying observation (or to fix the data point if there was a calculation/recording error) after you have determined that there is not some kind of hidden centrality to it (e.g. is the exception actually the rule?). Otherwise, you risk throwing off your ability to draw a prediction, which of course is the whole point.
So, from my perspective -- I have seen Gallup go bouncing back and forth like this before. It was a major pain in my you-know-what this summer as I was trying to make sense of the generic ballot. I'm not jumping on this bandwagon. Note that this should not be taken as good news for Democrats or for Republicans. I would ask that liberal bloggers avoid the "..." that I saw on the right last month. I think the generic ballot is so awash with analytical problems that any kind of direct use of it is trouble.
However, if you are comparing the generic ballot to some other final result, you might not encounter this problem. Emory University's Alan Abramowitz uses the generic ballot as a measure of partisan strength to help predict seat changes, as opposed to vote changes, in an upcoming scholarly article in the journal PS. By using a different dependent variable (seats instead of votes), and by using it in the context of other predictive variables, you can eliminate these problems. But comparing the generic ballot to final popular votes, which is what most everybody else uses it for, is inferential trouble.

