There seems to be pretty broad consensus in the polling community that the methods being used by Fox News to choose the participants in the first Republican debate is just terrible. The network will average the results of five recent major national polls and allow the top 10 candidates a spot on the debate stage.
The method suggests that a poll covering 17 different candidates can predict voter preferences with a degree of accuracy that nobody versed in statistical analysis believes. Worse still, it creates a system in which the polls themselves have an effect on the very thing they are trying to measure.
Related: Why Believing Donald Trump Will Be the GOP Nominee Is Delusional
In protest, the Marist Institute for Public Opinion actually removed a question about Republican voters’ presidential preferences from its most recent poll so that it wouldn’t be considered when Fox makes its call sometime tonight.
“It’s a bad use of public polls,” said Lee Miringoff, the director of the Marist Institute. “It asks public polls to have a precision that ignores the margin of error. There’s a big distinction made where there’s no statistical difference.”
Much of the criticism of the reliance on polls has to do with the margin of error. Any poll that doesn’t question every relevant member of the universe being polled is an estimate, and estimates are subject to potential sampling error. That is, a sample might not accurately reflect the universe from which it was taken.
So, when statisticians report their results, they cite a measure known as the “margin of error” along with a confidence level. For example, a poll might cite a 3 percent margin of error at a 95 percent confidence interval. That means that the sampling techniques used are expected, 19 times out of 20, to produce a result that is within 3 percentage points (higher or lower) than the true frequency in the population.
Related: Stop the Presses! Donald Trump Just Took a Policy Position on Planned Parenthood
The bigger the sample, the smaller the likely error, and the larger the sample, the higher the margin of error.
One thing that has been upsetting a lot of people writing about the debate selection criteria is that the margin of error in some of the most recent polls is really pretty big. A Wall Street Journal/NBC poll out this week polled 252 Republican voters on who among the candidates they preferred, with a margin of error of 6.17 percent.
The poll showed Donald Trump leading the field with 19 percent, trailed by Wisconsin Gov. Scott Walker at 15 percent and former Florida governor Jeb Bush at 14 percent. However, with a 6.17 percent margin of error, the three candidates are all running in a statistical dead heat.
It seems to get even more confusing at the low end of the poll, where 12 candidates are clustered between zero and 6 percent. It’s terribly unfair, it would seem, to say that a candidate at 3 percent gets a spot on the debate stage, while a candidate at 1 percent doesn’t even though they’re two scores are within the margin of error.
Related: Trump Shows Broadening Appeals as He Surges in Polls
However, there is one way in which the use of small differences– while still pretty bad – isn’t really quite as awful as margin of error figures make them sound.
It’s because when the results in a poll come in near the extremes, the margin of error doesn’t operate the same way anymore. In the most obvious sense, it’s not coherent to say that John Kasich, at 3 percent in the WSJ/NBC poll, has a 95 percent chance of having something between negative three percent of the vote and 9 percent. His vote tally can’t be less than zero.
And in this race, there are a lot of results coming in near the lower extreme. A handy calculator developed by Langer Research Associates helps illustrate the difference. In a sample size of 252, the margin of error for a result of one percent is about one percent. That means that former Hewlett-Packard CEO Carly Fiorina, who polled at 1 percent, might have actual numbers of close to zero, and might be around 2 percent, but almost certainly isn’t anywhere near 7 percent.
The same goes for Louisiana Gov. Bobby Jindal, at 1.2 percent, and former Pennsylvania Sen. Rick Santorum at 1.4 percent.
Related: Savior or Spoiler? What Happens If Joe Biden Runs Against Clinton?
That also means that candidates like Texas Sen. Ted Cruz, who polled at 6 percent, is probably within 2.9 percent of that in reality. It may not be in Trump territory, but Cruz actually has a statistically significant lead over Fiorina, et al. and can make a credible claim that by the rules, he clearly ought to have a spot in the debate before they do.
None of this is to say that the Fox debate arrangement is beyond reproach. It’s not a good system at all, in fact. But with a 17-candidate field, it’s far from clear what a good debate arrangement would actually look like. And while this may not be perfectly fair, it’s at least a little fairer than it looks.
