Just How Arbitrary Is Fox's 10-Person GOP Debate Cutoff?

Update: This post was updated at 6:55 p.m. ET to reflect Fox's announcement of debate participants.

The Republican presidential field has just had the most exciting fight for 10th place America has ever seen.

It also just might have been a meaningless fight.

With the major contenders for the GOP nomination now numbering 17, Fox News will only allow the top 10 candidates into the first GOP debate on Thursday. To determine the participants, Fox averaged together five national polls from Fox, Bloomberg, CBS, Monmouth and Quinnipiac.

And that methodology has had political scientists up in arms. The Marist Institute for Public Opinion suspended polling this week because it was worried polls were being used to make too fine of distinctions between candidates.

Averaging polls together can diminish the margin of error to a degree, but not enough to make these candidates easily rankable, as the Fox cutoff demands.

Using Real Clear Politics' tallies of the poll totals, I took a stab at averaging the latest five polls together and determining the resulting margin of error for each candidate with some help from Samuel Wang, neuroscientist and founder of the Princeton Election Consortium blog. Here's a look at the results.

The five polls Fox used had maximum margins of error of plus or minus 3.7 percentage points or greater on questions about which GOP candidates people support. Averaging those polls together (see my spreadsheet of step-by-step math here) clearly diminishes that for most candidates — but not enough to (in this example) totally eliminate the overlap between 10th-place Kasich and 11th-place Perry. In this example, Perry's margin of error now barely — but significantly — extends into Kasich's territory by a little more than a tenth of a percent.

That means Perry could argue that the difference between his support and Kasich's support isn't meaningful enough to keep him out of the debate. And Rick Santorum also overlaps with Perry — perhaps he should be in. In which case maybe Jindal should. And so on.

To be clear, this is just an example of how margins of error could look. But it gets at the fact that even with hard statistics, things can get fuzzy quickly.

For example, instead of simply averaging candidates' support levels together (as I did above), they could have chosen to weight polls by their sample sizes — Twitter user @Taniel has been tracking the differences between the two different averaging approaches in a helpful Google spreadsheet.

Also, consider that the polls didn't all measure the same populations — Fox, for example, surveyed likely Republican primary voters about their nominee preferences. Bloomberg, meanwhile, spoke to "Republicans or Republican leaners."

There may be a million ways you can nitpick how and why Fox did its debate math (and you'll hear a lot of it during the hours leading up to the debates themselves on Thursday night).

Still — however you do the math, using polls to cut down the debate field in this way risks stopping some candidates' runs by arbitrarily cutting them out of the conversation before the campaign has really even begun. (Fox is hosting an earlier debate for the lower-ranked candidates — a debate that many have likened to a "kids' table.")

"Polls are not very useful right now except for telling you about tiers of candidates. They really tell you that Trump has more support than Christie and Rand Paul. It really tells you about tier of public visibility," said Cliff Zukin, professor of public policy at Rutgers University, who serves on the executive council of the American Association for Public Opinion Research.

It's true that polls have been used to limit debate fields in the past. Fox spokeswoman Carly Shanahan pointed out to NPR that Fox has used polling to determine debate participants; in a 2011 South Carolina GOP debate, the network limited participation to anyone polling at 1 percent or above.

Still, a 1 percent cutoff is designed to include everyone with legitimate (if small) support. The top-10 cutoff this year is designed to winnow down the field. The idea this time around appears to be exclusivity, rather than inclusivity.

Think of it in terms of the tiers Zukin talks about: Cutting the field between 10 and 11 seems to cut the bottom tier in an arbitrary place. You could argue that Kasich and Santorum, both polling in the low single digits, are in the same "tier." But one made it in, while the other didn't.

"I suppose Fox hoped that a top tier would emerge by the time the first debate rolled around. But based on current polling, there's no good rationale for arbitrarily selecting a top 10," said Patrick Murray, director of the Monmouth Polling Institute, in a press release about his group's latest poll.

Fox spokeswoman Shanahan didn't respond to questions about Fox's methodology but has directed NPR to statements made by Fox in other outlets on Fox's criteria.

"National polls are the traditional, time-tested yardstick by which presidential hopefuls have long been measured and remain the fairest, most objective and most straight-forward metric for gauging the viability of these candidates," Michael Clemente, executive vice president of news for Fox, told Bloomberg.