Statistics professors give Fox News a B- on their big polling test

The results are in.

Fox News has settled on the 10 Republican candidates who will do battle on the “main stage” during the first televised GOP debate.

As expected, Donald Trump will take center stage as the undisputed leader of the polls, with an average reported by Fox News of 23%.

But while Fox is using polling and statistical analysis to justify their selections, their methods of calculation strike us – two statistics professors from Oklahoma State University – as too fast and loose.

The answers Fox got

According to Fox News, these candidates will be joining Trump based on their polling averages: former Florida Governor Jeb Bush (13%), Governor Scott Walker of Wisconsin (11%), retired neurosurgeon Ben Carson (7%), former Arkansas Governor Mike Huckabee (7%), Senators Ted Cruz of Texas (6%), Rand Paul of Kentucky (5%) and Marco Rubio of Florida (5%), and Governors Chris Christie of New Jersey (3%) and John Kasich of Ohio (3%).

The remaining seven candidates (Rick Perry, Rick Santorum, Bobby Jindal, Carly Fiorina, Lindsey Graham, George Pataki and John Gilmore) will duke it out at the debate undercard to be held earlier that day.

The Perry question

Rick PerryJim Young/Reuters

The notable candidate missing from the main event is Perry, whose recent tete-a-tete with Trump has received a lot of attention from the media. Perry referred to Trump as “a cancer on conservatism,” while Trump suggested that Perry should be required to take an IQ test prior to being allowed on the debate stage.

Fox News did not go out of its way to ensure Perry a spot, even though the fireworks that may have resulted could have scored big for the news network.

Had Fox News decided to include Perry, it could have picked a combination of polls that put Perry ahead of Kasich. For instance, the following five give Perry the lead: CBS News (completed on Aug. 2), Fox News (August 2), NBC/Wall Street Journal (July 30), Reuters/IPSOS (July 28) and Reuters/IPSOS (July 22).

The drawback to using those five polls is that they are not the five most recent, they use different data collection methods and they use different sampled populations. The five Fox actually picked have the advantage of consistency.

Points for consistency

According to Fox, the five polls utilized were CBS News, Bloomberg, Monmouth University, Quinnipiac University and their own Fox News poll.

An NBC/Wall Street Journal poll, which did fall within the five most recent, did not meet Fox’s criteria for inclusion because it did not include Kasich, the network reported.

A second important aspect of the inclusion criteria was the restriction against automated phone interviews. All the polls utilized consisted of live interviews, and both landlines and cellphones were included. This aspect reduced the bias inherent in using automated polling and relying solely on landline phone numbers.

A third consistency in the five polls selected was their sampled population. All five polls sampled “registered voters.” Keeping to a single population ensures that the polls can be compared to one another, and that the calculated average is meaningful.

Points off for averaging errors

The real concern is the method Fox News used to obtain their averages.

They employed the most basic method imaginable. The rank order was determined by a simple arithmetic average of publicly available results. Averages were rounded to the nearest tenth of a percentage point. The sample sizes and margins of error of the five polls were ignored in the calculations.

To see why this is an issue, let us look at an extreme case. In Poll A, 200 registered Republicans are asked their preference. In that sample, 5% prefer Perry and 2% prefer Kasich. In Poll B, 2,500 registered Republicans are asked their preferences. In this poll, 2% prefer Perry and 4% prefer Kasich. Simply averaging the percentages puts Perry on the stage with an average of 3.5% to Kasich’s 3%.

However, between the two polls, only 60 people preferred Perry – 10 from Poll A and 50 from Poll B. A total of 104 preferred Kasich – 4 from Poll A and 100 from Poll B. In this extreme example, Fox’s simple averaging would have put the candidate preferred by 2.2% in the debate and sidelined the candidate preferred by 3.9%.

Weighted averages matter.

Fox didn’t use weighted averages, but they used the concept to justify their result.

The network claimed that “given the over 2,400 interviews contained within the five polls, from a purely statistical perspective it is at least 90% likely that the tenth place Kasich is ahead of the eleventh place Perry.”

No real harm done

Had Fox use the more correct weighted averaging method, the only change would have been a change in position for Christie and Kasich.

That change is no big deal. And perhaps Fox wanted to use math that is easier to explain to viewers.

But what would have happened if the decision to use unweighted averages resulted in a different lineup? It could have meant that a candidate regulated to the “kiddie table” actually deserved to eat with the adults.

One thing Fox got right: It showed its work, giving voters the ability to see how the selections were made.

The authors do not work for, consult, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond the academic appointment above.