Trolling our confirmation bias: one bite and we're easily sucked in

Earlier in the year the world was finally treated to some good news from science: a report was published that claimed to show that eating chocolate could help you lose weight faster.

Although it all seemed too good to be true, the story was reported in news outlets around the world. Europe’s largest daily newspaper, Bild, ran it on the front page. It made TV news in Australia and the US, it landed on the Irish Examiner, The Times of India, and the Huffington Post in variouslanguages.

But it was too good to be true. Or, if you’re an aficionado of the work of trolls, it was even better.

Last week science journalist John Bohannon revealed that the whole study was an elaborate prank, a piece of terrible science he and documentary film makers Peter Onneken and Diana Löbl – with general practitioner Gunter Frank and financial analyst Alex Droste-Haars – had set up to reveal the corruption at the heart of the “diet research-media complex”.

Terrible science

So what did they do? Bohannon and his team went through all the standard practices of science. But at every stage they chose methods they knew would lead not to truth, but to clickbaity headlines.

Has the Daily Star gone coco?Screenshot by author

To begin the study they recruited a tiny sample of 15 people willing to go on a diet for three weeks. They divided the sample into three groups: one followed a low carbohydrate diet; another followed that diet but also got a 42 gram bar of chocolate every day; and finally the control group were asked to make no changes to their regular diet.

Throughout the experiment the researchers measured the participants in 18 different ways, including their weight, cholesterol, sodium, blood protein levels, their sleep quality and their general well being.

And here’s their first trick. Measuring such a tiny sample in so many ways means you’re almost bound to find something vaguely reportable. As Bohannon explains it:

Think of the measurements as lottery tickets. Each one has a small chance of paying off in the form of a “significant” result that we can spin a story around and sell to the media. The more tickets you buy, the more likely you are to win. We didn’t know exactly what would pan out — the headline could have been that chocolate improves sleep or lowers blood pressure — but we knew our chances of getting at least one “statistically significant” result were pretty good.

And so then they submitted it for publication. But again, Bohannon chose the path that led away from truth, picking a journal from his extensive list of open access academic journals (more on this below). Although the journal, (International Archives of Medicine), looks somewhat like a real academic journal, there was no peer review. It was accepted within 24 hours, and published two weeks later.

But great publicity!

Practiced in the white magic of science journalism and familiar with the dark arts of science PR, Bohannon then whipped up a press release he knew would bait the world’s media.

Hmm, chocolate study?Screenshot by author

The key, Bohannon stated, was to “exploit journalists’ incredible laziness” – to write the press release so that reporters had the story laid out on a plate for them, as it were. As he later wrote, he “felt a queazy mixture of pride and disgust as our lure zinged out into the world”. And a great many swallowed it whole.

Headlines around the world screamed Has the world gone coco? Eating chocolate can help you LOSE weight, Need a ‘sweeter’ way to lose weight? Eat chocolates! and, perhaps more boringly, Study: Chocolate helps weight loss.

Some of these reports remain online today in the same state as they were published, although some outlets, such as Cosmopolitan Germany and Huffington Post India, have since updated to reveal the sting. The Australian TV news piece has been deleted, like the mistake never happened.

What’s the washup?

The reporters around the world who cut-and-pasted Bohannon’s press release certainly aren’t blameless. None did the due diligence – such as looking at the journal, looking for details about the number of study participants, or even looking for the institute Bohannon claimed to work for (which exists only as a website) – that was necessary to find out if the study was legitimate.

But if we’re really looking to find fault here, we’ve got to cast our net a bit wider. As Bohannon and his colleagues noted, there’s a “diet research-media complex” here that’s almost rotten to the core.

From beginning to end we’ve got a system with almost as much scope for corruption as a BBQ at a high ranking FIFA official’s house:

• we’ve got researchers around the world who have taken to heart the dictum that the quantity of research outputs is more important than the quality

• we’ve got journal publishers at the high quality end that care about media impact more than facts

• we’ve got journal publishers at the no-quality end who exploit the desperation of researchers by offering the semblance of publication for a modest sum

• we’ve got media outlets pushing their journalists ever harder to fill our eyeballs with clickbaity and sharebaity content, regardless of truth

• and we’ve got us: simple creatures prone to click, read and share the things that appeal to our already existing biases and baser selves.

Not the heroes we want

In the stories they tell about themselves, scientists, journalists and popular and scholarly publishers share a common dogma: that they’re heroes in the pursuit of truth. This may be true as individuals, but the pressures of their respective industries distort them in ways which can be utterly cyncial.

And so it’s interesting that Bohannon has pulled a similar prank to this before, submitting a deeply flawed paper on possible cancer inhibiting molecules to a plethora of different journals, with many accepting the paper with nary a comment.

We should, perhaps, look at work like this in the abstract – as a form of trolling to expose the self serving, the cynical and the corrupt. Trolls like Bohannon may not be the heroes we want, but they’re the heroes this dirty world of ours needs.

Will J Grant does not work for, consult to, own shares in or receive funding from any company or organisation that would benefit from this article, and has no relevant affiliations.