Why A Journalist Scammed The Media Into Spreading Bad Chocolate Science

Earlier this spring, headlines around the world trumpeted an exciting bit of news that seemed too good to be true: "Eating chocolate ... can even help you LOSE weight!" as Britain's Daily Mail put it.

From India to Australia and Texas to Germany, news organizations shared findings published in the International Archives of Medicine in late March.

The problem? The study they were based on was pure junk. And the person behind it, John Bohannon, would be the first to tell you that.

In fact, that's exactly what he did Wednesday, in a story for io9 that's gone viral. Bohannon, a science journalist who also has a Ph.D., lays out how he carried out an elaborate hoax to expose just how easily bad nutrition science gets disseminated in the mainstream media.

Diet science, Bohannon stresses, is still science – and reporters need to know how to cover it. "You have to know how to read a scientific paper — and actually bother to do it," he writes. "For far too long, the people who cover this beat have treated it like gossip, echoing whatever they find in press releases. Hopefully our little experiment will make reporters and readers alike more skeptical."

To be clear, the study involved was real — a randomized controlled trial. Bohannon and his partners, a German television reporter and his collaborator, really did recruit 16 people for a study on dieting. And they found that the ones who followed a low-carb diet and also ate a 1.5-ounce bar of dark chocolate daily lost weight faster than the control group that was dieting alone.

So what's the problem, you ask? There are plenty.

For starters, as Bohannon explains in great detail, the study design itself was flawed — it had too few subjects, and the research measured too many factors, making it likelier that some random factor would appear to have statistical significance. (Bohannon does an excellent job of explaining the specifics himself.)

Then there's the journal that published it — a so-called pay-for-play publication, which failed to carry out peer review of the findings. As Bohannon himself exposed in another sting for the journal Science a couple of years ago, there are lots of these publications that will publish bad research for a fee.

And finally, none of the reporters who covered it asked an outside expert to weigh in on the research – standard operating procedure in good science journalism. If they had, an astute scientist would have spotted the problems with the study design immediately. Indeed, just this week, our sister blog, Shots, ran a great post by Tara Haelle explaining how this process of tapping independent experts helped steer her from reporting on a flawed study suggesting that a childhood vaccine prevents leukemia.

Bohannon's full story is long, but worth the read as an explanation of the pitfalls that plague science communication – especially nutrition information – in today's media climate.

And the problem is massive, says Gary Schwitzer, the publisher of Health News Review. For the past nine years, the site has been dedicated to critiquing the media's coverage of health in an effort to improve it.

"He's really only scratching the surface of a much broader, much deeper problem," Schwitzer says. "We have examples of journalists reporting on a study that was never done. We have news releases from medical journals, academic institutions and industry that mislead journalists, who then mislead the public." And the pressure to publish or perish, he says, can lead well-intentioned scientists to frame their work in ways that aren't completely accurate or balanced or supported by the facts.

"We are really mired in a mess, the boundaries of which few people really have a sense for," says Schwitzer.

Schwitzer says it's appropriate that Bohannon chose chocolate for his diet study stunt. From claims that chocolate can boost memory to press releases that oversell its benefit to heart health, the sweet stuff is the subject of many of the news stories Schwitzer's site criticizes. (To be clear, there is good research suggesting benefits from eating chocolate — but Bohannon's study isn't it.)

But does Bohannon's stunt reveal the credibility crisis in nutrition science journalism or add to it?

"I think it's a little of both," Schwitzer says.

Schwitzer is a fan of Bohannon's message – but he worries that it will get lost in our rapid churn, 24-hour news cycle, and the journalists who need to be schooled in proper nutrition science reporting will have already moved on to the next thing.

And there are others who aren't taking kindly to Bohannon's work. On Twitter last night, some scientists raised concerns about the ethics of running a study with human subjects with the ultimate intent to deceive, and whether participants could give informed consent if they didn't know that aim.

Art Caplan, a bioethicist with the New York University Langone Medical Center, says Bohannon's experiment straddles the line between journalism and research.

"I'm a fan of the study," he tells The Salt, "because I think it shows certain issues in understanding media reports about nutrition."

But, he says, participants should have been debriefed at the end of the study period about what Bohannon intended to do with the data. (Note: We've reached out to Bohannon to ask about what participants were told and will post an update if we get more details.)

And not everyone agrees that Bohannon's ends justify his means.

"It's totally not ethical. It's deliberate deception in a way that could harm people," says Kelly McBride, a media ethicist with the Poynter Institute. She adds: "There's so much bad information out there, especially around diet and science, that when you make it worse you become part of the problem."

About two months have gone by since the headlines touting Bohannon's study first appeared. By now, as McBride notes, many of the people who read the news stories touting chocolate's weight-loss benefits have moved on – and may not ever come across updates explaining it was all a ruse in the service of better nutrition journalism.

As for the journalists for whom this exercise was intended? McBride doubts they've learned their lesson, either.

"It's not a great teaching tool to humiliate people," she says, "because people learn when they have a need to learn something, not when they're embarrassed or fear being embarrassed. That's not how adults learn." (Update: Some of the media outlets that covered the chocolate study are starting to post updates to their original stories.)

But in an environment saturated with news, this kind of extreme measure, Schwitzer argues, might be exactly what's needed to highlight a big problem.

Update: Bohannon discussed his experiment with NPR's Robert Siegel on Friday. Listen to the interview.