How to fake science

Journalist Johannes Bohannon tells the story of how he crafted an “experiment” that “proved” that eating chocolate helps weight loss–and sold it to millions of readers in the media world. How timely, especially the part about journalistic laziness. Think of how often the techniques employed here been applied by medical equipment manufacturers and others. Excerpts, with my emphasis:

Could we get something published? Probably. But beyond that? I thought it was sure to fizzle. We science journalists like to think of ourselves as more clever than the average hack. After all, we have to understand arcane scientific research well enough to explain it. And for reporters who don’t have science chops, as soon as they tapped outside sources for their stories—really anyone with a science degree, let alone an actual nutrition scientist—they would discover that the study was laughably flimsy. Not to mention that a Google search yielded no trace of Johannes Bohannon or his alleged institute. Reporters on the health science beat were going to smell this a mile away. But I didn’t want to sound pessimistic. “Let’s see how far we can take this,” I said.

I know what you’re thinking. The study did show accelerated weight loss in the chocolate group—shouldn’t we trust it? Isn’t that how science works?

Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people. (One subject was dropped.) That study design is a recipe for false positives.

It was time to share our scientific breakthrough with the world. We needed to get our study published pronto, but since it was such bad science, we needed to skip peer review altogether. Conveniently, there are lists of fake journal publishers. Since time was tight, I simultaneously submitted our paper—“Chocolate with high cocoa content as a weight-loss accelerator”—to 20 journals. Then we crossed our fingers and waited.

Our paper was accepted for publication by multiple journals within 24 hours. Needless to say, we faced no peer review at all. The eager suitor we ultimately chose . . . emailed Johannes to let him know that we had produced an “outstanding manuscript,” and that for just 600 Euros it “could be accepted directly in our premier journal.”

With the paper out, it was time to make some noise. I called a friend of a friend who works in scientific PR. She walked me through some of the dirty tricks for grabbing headlines. It was eerie to hear the other side of something I experience every day.

The key is to exploit journalists’ incredible laziness. If you lay out the information just right, you can shape the story that emerges in the media almost like you were writing those stories yourself. In fact, that’s literally what you’re doing, since many reporters just copied and pasted our text.

Take a look at the press release I cooked up. It has everything. In reporter lingo: a sexy lede, a clear nut graf, some punchy quotes, and a kicker. And there’s no need to even read the scientific paper because the key details are already boiled down. I took special care to keep it accurate. Rather than tricking journalists, the goal was to lure them with a completely typical press release about a research paper.

I felt a queazy mixture of pride and disgust as our lure zinged out into the world.

When reporters contacted me at all, they asked perfunctory questions. “Why do you think chocolate accelerates weight loss? Do you have any advice for our readers?” Almost no one asked how many subjects we tested, and no one reported that number. Not a single reporter seems to have contacted an outside researcher. None are quoted.

There was one glint of hope in this tragicomedy. While the reporters just regurgitated our “findings,” many readers were thoughtful and skeptical. In the online comments, they posed questions that the reporters should have asked.

SOURCE: Not Running a Hospital – Read entire story here.