Costs Of Slipshod Research Methods May Be In The Billions

Laboratory research seeking new medical treatments and cures is fraught with pitfalls: Researchers can inadvertently use bad ingredients, design the experiment poorly, or conduct inadequate data analysis. Scientists working on ways to reduce these sorts of problems have put a staggering price tag on research that isn't easy to reproduce: $28 billion a year.

That figure, published Tuesday in the journal PLOS Biology, represents about half of all the preclinical medical research that's conducted in labs (in contrast to research on human volunteers). And the finding comes with some important caveats.

The $28 billion doesn't just represent out-and-out waste, the team that did the research cautions. It also includes some studies that produced valid results — but that couldn't be repeated by others because of the confusing way the methods were described, or because of other shortcomings.

Leonard Freedman, who heads a nonprofit called the Global Biological Standards Institute, decided to undertake the study with two Boston University economists, Iain Cockburn and Timothy Simcoe. Their goal was to identify ways to make research more efficient.

"We initially were asking a very simple question," Freeman says. "We simply wanted to know how much money is being spent each year on basic preclinical research that is not reproducible."

That turned out to be a very difficult question; only a few studies have addressed the issue head-on, and they aren't directly comparable. The economists eventually homed in on a best estimate: $28 billion per year.

But that's a contentious conclusion.

There's no question there are some fields of research where reproducibility is a glaring problem. For example, many scientists who use human cells grown in the lab are actually using cells that aren't what they appear to be. A widely used cell labeled as breast cancer is actually a melanoma cell, it was recently discovered, and there are hundreds of similar examples.

Freedman notes that the National Institutes of Health annually funds about
$3.7 billion worth of research that uses cell lines. "So," he reasons, "if a quarter of these projects apparently use contaminated or misidentified cells, that's a large [share] of the $3.7 billion."

He figures that $1 billion could be saved if scientists first ran DNA fingerprint tests on their cells, to verify that they're studying the right tissue.

Other causes of trouble the scientists identified include poorly designed experiments, poorly documented lab protocols and inappropriate data analysis.

"Better training for students, postdoctoral fellows, even principal investigators is in order here," Freedman says.

Stefano Bertuzzi, executive director of the American Society for Cell Biology, acknowledges these are all real problems in his field but says that's not the whole story.

"The problem I have with the study is it wants to be a little bit sensational," Bertuzzi says.

The analysis lumps together fatal flaws — like using contaminated cells — with much less serious weaknesses, he says. Some studies are labeled as irreproducible because they give poor descriptions of their methods, making it difficult for other scientists to repeat the experiment.

"While this could be a nuisance, it doesn't mean the study is flawed or is wrong," Bertuzzi says. "The problem is folding all of this into a monetary value that gives the impression that this is wasted money. That's not correct."

Bertuzzi has written a rebuttal to the study, published on his society's Web page.

Freedman agrees that the $28 billion isn't all wasted effort, though it may be easy to jump to that conclusion. "I hope this work is not misconstrued," he says. As science explores the frontiers of knowledge, some missteps are inevitable. And as researchers working in other labs attempt the same experiments, errors get weeded out — and good results get validated. Reproducibility is a cornerstone of scientific research.

Freedman says he hopes that the dollar values he calculated will help focus attention on areas where improvements will yield a big bang for the buck. And in fact, his organization has launched a campaign that's trying to get scientists to make sure they know the true identity of the cells they're studying in their labs.