Why Reporting On Scientific Research May Warp Findings

Transcript

DAVID GREENE, HOST:

Next we're going to report on scientific research, in particular on the way that reporting on scientific research might actually warp the findings. Scientists face pressure to publish new discoveries, which in turn might influence what they study, and that, of course, is not necessarily a good thing. There's work being published today that's part of an effort to fix this problem. NPR's Shankar Vendantam joined our colleague, Steve Inskeep, to talk about it.

STEVE INSKEEP, HOST:

Shankar, welcome back.

SHANKAR VEDANTAM, BYLINE: Good morning, Steve.

INSKEEP: Okay. So what's wrong with a pressure to publish particular findings as you discover them?

VEDANTAM: Well, the problem is that the incentives in science today are all about breaking new ground, and there's much less incentive to making sure that the ground you're standing on is actually solid. Most people think about science as being about breakthroughs, but science really is about the gradual accumulation of knowledge and retesting what we think we know.

I spoke with psychologist Brian Nosek of the University of Virginia. He's been working on this issue for a while.

BRIAN NOSEK: The issues are pervasive in every science. Scientists are rewarded for publication over accuracy. There isn't a culture of replication across most disciplines. We're not rewarded for redoing what someone else does. The main reward is for doing something novel.

INSKEEP: Rewarded for publications over accuracy. I guess we should explain that scientific findings are published in prestigious journals and it's a big deal when your findings are published there.

VEDANTAM: It can make or break your career, and so especially if you're a young researcher the pressure to come up with a publication is extraordinary. Now, Nosek is the editor of a journal called Social Psychology and this is a field where some recent results have been called into question. And the way that journals typically work is the scientist does a study, writes it up, sends it to the journal, journal sends out the study to peer reviewers who review the study.

And if they give the okay, the journal considers it for publication. Nosek thinks this model creates perverse incentives, because if a study confirms an older result, the journals tend to say, well, we knew that already, it's not a novel finding and they're less inclined to publish it. Now, if the replication contradicts an earlier finding, where the journal sends out the study to the peer reviewers, some of the peer reviewers might have been the researchers who conducted the original study and they can now find ways to shoot down the study and reject the study and say we shouldn't be publishing it anyway.

And so what this does is it creates a disincentive for researchers to conduct replications at all.

INSKEEP: And this is how you're supposed to figure out that science is true, by doing the experiment again and again and again.

VEDANTAM: That's exactly right. So in this current issue of the journal Social Psychology, Nosek turns this model on its head. Every article that's in the journal is focused on replicating an earlier study. And as a second innovation, all the replications were accepted for publication before they were conducted, so the peer reviewers only looked at the methodology of the study and if the science were solid, the study was guaranteed publication.

Nosek thinks that this creates the incentives to do the best science, not just come up with the sexiest result.

INSKEEP: Okay. So they're trying to repeat old experiments and see if they discover anything. Did they find anything new?

VEDANTAM: They found lots of things that are interesting, Steve, but let me give you one specific example. In 1999, the researchers Margaret Shea(ph), Todd Patinski(ph) and Alini Ambadi (ph) came up with a classic experiment. They were looking at how stereotypes affect how people behave. Not the people who hold the stereotypes, but the people who are affected by the stereotype. This study was looking at how Asian women perform in math tests.

There's a common stereotype that Asians are good at math and there's also a common stereotype that women are not good at math. When the researchers reminded the volunteers of either their Asian or their female identity, distinct differences in performance emerged. Here's Nosek again.

NOSEK: Asian women who were reminded that they were Asian performed better on the math test. Whereas Asian women reminded that they were women performed worse on the same math test. Something that seems like it shouldn't be flexible, how well we perform in math, is flexible as a function of the identities that we have in mind and stereotypes associated with those identities.

INSKEEP: Wow. That's an amazing finding. But you're saying that under this new model, someone went back and tried that experiment again. What did they find the second time?

VEDANTAM: Well, that's exactly right. Alice Moon(ph) and Scott Reader(ph) at UC Berkley re-did the study in Northern California and they found the effect did not hold. Now, our first reaction is to say, look, the original study was debunked, but it turns out the answer is actually more complicated than that because there was a second replication. At Georgia Southern University, Carolyn Gibson(ph), Joy Losi(ph), and Christine Vitiello(ph) confirmed the original study.

You know, a tendency, Steve, when we hear scientific results is to draw simplistic conclusions. Look, we have a breakthrough. Look, the breakthrough has been debunked. To me, what the replications are doing is open up questions we actually hadn't considered. In this case the fact the replication failed in Northern California but worked in the Deep South raises questions. Is the effect due to demographics? Is it because of geography? Is it something else?

And now we are forced to be curious rather than judgmental.

INSKEEP: I guess this is a reminder that you need to keep an open mind and how subtly you can close your mind without realizing it.

VEDANTAM: And in many ways, Steve, that's the point of science, to recognize that beneath what we think we know there are layers and nuances that are waiting to be discovered.

INSKEEP: Shankar, thanks very much.

VEDANTAM: Thank you, Steve.

INSKEEP: That's NPR's Shankar Vendantam. You can follow him as always on Twitter at HiddenBrain. You can follow this program @MorningEdition and @NPRInskeep as well as @NPRGreene. Transcript provided by NPR, Copyright NPR.