Scientific reproducibility

We were having a moderated discussion today among faculty and graduate students at my institution, devoted to issues of authorship, data manipulation and falsification, and other matters pertaining to scientific publication. Of course, those who fabricate or falsify or plagiarize should be run out of science on a rail, because their actions undermine all that science seeks to accomplish. And, fortunately, the number of con artists in science is low, and the failure of others to reproduce their work ultimately leads to those people being, at best, marginalized, and ideally sanctioned.

But even without outright fraud, there is still the challenge of reproducing others’ work, and having one’s own work be reproduced by others. As Carl Zimmer points out, failure to reproduce findings is a serious problem in the scientific enterprise. While it is nice to score publications in the most prestigious journals (in my field, Nature, Science, and Cell), if no one can reproduce what you’ve done, you’ve accomplished nothing. Nada. Zilch. Unfortunately, the many studies that fail to reproduce a big finding are never as ballyhooed as the finding itself. Rather, it is a slow drip, drip, drip of conflicting data, leading scientists in the field to eventually grumble to each other “I don’t take so-and-so’s paper seriously, even though I have to cite it, because no one has ever reproduced it or built on it.”

In my opinion, one of the biggest causes of a failure to reproduce data is simple sloppiness. Science is, after all, a human enterprise, and the temptation to not examine one’s own data critically enough are great. I know of many first-hand examples where sketchy data were published simply because the student/postdoc doing the work did not take the time to critically analyze all the possible ways in which the data generated might have been spurious rather than real. There should be more scientists who are their own harshest skeptics, because they know the ins-and-outs of their data better than a peer reviewer at a journal ever would.

I tell my students and postdocs of a tale from my own graduate career. I was working on a project that gave very dramatic and exciting results; we were about a month or two away from writing up the story and targeting Nature as the intended journal (not to say whether the paper would have gotten in, but just to give an idea of our assessment of its potential impact). But there was something about the results that never seemed quite right to me–something about the conditions under which we could generate the exciting results, and the conditions under which we could not. I puzzled over it with a more senior graduate student, who suggested a further experiment to test whether the effect we were seeing was arising after the actual experiment was concluded, but before the samples were analyzed. I did the experiment, and sure enough, his intuition was right.

At the time, the disappointment was crushing. A couple of years of work was now of no value, because the finding was an artifact of the experimental technique, and not a reflection of real biology. But it didn’t take too long to realize that I’d just saved myself from myself. It was much better in the long run to have discovered the artifact then and there, then to have the reviewers of the paper figure it out, or worse yet, to have other labs be unable to build on the finding.

Fortunately, this reproducibility requirement is one of science’s great strengths. One-off events (the scientific equivalent of miracles) do not hold currency for long, and over time, incorrect ideas tend to get filtered out. Nonetheless, every unreproducible finding is a great waste of time and resources for the community, and many problems of reproducibility could be circumvented simply by all of us being more critical of our own data.

The greatest compliment I ever received as a scientist was from a senior colleague at a meeting a few years ago. He said (paraphrasing), “We were asked to do some of the experiments that you reported in your recent paper (uprprof: no, I did not review that paper!). We did those experiments and got exactly the same results that you got.” Warmed my heart.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: