Reproducible Research, Just Not Reproducible By You - The Scholarly Kitchen

"Journals can greatly improve the reproducibility of research by requiring methodological transparency. The print paradigm of journal publishing led us to poor practices in an attempt to save space and reduce the number of printed pages. When trying to cut down an article to reach an assigned page/word limit, usually the first thing to go was a detailed methods section. In a digital era where journals are doing away with page limits, why not add back in this vital information? For a journal that still exists in print, why not require detailed methodologies in the supplementary material? If you have a policy requiring public posting of the data behind the experiments, why not a similar policy for the methods? To their credit, Nature has expanded their methods sections and Cell Pres has implemented STAR Methods, doing away with page limits to create methods sections that are actually useful.

But even with openly available methodologies, we still need to recognize that science is hard. Some research results stem from once-in-a-lifetime events, like a particular storm or celestial event. A hurricane can’t be replicated.

Often, a bench technique will take years to perfect, and even then, some things can only be done by the most skilled practitioners. This can lead to scientific results that are entirely accurate yet very difficult to reproduce. An inability to replicate an experiment can tell us more about the technical skills of the replicator than the validity of the original work. Maybe you can’t reproduce my experiment because you’re not very good at this particular complicated technique that I spent much of my career mastering. Does this mean that my work should be labeled 'invalid'? Mina Bissel from the Lawrence Berkely National Laboratory puts it succinctly:

People trying to repeat others’ research often do not have the time, funding or resources to gain the same expertise with the experimental protocol as the original authors, who were perhaps operating under a multi-year federal grant and aiming for a high-profile publication. If a researcher spends six months, say, trying to replicate such work and reports that it is irreproducible, that can deter other scientists from pursuing a promising line of research, jeopardize the original scientists’ chances of obtaining funding to continue it themselves, and potentially damage their reputations."