First open-access reproducibility project reveals roadblocks to performing replication studies

"Reproducibility has re-emerged at the forefront of public awareness this week, as the first five replication studies executed by the Reproducibility Project: Cancer Biology (RP:CB) have just been published in the open-access journal eLife.

The project is a collaboration between Science Exchange and the Center for Open Science (COS) to independently replicate key experiments from high-impact, published cancer biology studies. Unlike other assessments of reproducibility, the RP:CB studies and their results are completely open to the public.

The RP:CB studies highlight some of the practical considerations associated with replicating an existing study. For example, the RP:CB studies tackle the questions:

• How do we define 'replicate'?
• What are the minimum requirements for reporting to enable a replication study?
• How much time do replication studies take?
• How much do replication studies cost?

The preliminary results of the RP:CB project, as eloquently summarized in The Atlantic, indicate that replication studies are lengthy and difficult.

Are the resources required for replication studies worth the benefits? Undoubtedly.

High-profile reports, from researchers at Amgen, Bayer, and elsewhere, illustrate the industry’s concerns that this lack of reproducibility might be driving the low success rate of drug candidates. Despite the costs of irreproducibility, researchers have few incentives to replicate studies. Results from replication studies have reduced chances of being published in traditional journals and are rarely prioritized for grant funding. The Reproducibility Project: Cancer Biology is helping initiate a cultural shift in the research community to motivate scientists to perform independent replication.

Our mission at Science Exchange is to facilitate collaboration between the world’s best scientific labs.We hope to play a big part in that cultural shift."