David H. Bailey: Set the Default to "Open": Reproducible Science in the Computer Age

"It has been conventional wisdom that computing is the 'third leg' of the stool of modern science, complementing theory and experiment. But that metaphor is no longer accurate. Instead, computing now pervades all of science, including theory and experiment. Nowadays massive computation is required just to reduce and analyze experimental data, and simulations and computational explorations are employed in fields as diverse as climate modeling and research mathematics. Unfortunately, the culture of scientific computing has not kept pace with its rapidly ascending pre-eminence in the broad domain of scientific research. In experimental research work, researchers are taught early the importance of keeping notebooks or computer-based logs of every detail of their work---experimental design, procedures, equipment used, raw results, processing techniques, statistical methods used to analyze the results, and other relevant details of an experiment. In contrast, very few computational experiments are performed with such documented care. In most cases, there is no record of the workflow process used, the specific computer hardware and software configuration. Often even the source code is not retained. In addition to concerns about reproducibility of results, these regrettable practices ultimately impede the researchers' own productivity. Scientific fraud A related concern is the disturbing rise in outright fraud in scientific research. Perhaps the most significant case is the scandal surrounding the work of Diederik Stapel, a social psychologist at the Tilburg University in the Netherlands. A report by the university found that fraud was certain in 55 of Stapel's publications, and that an additional 11 older publications are questionable. Although the university noted that Stapel is 'fully and solely responsible' for his instances of fraud, the review committee was critical of the larger research culture: 'From the bottom to the top there was a general neglect of fundamental scientific standards and methodological requirements.' In addition, panelists found countless flaws in the statistical methods that were used...  An interesting commentary on this topic has just been published by Robert Trivers, author of The Folly of Fools. He mentions that in an analysis of papers published in the 50 psychological journals, authors whose results were closer to the statistical cut-off point (p=0.05) were less likely to share their raw data...  These and related concerns were the topic of a recent workshop on Reproducibility in Computational and Experimental Mathematics, which was held December 10-14, 2012, in Providence, Rhode Island. Meeting participants included a diverse group of computer scientists, mathematicians, computational physicists, legal scholars, journal editors and funding agency officials, representing academia, Google and all points in between.  While different types and degrees of reproducible research were discussed, many argued that the community needs to move to 'open research,' which means research where widely used software tools are routinely used to (a) fully 'audit' the computationally procedure, (b) replicate and independently reproduce the results of the research, and (c) extend the results or apply the method to new problems.  Workshop participants strongly agreed that cultural changes are required. To begin with, most researchers need to be persuaded that their efforts to ensure reproducibility will be worthwhile, in the form of increased productivity, less time wasted recovering lost data or computer code, and more reliable conversion of results from data files to published papers.  Secondly, the research system must offer institutional rewards at every level from departmental decision making to grant funding and journal publication. The current academic and industrial research system, which places primary emphasis on publications and project results and very little on reproducibility matters, effectively penalizes those who devote extra time to develop or even merely follow community-established standards  ... Finally, standards for peer review must be strengthened---perhaps emulating requirements already routine in the computational biology and genomics community. Journal editors and reviewers need to insist on rigorous verification and validity testing, along with a full disclosure of computational details. Some of this material might be relegated to a website, rather than appearing in the paper itself, but in that case there needs to be assurances that this information will persist and remain