Active Maintenance: A Proposal for the Long-Term Computational Reproducibility of Scientific Results

Computational reproducibility, or the ability to reproduce analytic results of a scientific study on the basis of publicly available code and data, is a shared goal of many researchers, journals, and scientific communities. Despite great strides toward realizing that goal, code that successfully executes on the day it was deposited in an archive may become obsolete within only a few years, hindering reproducibility. We document this problem with a random sample of studies posted to the Institution for Social and Policy Studies (ISPS) Data Archive. We encountered errors in seven of 20 studies illustrating that, even in the relatively short 10-year period, code that originally allowed for computational reproducibility broke down. We argue that the commitment to reproducible research should not end on the day the study is deposited in the archive. In line with similar proposals for the long-term maintenance of data and commercial software, we propose that researchers dedicated to computational reproducibility should have a plan in place for “active maintenance” of their analysis code. We offer suggestions for how data archives, journals, and research communities could engage in active maintenance of scientific code and data.