By Winston Lin (Columbia University) and Donald P. Green (Columbia University)
“Fishing” and “file drawer” problems undercut the credibility of published empirical research. The fishing problem is that with unlimited discretion in data analysis, researchers may consciously or unconsciously tilt a study toward a desired result. The file drawer problem is that entire studies go unreported because they do not reach conventional thresholds for statistical significance. Concerns about these problems have led to a growing use of pre-analysis plans (PAPs), specifying methods of analysis before researchers see the outcome data. PAPs improve transparency, but are time-consuming to prepare and may give insufficient guidance when unforeseen contingencies arise. This article proposes that researchers bolster their PAPs by specifying “standard operating procedures”—default practices to guide decisions that were not anticipated in the PAP. The authors offer an example from their own research group, focusing on randomized experiments. In the foreword, Eric S. Dickson, editor of the Journal of Experimental Political Science, writes: “Our research community has discovered how much scope there is radically to improve the reliability of published research. … Few concrete contributions toward this end that I have encountered are as practically useful, and intellectually interesting, as the authors’ advocacy for Standard Operating Procedures.”
Read more here.
PS: Political Science & Politics / Volume 49 / Issue 03 / July 2016, pp 495-500 / Copyright © American Political Science Association 2016