Does Peer Review Identify the Best Papers? A Simulation Study of Editors, Reviewers, and the Scientific Publication Process
by Justin Esarey, Rice University
How does the peer review process, which can vary from journal to journal, influence the quality of papers published in that journal? This paper studies the peer review system by computationally simulating the scientific publication lifecycle: the creation of a paper, its acceptance or rejection through peer review, and the subsequent evaluation of the paper’s quality by the wider field. The paper finds that, when a journal’s reviewers and readers have heterogeneous standards for scientific importance and quality, a majority of papers accepted via peer review will be evaluated by the average reader as not meeting the standards of the journal under any review system that I study. Consequently, general interest journals (even highly selective ones) may be perceived as lower quality compared to narrowly-tailored subfield journals. In addition, systems with more active editorial control over acceptance decisions (e.g., with a high desk rejection rate) tend to result in a journal producing more consistently high-quality publications compared to systems that rely primarily on reviewer up-or-down voting. One implication is that scientists may want to reconsider their attitudes about the prestige and importance of general interest journal publications relative to those in topically and/or methodologically specialized journals.