Data Access and Research Transparency Initiative (DA-RT)

Over the past few months, APSA members have participated in an extensive discussion of the values, risks, and implementation strategies involved in the desire to enhance the transparency and interpretability of empirically-based scholarship.  The immediate impetus, and shorthand, for this discussion is the Data Access and Research Transparency (DA-RT) initiative.  We want first to acknowledge and thank our many colleagues who have engaged with this initiative; the broad and respectful involvement indicates the shared concerns of APSA members and signals how seriously political scientists take our discipline, association, and professional responsibilities.

We are writing to express our views and place them in context in order to contribute to this discussion. Under long-standing association practice, editors of our journals have wide discretion in editorial standards and procedures, including a decision on whether, when, and how to implement strategies for promoting access to evidence and research interpretability.  Editors of some section journals and the APSR editors have issued guidelines for implementation (the draft APSR guidelines are here, in the section on “Data Access, Production Transparency and Analytic Transparency).” Editors of other association-wide and section journals have decided not to adopt the proposed standards at present. This variation is consistent with current practice, and it will provide useful information that will help association members evaluate innovations, address implementation problems, and assess the value of research transparency.

In the following paragraphs, we provide some background for our support for DA-RT implementation in the APSR in January 2016, and describe plans for review and action over the next year by the APSA Council on DA-RT and related issues of scholarship and publication.

An Array of Issues:  It is no surprise that so many people have worked to develop, modify, contest, and query the implementation of strategies for transparency and interpretability. These strategies raise issues ranging from epistemological differences in understanding “truth” or “fact,” to protection of human subjects, professional advancement, workload, subfield development, privacy, editorial authority, and implementation details.

To some political scientists, implementing guidelines for research transparency is essential to ensure our legitimacy, internally and externally, as scholars who seek the best evidence and interpretations and who fully engage with other researchers and the public.  To other political scientists, who share the commitment to the highest standards of research integrity and engagement, implementing these guidelines without further deliberation and revision risks making qualitative research even more difficult to pursue and publish, provides disincentives to new scholars to venture into qualitative terrains, and for some scholars, is an inappropriate model for scholarly engagement, evaluation and communication.  To yet others, the devil is in the details; some versions of research transparency and interpretability seem valuable or at least acceptable while others could harm valued research endeavors.

The first question, then, is why engage with this initiative at all?

Goals of Strategies for Transparency and Interpretability: We see three reasons to implement some version of DA-RT. The first is to protect scholarly integrity. Research is emerging across many disciplines on the difficulty of replicating or interpreting earlier research findings. Some of these findings are in canonical articles frequently taught to students; some involve drugs that can affect the course of a disease or even a life; some influence governments’ policy choices. Examples outside political science include psychology, economics (here and here) and medical research.  Data transparency can also shield students from undue influence by  powerful professors, and permit examination of claims about intentional or inadvertent plagiarism or about assertions that go beyond an evidentiary base.

Within political science, scholars have demonstrated the “.05 effect,” our version of the classic file drawer problem (here  and here).   Other political scientists have also pointed to additional values in reanalyzing  or reinterpreting the evidence that lies behind empirical results.

These concerns are not limited to research in quantitative disciplines or subfields.  Within qualitative or historical research—again looking only outside political science—disputes over books by Michael Bellesiles and  Alice Goffman (here  and here) suggest the worth of clear guidelines about how relevant portions of archives, field notes, or interviews should be made available to other scholars. Suggestions of cherry-picking a few quotations from interviews, a few sentences from archives, or a few incidents in field notes can similarly undermine confidence in research-based publications (for example, see here).  It is in the interest of all political scientists to avoid unnecessary controversies and embarrassment to ourselves and our discipline.

A second reason to implement some version of DA-RT is positive; scholars and scholarship benefit when there is shared engagement around a set of evidence and plausible interpretations.  One of us (Hochschild), for example, has learned a lot from students who have reanalyzed interview transcripts from What’s Fair?.  For other examples, see here (gated) and here.

A third reason to support provision of enough evidence to enable others to replicate one’s results or evaluate one’s interpretation is to respond to external signals.  The social sciences, like the natural sciences and medical research, are facing rising expectations for accountability. Public and private funders (such as the National Institutes of Health,  National Science Foundation, and  The Economic and Social Research Council,) increasingly require grantees to make their evidence publicly accessible, including, albeit more slowly, qualitative research (see Germany’s RatSWD).  Publishers such as Cambridge University Press, Oxford University Press, and Sage are similarly moving to support greater transparency of evidence and analytic strategies.  Our legitimacy as scholars and political scientists in speaking to the public on issues of public concern rests in part on whether we adopt and maintain common standards for evaluating evidence-based knowledge claims.

What have political scientists done so far? If one agrees that there are good reasons to engage in some fashion with the commitment to research transparency and interpretability, the next question is what have political scientists done in this arena to date?  Depending on one’s perspective, the answer is a lot, some, or nowhere nearly enough.

After appointing a drafting committee in 2010, the APSA Council approved in 2012 new language on transparency and replicability for the APSA Guide to Professional Ethics in Political Science (see pp. 8-10).  The new guidelines were made public.[1]  Since then the DA-RT committee, working on its own, has held meetings, sessions at annual conventions, and a conference (details are here). In 2014, editors from some APSA section journals and the APSR made a commitment to implement strategies for research transparency.  Since then, editors of some non-APSA political science journals have implemented guidelines and, after convening working groups and consulting with its Editorial Board to help refine their own version, the APSR editors have recently promulgated draft guidelines.  These will go into effect on a trial basis in January 2016.

Skeptics, reformers, interested bystanders, and opponents have also published, met, and discussed their understanding of DA-RT and its possible implications, especially for the many varieties of qualitative research. Publications include an Editor’s Statement in Perspectives on Politics, a newsletter of the section on Qualitative and Multi-Methods Research, and a recent petition of APSA members.  A new website offers “a place to deliberate about standards of data access and research transparency in political science.” Blog posts, tweets, and comments on all possible sides abound. Questions remain about what issues have not been fully addressed by proponents, how DA-RT might evolve over time, whether some methodological or epistemological approaches will be disproportionately advantaged or disadvantaged, and who should make crucial decisions.  Those who believe that implementation of transparency guidelines should be postponed until after further deliberation are developing a method for consultation with members of APSA sections.

For some scholars, further discussion in the absence of much concrete experience in implementing research and analytic transparency in qualitative research is unnecessary and perhaps detrimental; for others, it is essential.  We take the concerns very seriously and have spent many hours deliberating on the best way for the APSA to move forward on this issue.  Here is our view:

We Support DA-RT Implementation in the APSR in January 2016:  We have three reasons, beyond noting the general move across the social sciences toward research transparency and interpretability:

  • The APSR editors made a public commitment to implementation several years ago, have reiterated it with Council support, and feel honor-bound to carry out their commitment. They are our colleagues; they provide an important public good by editing the journal; they are internally diverse; they have thought long and hard and consulted widely about DA-RT.  They recognize and respect the varying implications of DA-RT implementation for different research communities.  They deserve the opportunity to fulfill their commitment, and by long-standing APSA practice, editors  of association-sponsored  journals have discretion to set policy for their journal.
  • Much of the discussion of DA-RT is necessarily based on expectations of or projections about how implementation will actually work in various journals—necessarily, because we have little experience with transparency of evidence or process on the qualitative side. Implementation in the APSR for the remainder of the term of this set of editors (through July 2016) will provide some information about what works there, what needs refinement, what does not work. Setting aside normative or epistemological rejections of the concept of transparency or interpretability, the discussion of DA-RT’s role in the APSR and other association journals will be greatly enhanced if discussants can refer to some actual practice.
  • The current APSR editors fully intend implementation of the new guidelines to be measured and deliberate, and they have pledged to work with authors to ensure that concerns are resolved.

What Comes Next?  In our view, it is becoming increasingly important for the Council to establish an explicit policy with regard to Council oversight of APSA-sponsored journals.  That policy could range from 1) “each editor or editorial team has complete autonomy” to 2) “journals published under the auspices of the APSA must adhere to the following guidelines [to be filled in],” to 3) “journals published under the auspices of the APSA may [or must] adhere to the following guidelines [to be filled in], with explicit Council review in XX years.” Which option is best is not yet clear, and whatever policy the Council adopts should be the outcome of substantive engagement and deliberation among members of our profession.

We have appointed a publications policy committee comprised of current APSA Council members, who will work with the extant membership-based Publications Committee to address this growing need for Council policy on publications.  The committee has been asked to propose a set of guidelines for the Council to discuss at its meeting in April 2016.  The proposed policy guidelines will be disseminated widely and posted on APSAnet.org and www.politicalsciencenow.com. The drafting committee will solicit members’ comments. The proposed guidelines will be further discussed and voted on at the Council’s meeting in September 2016.  Review of the current implementation of DA-RT guidelines in the APSR will be part of that discussion.

In addition, over the next year, the President-Elect of APSA will propose, subject to the Council’s approval, a presidential Task Force on Professional Ethics.  It will continue and develop current discussions by APSA committees of issues in professional ethics, including but not limited to matters of research transparency and interpretability in APSA-sponsored publications. As this proposed Task Force and its agenda are developed, we will solicit additional insights and topics of concern from the membership.

We welcome responses and comments to this letter below, and encourage continued conversations about how best to achieve the goals of research transparency and interpretability in ways that are responsive to the entire APSA membership.  Disagreements will surely persist, but we hope that decisions about policies such as DA-RT will be based on deliberation, mutual trust, and shared accommodation; that is the mark of a professional association of which we can all be proud members.

Jennifer Hochschild, President

David A. Lake, President-Elect

Rodney E. Hero, Immediate Past President

[1] In the member newsletter, APSA website, and PS: Political Science and Politics:  1) Minutes of the Council Meeting,  45 (3):589;  46 (1):197; and 46 (3): 707;  2) Openness in Political Science: Data Access and Research Transparency 47 (1):19; and 3)  2014 Report of the Committee on Professional Ethics, Rights, and Freedoms 48 (1): 210-11.

Dart Discussion Resources from APSAnet.org

2 Comments

  1. It is regrettable that APSA leadership issued a statement endorsing DA-RT at a time when a broad and very constructive discipline-wide conversation is only now beginning. We think this is premature and, as such, risks preempting rather than facilitating the kind of “deliberation, mutual trust and shared accommodation” that the presidents want to promote.

    We do not see much indication in this statement that the presidents have engaged the legitimate concerns raised (e.g., in the thoughtful QMMR newsletter or in the recent petition) by our colleagues who confront very real conundrums in reconciling DA-RT transparency standards with ethical commitments to the protection of human subjects, the ability to publish out of original data sets without being required to share them too early, the logistical burden placed particularly on young scholars and by scholars at non-elite universities, and other concerns. Instead, we note with disappointment that the justification offered for embracing DA-RT relies almost exclusively on a few references to egregious cases of flagrant ethics violations that all of us can of course agree are to be avoided and that in any event are well covered by the existing Ethics guidelines. If there is one thing that the discussion of DA-RT has already made very clear, it is that the issues are a great deal more nuanced and complex than that.

    Fortunately, there is a substantive on-going process to deliberate on these issues, investigating what the guidelines should be for issues such as the incentives for young scholars to create their own datasets, and the kinds of research materials, if any, that scholars engaged in immersive research may be required to deposit.

    Our approach to this issue is thus different from that of the presidents. Rather than take sides in a process that is underway, we reiterate our request that the journals hold off until some of these issues can be deliberated more adequately in the profession. The Qualitative and Multi-Method Research organized section is launching a wide-ranging and structured deliberation within the profession and we look forward to discussing the results at the next annual meeting. We believe that any significant implementation before such issues have been more fully deliberated is premature and potentially divisive.

    Please visit our website http://dialogueondart.org/

    Nancy Hirschmann, Mala Htun, Jane Mansbridge, Kathleen Thelen, Lisa Wedeen, and Elisabeth Wood

  2. You know how they have locked repository rooms for data? Or special archives? If someone wanted to see each and every page of a certain interview or my entire corpus (because why? – but if you want to read thousands of pages have at it) – they are free to apply to me with their reasons, and then come and sit in my office with my computer and go through my notes. I actually would not mind this. You can’t copy them, you can’t use them for your own work unless I already published certain quotes or you’re my collaborator etc. but you can see whatever you want if you doubt that my quotes are true and me sending you an extract isn’t going to cut it. Most of the time an author in my field will read my written work and will probably know whether it jibes with what s/he’s seen in the field – I’ve never had another field researcher question my archival quotes because they’ve heard similar things themselves. They will question my interpretation or theoretical conclusions, but not the actual quotes.

    I think qualitative data is personal, its not a public data set but a proprietary one – I write a lot of notes to self in there, to remove them is onerous. Some of them are my comments on the veracity and reasonableness of my interviewee’s comments – I don’t want these public because this is before I have converted them into academic speak, while they are just conversations with myself. I do note my concerns about veracity in my written work, I note the possible alternative explanations as well, thats just good analysis. My interviewees are assuming confidentiality more than anonymity. The anonymity only emerges in the published work, the confidentiality is required for the data.

    Even with archival data – why should I spend months in the dusty dirty sweaty archives, and then place all my material into a repository? The archives are publicly accessible, I have noted all the call numbers and references in my work – if you want to check my cites from my notes, come to my office and read. If you want the sources for your own work, then use the call numbers that I have so helpfully provided for transparency, go to the field, into the archive, call it up, wait a day till they bring you the wrong reference, send it back, call it up again, sit there with your elbows deep in red dust, and read.

    Goffman’s problem did not require notes to identify. It was partly ethical (should she have been driving that car), partly coming from readers knowledge of police procedure (is it legal for hospitals to inform police about visitors), partly coming from internal consistences between her article and the book etc. There is plenty of controversy without her notes. She should not have destroyed the notes. But even if she still had them, why does everyone have a right to trawl through all her notes? With this kind of controversy, if she had her notes, she could have produced key pages to address it.

1 Trackback / Pingback

  1. APSA leadership: APSR will implement DA-RT in Jan. 2016 | Dialogue on DA-RT

Comments are closed.