Pre-Analysis Plans: Pros and Cons of Limiting Discretion in the Analysis of RCTs

1. Pre-Analysis Plans: Pros and Cons of Limiting Discretionin the Analysis of RCTs Donald P. Green Columbia University

2. Overview of talk • What is a registry? • What are pre-analysis plans? • Arguments in favor of pre-analysis plans • What do pre-analysis plans look like in practice? • Critiques and resistance • A proposal for reducing the burden that pre-analysis plans impose on researchers: “standard operating procedures” • An example SOP • Next steps

3. Distinguishing between pre-registration and pre-analysis plans • Pre-registration describes a study in a public, searchable format but need not make any commitments with regard to analysis • Usually registration occurs prior to the launch of an RCT, but the AEA website allows for retrospective registration • Pre-analysis plans lay out specific coding and analysis rules (e.g., the script to analyze the data) • Prior to the launch of the study or prior to obtaining outcome measures?

4. Social Science Registries: Source APSA Roundtable

5. Arguments in favor of registration • Facilitates meta-analysis • Assembling a more comprehensive set of studies lessens the “file-drawer problem” • Assuming that one can ascertain the findings obtained by registered but unpublished studies

6. Arguments in favor of pre-analysis plans for RCTs (and prospective observational studies) • Limiting discretion (and cherry-picking) • Example: Casey, Glennerster, and Miguel (2012) “Reshaping Institutions: Evidence On Aid Impacts Using A Preanalysis Plan”JPE • Modest overall results but many opportunities for cherry-picking, naïve p-values that change markedly after correcting for multiple comparisons • Simonson: inventory of model specifications

7. Simonsohn’s inventory as applied to Bertrand & Mullainathan (2004)

8. Arguments in favor of pre-analysis plans for RCTs (and prospective observational studies) • Allows readers to more accurately gauge which analyses were planned and which were derived ex post • Does NOT preclude exploratory analysis but makes the distinction between exploratory and confirmatory more transparent

9. More arguments in favor • Forces the researchers (and their funders?) to do more careful thinking up front • Example: Humphreys et al. evaluation of community driven reconstruction in the Democratic Republic of Congo • Authors did a dry run with funders using simulated data so that everyone was clear ex ante of what the analyses would be and how they would be interpreted in light of priors

10. Potentially changes the journal review process • Upcoming issue of Comparative Political Studies, with articles accepted based on their designs, pending results • Similar plan for J. of Experimental Political Science • Lingering issue of monitoring and enforcement • “Badges” issued by third-party groups? Initiative underway coordinated by the Center for Open Science

11. Critiques and resistance • Kills innovation • Produces damaging inflexibility • Creates an unaccountable gatekeeper • Will not stop fraud • Exposes researchers to threat of being scooped • Only meaningful for prospective analyses • Creates unnecessary bureaucratic hurdles for scholars • Replication is a better use of finite resources • Generate inequalities across scholars and methods

12. What do pre-analysis plans look like in practice? • Currently no standard format, although work is underway to create an R package • Browsing the major registries suggests that the level of detail varies widely • Some include code, while others offer a verbal characterization of measurement and modeling choices • Foreshadows the push for SOP

13. A proposal: Standard Operating Procedures • SOP: a set of default procedures that a researcher adopts when there is a gap in the pre-analysis plan (or no pre-analysis plan at all) • Date-stamped and public, with versioning • May be adapted to suit other labs’ substantive and methodological requirements (e.g., our SOP says nothing about biomedical procedures)

14. An Example: the SOP for our lab • Lin, Green, and Coppock (2015) • https://github.com/acoppock/Green-Lab-SOP • Safety net procedures rather than a manual of best practices, in part because best practices are often subject to debate (e.g., optimal covariate selection)

15. SOP contents and omissions • Included: Covariates, noncompliance, attrition, estimation under clustered assignment or assignment with varying treatment probabilities, hypothesis testing • Blinded juries for handling idiosyncratic issues (e.g., covariates discovered after the fact) • Included: Coding and analysis choices that are specific to voter mobilization research • Omitted: Multiple comparisons, multiple treatment arms…our SOP will grow over time

16. Next steps • Reducing the burden of writing pre-analysis plans through defaults and automation • Rebutting common misunderstandings about pre-analysis plans (e.g., exploratory analysis is forbidden) • Continued reflection on sticky institutional questions such as monitoring and adjudication