Pre-analysis plans

1. Pre-analysis plans Module 8.3

2. Recap on statistics • If we find a result is significant at the 5% level, what does this mean? • there is a 5% or less probability it is the result of chance • If we test 10 independent hypotheses there is a 40% chance we will fail to reject the null at the 5% level at least once • i.e. 40% chance find one hypothesis is significant at the 5% level

3. Publication bias: what is it? • Evaluations with positive or negative significant impacts more likely to get published • If enough evaluations are run on a given type of program, some will give positive result by chance • Those with a vested interest may even deliberately run many studies and publicize only some of the results • Published results will suggest the program is effective even if it isn’t

4. Study registers as a solution • All studies are registered before the results are known • Shows how many studies examined a type of program • If 20 studies are started on a given question and only one positive study is published, we may worry the others found no result • Incentives for registration • Some medical journals require registration before study started for publication • New registries for RCTs in social sciences • E.g by the American Economic Association

5. Coverage of registries • Q: what are the advantages of having a registry that only covers RCTs? • Q: Is publication bias only a problem for RCTs? • If not why not have a registry for non RCTs? • Q: Should registration only be allowed before the start of a study?

6. Data mining: what is it? • Looking at the data many different ways, trying to find the result you want • If test impact of program on many different outcomes, some will show positive (or negative) impact by chance • If test impact of program on many different subgroups, some will show positive (or negative) impact by chance • We may be falsely accused of data mining • E.g. we test one subgroup and report the results but readers think maybe we tested many and only reported the one that was significant

7. Preanalysis plans as a solution • Write down in advance how the data will be analyzed • What outcomes are of primary interest • What subgroups will be examined • Register the plan with some external organization • When presenting results, show all those covered in the PAP • highlight any deviations from or additions to the PAP

8. When is a PAP most useful? • When a study has a large number of outcomes with no obvious hierarchy of which are the most important • When researchers know they are interested in differential impact on different subgroups (heterogeneous treatment effects) • When researchers are concerned others will push them to find positive impacts • When want to adjust statistical tests for multiple hypothesis tests

9. Incentives for immunization (no PAP) • Program to increase immunization in rural India with two arms • Predictable and reliable immunization camps • Camps plus 1kg lentils for each shot and set of plates on completion of full immunization • One main outcome (immunization), more than one indicator • Ever received any immunization • % with full immunization • Showing two indicators illustrated why the program worked • Regular camps increased children with at least one shot • Incentive increased % with full immunization • Suggests unwillingness to immunize not main barrier to full immunization, instead incentive helped maintain persistence

10. Results: number of immunizations Source: Banerjee et al 2010

11. Results fully immunized Source: Banerjee et al 2010

12. Community driven development (with PAP) • Evaluation of the impact of a CDD program including: • Quality and quantity of public goods • Social capital • Trust • Participation • Each outcome area had many indicators, eg many public goods • >300outcome indicators in total • Several important subgroups to examine: • Women and youth primary targets • Program implemented in two very different regions

13. When should a PAP be written? • Before the baseline? • Prevents us from learning about what questions have low response rates or inconsistent answers • Before the program starts? • prevents researchers taking advantage of random shocks which happen to mainly impact treatment or comparison • Also prevents researchers thinking of new hypotheses, e.g. unthought-of of negative consequences • Before looking at any data? • Can be useful to look at comparison data to determine appropriate control variables • drop variables where little chance of improvement (e.g. 95% of control already do)

14. What should be covered in a PAP? • Main outcome measures • Which outcome measures are primary and which secondary • The precise composition • Subgroups to be analyzed • Direction of expected impact if we want to use a one-sided test • Primary specification to be used • What control variables to include • How is the outcome variable specified, in logs, changes etc

15. Disadvantages of a PAP • Any analysis that is not included in the PAP has less credibility • Only do a PAP if you have the time to think it through carefully • Sometimes patterns in the data tell consistent stories we never thought of • We might want the flexibility to pursue these • With complex evaluations it can be hard to think through all outcome combinations and how analysis should proceed with each • We may do different secondary analysis if the impact is positive vs. negative • One option is to do PAP in stages, look at some data, then write another PAP