pre analysis plans n.
Skip this Video
Loading SlideShow in 5 Seconds..
Pre-analysis plans PowerPoint Presentation
Download Presentation
Pre-analysis plans

Loading in 2 Seconds...

play fullscreen
1 / 15

Pre-analysis plans - PowerPoint PPT Presentation

  • Uploaded on

Pre-analysis plans. Module 8.3. Recap on statistics. If we find a result is significant at the 5% level, what does this mean? there is a 5% or less probability it is the result of chance

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Pre-analysis plans' - eron

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
recap on statistics
Recap on statistics
  • If we find a result is significant at the 5% level, what does this mean?
    • there is a 5% or less probability it is the result of chance
  • If we test 10 independent hypotheses there is a 40% chance we will fail to reject the null at the 5% level at least once
    • i.e. 40% chance find one hypothesis is significant at the 5% level
publication bias what is it
Publication bias: what is it?
  • Evaluations with positive or negative significant impacts more likely to get published
  • If enough evaluations are run on a given type of program, some will give positive result by chance
  • Those with a vested interest may even deliberately run many studies and publicize only some of the results
  • Published results will suggest the program is effective even if it isn’t
study registers as a solution
Study registers as a solution
  • All studies are registered before the results are known
  • Shows how many studies examined a type of program
  • If 20 studies are started on a given question and only one positive study is published, we may worry the others found no result
  • Incentives for registration
    • Some medical journals require registration before study started for publication
  • New registries for RCTs in social sciences
    • E.g by the American Economic Association
coverage of registries
Coverage of registries
  • Q: what are the advantages of having a registry that only covers RCTs?
  • Q: Is publication bias only a problem for RCTs?
    • If not why not have a registry for non RCTs?
  • Q: Should registration only be allowed before the start of a study?
data mining what is it
Data mining: what is it?
  • Looking at the data many different ways, trying to find the result you want
  • If test impact of program on many different outcomes, some will show positive (or negative) impact by chance
  • If test impact of program on many different subgroups, some will show positive (or negative) impact by chance
  • We may be falsely accused of data mining
    • E.g. we test one subgroup and report the results but readers think maybe we tested many and only reported the one that was significant
preanalysis plans as a solution
Preanalysis plans as a solution
  • Write down in advance how the data will be analyzed
    • What outcomes are of primary interest
    • What subgroups will be examined
  • Register the plan with some external organization
  • When presenting results, show all those covered in the PAP
    • highlight any deviations from or additions to the PAP
when is a pap most useful
When is a PAP most useful?
  • When a study has a large number of outcomes with no obvious hierarchy of which are the most important
  • When researchers know they are interested in differential impact on different subgroups (heterogeneous treatment effects)
  • When researchers are concerned others will push them to find positive impacts
  • When want to adjust statistical tests for multiple hypothesis tests
i ncentives for immunization no pap
Incentives for immunization (no PAP)
  • Program to increase immunization in rural India with two arms
    • Predictable and reliable immunization camps
    • Camps plus 1kg lentils for each shot and set of plates on completion of full immunization
  • One main outcome (immunization), more than one indicator
    • Ever received any immunization
    • % with full immunization
  • Showing two indicators illustrated why the program worked
    • Regular camps increased children with at least one shot
    • Incentive increased % with full immunization
    • Suggests unwillingness to immunize not main barrier to full immunization, instead incentive helped maintain persistence
results number of immunizations
Results: number of immunizations

Source: Banerjee et al 2010

r esults fully immunized
Results fully immunized

Source: Banerjee et al 2010

c ommunity driven development with pap
Community driven development (with PAP)
  • Evaluation of the impact of a CDD program including:
    • Quality and quantity of public goods
    • Social capital
    • Trust
    • Participation
  • Each outcome area had many indicators, eg many public goods
    • >300outcome indicators in total
  • Several important subgroups to examine:
    • Women and youth primary targets
    • Program implemented in two very different regions
when should a pap be written
When should a PAP be written?
  • Before the baseline?
    • Prevents us from learning about what questions have low response rates or inconsistent answers
  • Before the program starts?
    • prevents researchers taking advantage of random shocks which happen to mainly impact treatment or comparison
    • Also prevents researchers thinking of new hypotheses, e.g. unthought-of of negative consequences
  • Before looking at any data?
    • Can be useful to look at comparison data to determine appropriate control variables
    • drop variables where little chance of improvement (e.g. 95% of control already do)
what should be covered in a pap
What should be covered in a PAP?
  • Main outcome measures
  • Which outcome measures are primary and which secondary
  • The precise composition
  • Subgroups to be analyzed
  • Direction of expected impact if we want to use a one-sided test
  • Primary specification to be used
    • What control variables to include
    • How is the outcome variable specified, in logs, changes etc
disadvantages of a pap
Disadvantages of a PAP
  • Any analysis that is not included in the PAP has less credibility
    • Only do a PAP if you have the time to think it through carefully
  • Sometimes patterns in the data tell consistent stories we never thought of
    • We might want the flexibility to pursue these
  • With complex evaluations it can be hard to think through all outcome combinations and how analysis should proceed with each
    • We may do different secondary analysis if the impact is positive vs. negative
    • One option is to do PAP in stages, look at some data, then write another PAP