1 / 29

Randomisation Bias and Post-Randomisation Selection Bias in RCTs:

Randomisation Bias and Post-Randomisation Selection Bias in RCTs:. The role of non-experimental methods in the ERA demonstration. Barbara Sianesi Institute for Fiscal Studies September 14, 2006. Randomised Controlled Trials in the Social Sciences: Challenges and Prospects

melia
Download Presentation

Randomisation Bias and Post-Randomisation Selection Bias in RCTs:

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Randomisation Bias and Post-Randomisation Selection Bias in RCTs: The role of non-experimental methods in the ERA demonstration Barbara Sianesi Institute for Fiscal Studies September 14, 2006 Randomised Controlled Trials in the Social Sciences: Challenges and Prospects The University of York

  2. Talk Outline • RCTs are the gold standard in evaluation • BUT not immune from limitations • parameter retrieved • outcomes that can be looked at •  Judicious combination with non-experimental methods can enhance (under suitable assumptions!) what can be learnt from a RCT • Excellent example to illustrate this: ERA

  3. What is ERA • a new package of • support • financial incentives (job retention, training) • to assist ND25+ and NDLP customers obtain, retain and advance in work • evaluated via RA (~14,000 in 6 districts)

  4. Issue #1 • Some eligibles in the ERA Districts • did not reach the decision stage or • refused to take part in research scheme • Experimental contrast  unbiased estimate of ERA impact for those who have reached the RA stage & have agreed to participate.

  5. ERA impact on ERA participants

  6. Staff • discretion. choice of marketing strategy • Customer • finding job unlikely • finding job likely but no desire to stay in touch with JCP • antipathy to government, systems of support, mandatory programmes • resistant to change, …

  7. Staff • Some may not think customer would benefit / be interested. • Some non-ERA advisers may think customers close to job entry may provide quick win.

  8. ?

  9. ? Non-participants

  10. Why do non-participants pose a potential issue? • Would have liked experimental estimate of impact of ERA for the full eligible population in the ERA Districts. • Benchmark: pilot/control area evaluation • In ideal scenario • staff would offer ERA to any eligible (no discretion) • all eligibles would participate (no need for consent)

  11. Why do non-participants pose a potential issue? • Would have liked experimental estimate of impact of ERA for the full eligible population in the ERA Districts. • But: ERA tested only on a subset of ERA eligibles in ERA Districts – the participants.

  12. How to view this • Interested in impact on eligibles but only get impact on participants. “Randomization Bias occurs when random assignment causes the type of persons participating in a program to differ from the type that would participate in the program as it normally operates.” (Heckman and Smith, 95, p.99)

  13. How to view this • Focus on what the RCT consistently estimates (impact on participants) and interested in how well it generalizes to wider population (impact on eligibles). Issue of External Validity. How representative of the full eligible population?

  14. When are non-participants a problem? E(impact | eligibles) What we want

  15. what we get When are non-participants a problem? E(impact | eligibles) = E(impact | eligible partic.) what we want

  16. what we get When are non-participants a problem? E(impact | eligibles) = E(impact | eligible partic.) Prob(eligible partic.) E(impact | eligibles) = E(impact | eligible partic.) Prob(eligible partic.) what we want observed

  17. what we get When are non-participants a problem? E(impact | eligibles) = E(impact | eligible partic.) Prob(eligible partic.) + E(impact | eligible non-part.) what we want observed ?

  18. When are non-participants a problem? E(impact | eligibles) = E(impact | eligible partic.) Prob(eligible partic.) + E(impact | eligible non-part.)Prob(eligible non-part.) what we want observed what we get ?

  19. When are non-participants a problem? eligibles – partic= (nonpartic– partic)Probnonpart bias = p It depends on:  • relative size of eligible non-participant group • whether it is very different from RA group

  20. Descriptive Analysis • Extent of non-participation • How different • observable characteristics • outcomes (non-participants vs. controls) • Non-Experimental Analysis

  21. Issue #2 • Does ERA enhance • hourly wages (productivity) • wage growth (advancement) • employment duration (retention) • Cannot use experimental contrast due to post-randomisation selection bias into employment. ?

  22. To conclude … “Since experiments can answer only a subset of the questions of interest to the evaluator, it remains important to build up the stock of basic social science knowledge required to successfully utilize non-experimental methods, both in themselves and as tools for more extensive analyses of experimental data.” Heckman and Smith (1995, p.95)

More Related