1 / 50

WHY RANDOMIZE

WHY RANDOMIZE. Presented by: Raul Sanchez de la Sierra. March 2014. Why randomize. ?. Because correlation is not causation. Example: Impact of a medicine. Does not take a drug. Does the medicine increase weight??. Takes the drug. Overview. Causal impact and counterfactual

dextra
Download Presentation

WHY RANDOMIZE

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. WHY RANDOMIZE Presented by: Raul Sanchez de la Sierra March 2014

  2. Why randomize ?

  3. Because correlation is not causation

  4. Example: Impact of a medicine Does not take a drug Does the medicine increase weight?? Takes the drug

  5. Overview • Causal impact and counterfactual • Comparison groups • Randomization

  6. CAUSAL IMPACT andcounterfactual

  7. Causal impact of a policy • Change due to the policy

  8. Causal impact of a policy What really happened What would have happened without the policy Real world Imaginary world Pay for performance Traditional wage Counterfactual

  9. Counterfactual Counterfactual: Would would have happened (imaginary) Problem: Will never observe what would have happened How to measure the causal impact of a policy?

  10. Causal impact of a policy What really happened What would have happened without the policy Real world Imaginary world Pay for performance Traditional wage Similar to Counterfactual Counterfactual

  11. Comparison Groups

  12. Selecting the comparison group • Select a group that is exactly likethe group of participants in all ways except the policy Good Bad

  13. Pratham’s Balsakhi program

  14. The intervention • Work with Pratham in 124 Municipal Schools • Hire local women (Balsakhis) from the community • Train them to teach basic literacy, numeracy • Identify lowest performing students • Balsakhi teaches them basic competencies

  15. Possible comparisons • Compare after to before the policy • Compare policy group to other group • Compare change in policy group to change in other group • Other non-experimental methods • Randomized Experiment

  16. Method 1: Compare after to before the policy • Test scores after Balsakhi • Test scores before Balsakhi

  17. Method 1: Compare after to before the policy

  18. Method 1: Compare after to before the policy Improvement could be due to other reasons related to TIME • Problem with this comparison:

  19. Why this may not be a good strategy What would have happened without the policy What really happened True counterfactual After Before Wrong counterfactual Pay for performance Traditional wage

  20. Method 2: Compare to other group Compare test scores of… With test scores of… Children who did not get balsakhi Children who got balsakhi

  21. Method 2: Compare to other group

  22. Method 2: Compare to other group Balsakhi students may be different: Examples: - Poorer - Worst performing • Problem with this comparison:

  23. Why this may not be a good strategy What would have happened without the policy What happened to the other group What really happened Wrong counterfactual After True counterfactual Before Pay for performance Counterfactual Comparison group

  24. Method 3: Difference-in-Differences Children who got balsakhi Compare gains in test scores of… With gains in test scores of… Children who did not get balsakhi

  25. Method 3: Difference-in-Differences

  26. Method 3: Difference-in-Differences

  27. Method 3: Difference-in-Differences

  28. Method 3: Difference-in-Differences Balsakhi students may IMPROVE systematically faster for other reasons than the program: Examples: • They may start from a lower level • Problem with this comparison:

  29. Why this may not be a good strategy What would have happened without the policy What happened to the other group What really happened After Before Pay for performance Counterfactual Comparison group

  30. Method 4: Regression Analysis • Compare test scores at the startand at the endof the program. • For students of the same gender, age (CONTROL) Children who got balsakhi With Children who did not get balsakhi

  31. Method 4: Regression Analysis Balsakhi students may IMPROVE systematically faster than other students of their own gender and age for other reasons than the program: Examples: • They may start from a lower level • Problem with this comparison:

  32. Why this may not be a good strategy What would have happened without the policy What happened to the other group What really happened After Before Pay for performance Counterfactual Comparison group

  33. Method 4: Regression Analysis • Fundamental problem: • Cannot know what makes Balsakhi student different before the program • Never fully credible

  34. Problem with non experiments • SELECTION BIAS

  35. Non-experimental Methods • Instrumental Variables • Regression Discontinuity

  36. Randomization

  37. Non-random assignment HQ Monthly income, per capita 1000 500 0 1457 947 Treatment Control

  38. Randomassignment HQ Monthly income, per capita 1000 500 0 1242 1257 Treatment Control

  39. How does randomization work? 3. INTERVENTION 1. Baseline 4. Endline 2. RANDOMIZED ASSIGNMENT 3. STATUS QUO TARGET POPULATION

  40. Randomization at work: education April 06/07 Tests Tests Aug 05 Initial Test Pay for performance Fixed wage Jun 05

  41. One of the two then gets the treatment. Outcomes are then compared between those that did and did not get treatment Randomization at work: medicine In a randomized trial, two groups are formed through a lottery(to make sure that they are comparable) In a randomized trial, two groups are formed through a lottery(to make sure that they are comparable) One of the two then gets the treatment. Outcomes are then compared between those that did and did not get treatment This groups does not This group gets the treatment How they would have looked without treatment How the treated group looks now

  42. T T T T T C T T C C C C C C T T C C If project goes to easy places The truth T T C T C C C C T T T C If project goes to hard places If lotteries are used

  43. Key advantage of experiments Members of the groups are statistically identical any change can beattributedto the program 43

  44. Method 5: Randomized Experiment • Suppose we evaluated the balsakhi program using a randomized experiment

  45. Impact of Balsakhi - Summary *: Statistically significant at the 5% level

  46. Impact of Balsakhi - Summary *: Statistically significant at the 5% level

  47. What is the impact of this program? Program starts Impact Primary Outcome Counterfactual Time

  48. Impact: What is it? Program starts Counterfactual Impact Primary Outcome Time

  49. Impact: What is it? Program starts Primary Outcome Impact Counterfactual Time

More Related