1 / 48

An Evaluation of the Performance of Regression Discontinuity Design on PROGRESA

Hielke Buddelmeyer Melbourne Institute of Applied Economic and Social Research Hielkeb@unimelb.edu.au. Emmanuel Skoufias The World Bank eskoufias@worldbank.org. An Evaluation of the Performance of Regression Discontinuity Design on PROGRESA. Introduction.

jalene
Download Presentation

An Evaluation of the Performance of Regression Discontinuity Design on PROGRESA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Hielke Buddelmeyer Melbourne Institute of Applied Economic and Social Research Hielkeb@unimelb.edu.au Emmanuel Skoufias The World Bank eskoufias@worldbank.org An Evaluation of the Performance of Regression Discontinuity Design on PROGRESA

  2. Introduction • By consensus, a randomized design provides the most credible method of evaluating program impact. • But experimental designs are difficult to implement and are accompanied by political risks that jeopardize the chances of implementing them • The idea of having a comparison/control group is very unappealing to program managers and governments • ethical issues involved in withholding benefits for a certain group of households

  3. Introduction • Alternative: Quasi-experimental methods attempting to equalize selection bias between treatment and control groups • Objective: Evaluate the performance of the RDD (a quasi-experimental estimator) by comparing the treatment effects estimated using RDD to the experimental estimates. • Focus on school attendance and work of 12-16 yr old boys and girls.

  4. Introduction • This analysis is one of the first to evaluate the performance of RDD in a setting where it can be compared to experimental estimates. The PROGRESA experiment provides a unique opportunity to analyze this issue. • The RDD approach is economical (requires relatively little info), and potentially has many applications in different contexts and types of programs (microfinance, labor market training, education, CCT programs, etc.).

  5. Some Background on PROGRESA • What is PROGRESA? • Targeted cash transfer program conditioned on families visiting health centers regularly and on children attending school regularly. • Cash transfer-alleviates short-term poverty • Human capital investment-alleviates poverty in the long-term • By the end of 2004: program (renamed Oportunidades) covered nearly 5 million families, in 72,000 localities in all 31 states (budget of about US$2.5 billion).

  6. Some Background on PROGRESA • Two-stage Selection process: • Geographic targeting (used census data to identify poor localities) • Within Village household-level targeting (village household census) • Used hh income, assets, and demographic composition to estimate the probability of being poor (Inc per cap<Standard Food basket). • Discriminant analysis applied separately by region • Discriminant score of each household compared to a threshold value (high DS=Noneligible, low DS=Eligible) • Initially 52% eligible, then revised selection process so that 78% eligible. But many of the “new poor” households did not receive benefits

  7. The RDD method-1 • A quasi-experimental approach based on the discontinuity of the treatment assignment mechanism. • Sharp RD design • Individuals/households are assigned to treatment (T) and control (NT) groups based solely on the basis of an observed continuous measure such as the discriminate score DS. For example, B =1 if and only if DS<=COS (B=1 eligible beneficiary) and B=0 otherwise . Propensity is a step function that is discontinuous at the point DS=COS. • Analogous to selection on observables only. • Violates the strong ignorability assumption of Rosenbaum and Rubin (1983) which also requires the overlap condition.

  8. Sharp and Fuzzy RD Designs Propensity score Pr[B=1|DS] ______ Sharp Design --------- Fuzzy design Selection variable DS Regression Discontinuity Design; treatment assignment in sharp (solid) and fuzzy (dashed) designs.

  9. The RDD method-2 • Fuzzy RD design • Treatment assignment depends on an observed continuous variable such as the discriminate score DS but in a stochastic manner. Propensity score is S-shaped and is discontinuous at the point DS=COS. • Analogous to selection on observables and unobservables. • Allows for imperfect compliance (self-selection, attrition) among eligible beneficiaries and contamination of the comparison group by non-compliance (substitution bias).

  10. Evaluating the Performance of RDD • A true evaluation of the RDD approach can only be done if • the experimental estimates are unbiased; and • both estimators estimate the same impact. Important to keep in mind what are the treatment effects/parameters estimated by the experimental approach and by the RDD

  11. Treatment effects estimated from an experimental design • Experimental approach yields an estimate of the Average Treatment Effect on the Treated (ATT) • Depending on the success of the randomized design one may estimate the ATT using CSDIF or 2DIF • There may be heterogeneity of impacts, in which case it is wiser to use a “local” ATT among individuals close to the cutoff score for eligibility for the program (CSDIF-50) • In fact, we estimate the Average Intent to Treat effect (AIT) which provides an estimate of the average impact of the availability of the program to eligible households (in treatment communities) • The binary variable B identifies whether a household has been classified as eligible for the program. • AIT is a lower bound estimate of the impact of the program • ATT=AIT/(% of eligible households actually receiving benefits)

  12. Treatment effects estimated by a RD design • Sharp design: • The treatment effect estimated by a sharp RDD is an Average Local Treatment Effect (to be distinguished from the LATE) • can be estimated by a simple comparison of the mean values of Y of individuals to the left (eligible) and to the right of the threshold score COS (noneligible) • Fuzzy design: • The treatment effect estimated is a local version of the LATE of Angrist et al. (1996).

  13. Kernel Regression Estimator of Treatment Effect with a Sharp RDD where and Alternative estimators (differ in the way local information is exploited and in the set of regularity conditions required to achieve asymptotic properties): Local Linear Regression (HTV, 2001) Partially Linear Model (Porter, 2003)

  14. PROGRESA Evaluation Design • EXPERIMENTAL DESIGN: Program randomized at the locality level • Sample of 506 localities – 186 control (no PROGRESA) – 320 treatment (PROGRESA) • 24, 077 Households (hh)

  15. Table 2: A Decomposition of the Sample of All Households in Treatment and Control Villages

  16. PROGRESA Evaluation Surveys/Data

  17. Issues for consideration • Benchmark/Experimental estimates • CSDIF or 2DIF? • Choice of bandwidth (+/-50, Appendix) • Differences in program impacts across regions (Appendix) • Heterogeneity of program impacts • RDD is a local estimate of program impact • Best to compare it with CSDIF-50

  18. Treatment Effect estimates within a regression framework. Using only the eligible (i.e. Poor) and running the regression in each survey round Then CSDIF-50: equation above estimated on hh within zone of 50 points below threshold.

  19. Table 2: A Decomposition of the Sample of All Households in Treatment and Control Villages

  20. Main Results • Overall the performance of the RDD is remarkably good. • The RDD estimates of program impact agree with the experimental estimates in 10 out of the 12 possible cases. • The two cases in which the RDD method failed to reveal any significant program impact on the school attendance of boys and girls are in the first year of the program (round 3).

  21. Spillover Effects • Table 4 (Groups C vs. D)

  22. Table 2: A Decomposition of the Sample of All Households in Treatment and Control Villages

  23. Evaluation Bias • Table 5 (Group C vs. B)

  24. Table 2: A Decomposition of the Sample of All Households in Treatment and Control Villages

  25. Integrity of control group • Table 6 (Group B vs. D)

  26. Table 2: A Decomposition of the Sample of All Households in Treatment and Control Villages

  27. Using Non-Eligible households from control villages as a comparison group • Table 7 (Group A vs. D)

  28. Table 2: A Decomposition of the Sample of All Households in Treatment and Control Villages

  29. Concluding Remarks • Our analysis reveals that in the PROGRESA sample, the RDD performs very well (i.e. yields program impacts close to the ideal experimental impact estimates). • Critical to be aware of some of the limitations of the RDD approach: • Estimates treatment effects at the point of discontinuity (eligibility threshold). Impact on this group of households may be of less interest than impact of the program on the poorer households

  30. Concluding Remarks • The integrity/quality of the control/comparison group is of vital importance. • Spillover effects do not necessarily lead to a violation of the RDD approach. As long as the local continuity assumption continues to hold even though there are spillover effects the presence of spillover effects would only affect the interpretation of the RD effect: It is the effect of being eligible for program participation in treatment villages net of spillover effects. • Social programs at the national scale may be very difficult to evaluate ex-post because of the difficulty in finding an adequate comparison group

More Related