Impact Evaluation: The case of Bogotá’s concession schools Felipe Barrera-Osorio World Bank

1. Impact Evaluation: The case of Bogotá’s concession schoolsFelipe Barrera-OsorioWorld Bank October2010

2. Outline • Hypotheses • Detour: Evaluation of Programs • Empirical strategy • The data • Results

3. Why concessions (a PPP) may increase quality of education? • An application of an already proven pedagogic model • Concession schools are financially stable, ensuring the stability of the pedagogic model. • Freedom to choose (and fire) teachers and management staff • Better infrastructure • Concession schools work actively with the families and the community • “Affection deficit” and family problems are important issues in these communities • Nutrition is an important variable in educational outcomes

4. Three hypothesis • Dropout rates are lower in concessions schools than in similar, public schools. • Nearby schools have lower dropout rates than public schools outside the influence of concessions. • Test scores are higher in concessions schools than in similar, public schools.

5. Detour: Evaluating Programs

6. What is Impact Evaluation? • Impact evaluation is a set of methods to identify and quantify the causal impact of programs • For example, What is the effect of a concession schools on drop-out rates and test scores?

7. Why are these effects difficult to estimate? • The basic question: What would it happen in the absence of the program? (e.g., what is the contrafactual?) • We need the same individual with and without the program • For example, we need to observe John attending one concession school, and (the same) John attending a public school • …but it is impossible to observe the same individual in both states! • Solution: “build” the correct contrafactual • Find individuals who do not have the benefits of the program, but are very similar to the ones that have the program • For example, find David who is very similar in all characteristics to John…

8. What is the problem? • In short, the problem is to find the correct comparison (control) group • The control group and the treatment group should have the same characteristics, observable and non-observable, before the beginning of the program • External factors will affect in the same way control and treatment group • An, usually, people self-select into programs, and therefore, the beneficiaries are (with high probability), different that the ones that did not enter into the program

9. The basic intuition: only data before and after the program Y: Test scores Impact of the program? NO! We need a contrafactual Time t = 0 before program t = 1 after program Intervention

10. We need the right comparison group Y: Test scores Y Treatment Impact of the program Control Time t = 0 before program t = 1 after program Intervention

11. The basic intuition 2: only data after the program…. Y: Test scores Y Treatment Impact of the Program? Control Time t = 0 before program t = 1 after program Intervention

12. We need the right comparison group! Y: Test scores Y Treatment Impact of the Program? NO! At t=0, two groups were very different… Control Time t = 0 before program t = 1 after program Intervention

13. Four possibilities to find or construct the right control group • Prospective evaluation • Randomization of benefits • Randomization of entry (phase-in approach) • Randomization of information (encouragement design) • Retrospective evaluation • Regression discontinuity analysis • Instrumental variables • Differences in differences • Propensity and matching estimators

14. Randomization • Lottery among individuals will separate the sample between winners and losers • Model of “over-subscription” • A lottery will create homogenous control and treatment group: they will be very similar in all characteristics, observable and non-observable.

15. Regression discontinuity • When to use this method? • The beneficiaries/non-beneficiaries can be ordered along a quantifiable dimension. • This dimension can be used to compute a well-defined index or parameter. • The index/parameter has a cut-off point for eligibility. • The index value is what drives the assignment of a potential beneficiary to the treatment. (or to non-treatment)

16. Intuition • The potential beneficiaries just above the cut-off point are very similar to the potential beneficiaries just below the cut-off point. • We compare outcomes for individualsjust above and below the cutoff point.

17. Non-poor Poor

18. Treatment effect

19. Difference in differences • Estimating the impact of a “past” program • We can try to find a “natural experiment” that allows us to identify the impact of a policy. For example, • An unexpected change in policy could be seen as “natural experiment” • A policy that only affects 16 year olds but not 15 year olds • We need to identify which is the group affected by the policy change (“treatment”) and which is the group that is not affected (“control”). • The quality of the control group determines the quality of the evaluation.

20. Intuition • Find a group that did not receive the program • …with the same pattern of growth in the outcome variable before the intervention • The two groups, treatment and control, have the same profile before the intervention

21. Intuition: when it is right to use DD? Y: Test scores Impact of the Program Control Time t = -1 t = 0 t = 1 after program Slopes, before intervention (between -1 and 0) were equal Intervention

22. Intuition: when it is right to use DD? Y: Test scores Impact of the Program? NO: wrong comparison group Control Time t = -1 t = 0 t = 1 after program Slopes, before intervention (between -1 and 0) were different Intervention

23. Propensity and matching estimation • Find the comparison group from a large survey • Each treatment will have a comparison that has observable characteristics as similar as possible to the treated individuals • This method assumes that there is not self-selection based on unobservable characteristics: • Selection is based on observable characteristics

24. How is this procedure done? Two steps • Estimation of the propensity score: The propensity score is the conditional probability of receiving the treatment given the pre-treatment variables. • Estimate the probability of been treated based on the observable characteristics • Estimation of the average effect of treatment given the propensity score • match cases and controls with exactly the same (estimated) propensity score; • compute the effect of treatment for each value of the (estimated) propensity score • obtain the average of these conditional effects

25. …Back to Concessions

26. Empirical Strategy for Concessions: propensity and matching estimation • Propensity score estimation: • “Find” controls that have similar characteristics as the treatment one • Matching estimation: • Once the controls are found, estimate impact:

27. The data • C100 and C600: Rich information about schools • For 1999 and 2003: Administrative personnel, number and level of education of teachers, physicians and psychologists, students by grade and by age, students who failed a grade and dropped out. • For 1999: infrastructure variables like computers, rooms, labs, etc. • ICFEX 2003: Standardized test scores • Test scores at the individual level • Some information at the individual level

30. Flavor of the (strong) result Concessions reduced the dropout rate significantely Concessions reduce the jump in the desertion in grade 6

31. Test scores: Public schools have lower test scores Concessions and public non-concessions are “similar”

34. Dropout Results: Probit estimates

35. Dropout results: impact • Matching: • 10 nearest estimators, common support, balance groups • Direct Effect: reduction in 1.7 points dropout rates • Indirect Effect: reduction in 0.82 points

36. Test Results: Probit estimates

37. Test results: impact • Matching: • 10 nearest estimators, common support, balance groups • Effect over math test scores: improvement of 2.4% • Effect over language test scores: improvement of 4%

38. Some conclusions (and some questions) • Concessions delivers: • Strong evidence of differences in dropout rates • Some evidence of impact over test scores • Colombian government is implementing the II stage of Concessions • What makes concessions different from public? • Is it the work with families and community? • Freedom of hiring teachers? • Pedagogic method?

39. A discussion about the data • Limits of the data: • Not a randomized experiment • Still some source of bias • Ideally, we want to have more hard-evidence • Randomization of students who applied to concessions

40. Why is this important? • Impact evaluation can provide reliable estimates of the causal effects of programs • Impact evaluation can potentially help improve the efficacy of programs by influencing design and implementation • Impact evaluation can broaden political support for programs • Impact evaluation can help in the sustainability of successful programs and the termination of programs that are a failure • Impact evaluation can help expand our understanding of how social programs produce the effects that they do