Private involvement in education: Measuring Impacts

1. Private involvement in education:Measuring Impacts Felipe Barrera-Osorio HDN Education Public-Private Partnerships in Education, Washington, DC, March 31, 2010

2. The quest • Ideally, we want to know what is the effect of private participation in the provision of education • Does donation into an school produce better learning outcomes of students? • Does a donation that goes to teacher training, increases the pedagogic capacity of teachers and better education outcomes of kids? • Does a private funded computer program induce in the long run higher labor opportunities for individuals? • Does a Public-Private Partnership induce higher learning outcomes?

3. The quest (continuation) • Also, we want to know how we can improve design of programs that will lead to improvement of donations • If the program donate to schools, what is the best strategy to allocate resources: towards management, or towards educational management information systems, or towards teacher training? • Should we donate computers with specific software for education or generic software? • Can we improve efficiency of private donation by inducing involvement of the community?

4. Why is this important? • Impact evaluation can provide reliable estimates of the causal effects of programs • Impact evaluation can potentially help improve the efficacy of programs by influencing design and implementation • Impact evaluation can broaden political support for programs • Impact evaluation can help in the sustainability of successful programs and the termination of programs that are a failure • Impact evaluation can help expand our understanding of how social programs produce the effects that they do

5. These questions are not easy to answer • Impact evaluation is a new name for an old quest: What is the effect of programs? • A good evaluation will have the following characteristics: • A clear definition of the intervention: the concrete objective, what is being modified, the new set of incentives, and to whom the modifications apply. • A description of how the intervention is expected to achieve the final desired outputs: how the intervention will lead to the desired result. • A definition of the identification strategy that allows to attribute causal effects between an intervention and a set of outcome variables.

6. Definition of the program: objectives • Sometimes, the objectives are difficult to measure • For example, a program can have an objective such as “social cohesion” • Solution: formulate a concrete, measurable objective • For example, an objective can be to increase learning outcomes of students • A measurable, concrete outcome can increase accountability of a program

7. Definition of a program: an special challenge for the private sector • Usually, the evaluation is not about the private donation, but about the specific type of intervention • For example, a private individual who donate to schools for teacher training • The evaluation is not about the dynamics of the private sector into the schools, but about the effect of teacher training

8. How the intervention will lead to the desired result. • It is critical to understand the pathways in which the program operate • An example: a computer donation to schools • Is the program affecting the way teachers teach? • Are the students using the computers? • What are the students using the computer for?

9. Channels of transmission • PPP contracts allow more flexibility in the provision of education than in the public sector • Monitor critical areas of flexibility: payroll of teachers, length of shifts, firing and hiring of personnel • Usually private providers in PPP contracts are chosen by an open bid process based on quality criteria • Quality of providers: previous experience

10. Channels of transmission (cont.) • A PPP contract can achieve an optimal level of risk-sharing between the government and the private sector • Very difficult to quantify • The private sector may have higher standards in the delivery of education services • Analysis of educational outcomes like test scores, internal efficiency, etc • PPPs can induce competition in the market for education • Important to track entering and exiting students

11. Attributing causal effects • Why is so difficult to find the effects of a program? • Suppose a computer program: donation of private sector, schools receive them, teachers use them in classrooms, students have direct access to computer and software • Suppose that we follow two strategies: • First, we compare the students that receive the program before and after the program • Second, we compare students that receive the program versus students that did not receive the program

12. The basic intuition: only data before and after the program Y Impact of the program? NO! We need a contrafactual Time t = 0 before program t = 1 after program Intervention

13. We need acomparison group…. Y Impact of the program Control Time t = 0 before program t = 1 after program Intervention

14. The basic intuition 2: only data after the program…. Y Impact of the Program? Control Time t = 0 before program t = 1 after program Intervention

15. We need the right comparison group! Y Impact of the Program? NO! At t=0, two groups were very different… Control Time t = 0 before program t = 1 after program Intervention

16. Some possibilities to find or construct the right control group • Prospective evaluation • Randomization of benefits: a lottery to get benefits • Randomization of entry: a lottery to determine order of entry • Retrospective evaluation • Regression discontinuity analysis: groups are found using an index (e.g., if an individual score above certain number, she receives the benefits) • Differences in differences: information at baseline and follow up for a group that received the program and for a group that didn’t • Propensity and matching estimators: very detail data at some point before the program

17. When do we apply each method? • Ideally, randomization is first best • Individual / geographic randomization: program is a pilot and not universal • Phase-in randomization: program is universal and implementation is done in steps • RD • Program is targeted using an instrument • DD and Matching: • If there is good amount of information and the program is not universal

18. Who should be pay for an evaluation? • If the evaluation produces a public good the evaluation should be finance by governments / multilateral institutions. • If the evaluation produces a private good (e.g., the results are appropriated by the private agent), the evaluation should be finance by the private agent • If so, what are the incentives to provide a real, strong evaluation?