Beyond Beneficiaries: The Use of Information Systems for Cost-Effective Evaluation

1. Beyond Beneficiaries: The Use of Information Systems for Cost-Effective Evaluation Suzanne Duryea Research DepartmentInter-American Development Bank December 10, 2003

2. Information Systems • Information systems such as SISBEN, CAS, SIPO were developed to better target beneficiaries (predominantly the poor and at-risk populations). • A quantitative score is used as a proxy for poverty and/or the lack of income generating resources within the family. Families scoring below a threshold are eligible to participate in certain programs.

3. Information Systems for Social Programs • Can facilitate certain types of evaluations • With some complementary additions: the databases can form a stronger foundation for evaluation

4. What do we mean by “impactevaluation?” • “What would have happened to the beneficiaries in the absence of the program?” • “What was the effect of the program?”

5. Some options 1) Randomized Designs • Examples: a) Geographic regions are randomly selected for later phase-in of a program (Oportunidades, PRAF) b) Individuals are randomly selected to form a control group (such as school vouchers Colombia) • Randomized evaluations are viewed by the experts as the best option because they solve many of the problems encountered in evaluation. (Duflo and Kremer, 2003) • However, randomization can be politically difficult and is not always possible or appropriate.

6. Some options 2) Ex-Post Evaluation • Aim is to compare outcomes of beneficiaries and non-beneficiaries who were similar BEFORE program participation • Only observe individuals at one point in time -- after enough time has passed for program to have an effect. • Critical assumption: After controlling for observed characteristics, beneficiaries and non-beneficiaries do not differ in unobserved characteristics

7. Some options 3) Before and After Comparisons -- “Double Difference” • Monitor the beneficiaries and non-beneficiaries over the length of time necessary for the program to have effects • I.e., Take baselines for both groups then compare the change in the beneficiaries’ behavior to the change in the non-beneficiaries’ behavior. All time invariant unobservable differences are removed. • Critical assumption: No time varying unobserved differences between beneficiaries and non-beneficiaries.

8. Creation of Counterfactual Group • The most critical part of evaluation design: who can be used as a natural comparison • Information Systems Can Plan an Important Role • There are families who will not quality for programs because they are slightly above the eligible score (threshold). These families look very similar to families with similar scores who have qualified. Discontinuity Design: (Campbell, 1969, Buddlemeyer and Skoufias 2003) • Assign program and then compare outcomes across the families who are close to the threshold.

9. Important Caveats • This is a local average treatment effect: not necessarily comparing the effect on the poorest families • If there are heterogeneous effects at different levels of scores this is only measuring the effect at the cutoff score • Threshold must exist: enforcement of rules necessary.

10. Example of an ex-post evaluation: Superémonos S. Duryea and A. Morrison (2003) Relied heavily on information systems from Instituto Mixto de Ayuda Social (IMAS), Costa Rica: Sistema de Información sobre la Población Objetivo (SIPO) Sistema de Atención a Beneficiarios (SAB)

11. Superémonos • Food coupon (U.S. $30 per month during school) • Targets poor households with school age kids (ages 6 to 18) at risk for poor school attainment using SIPO (proxy means test) • Conditional transfer: families agree that all children will regularly attend school

12. Sistema de Información sobre la Población Objetivo (SIPO) • Targeting mechanism • SIPO score depends on Occupation of household head Material used in house construction Household income Education of household head Net household wealth • Over 250,000 households

13. SIPO and SAB • Very efficient • We provided IMAS with a list of characteristics (requested 3 regions, ages 10-16, started program in 2001) and they provided a cross-referenced list of beneficiaries within minutes • Were in the field with our survey within two months • Very fast in contrast to adding questions to a national household survey

14. First however: we were missing a critical component for evaluation • No information was available in SAB regarding potential counterfactual group (those just above the threshhold) We formed a counterfactual group by conducting surveys in the same neighborhoods and getting families with similar probabilities of participating in Superemonos (propensity matching approach) Cost of data collection for evaluation of Superémonos 1788 householdsunder $30,000 Mexico Progresa 24,407 households $450,000 Argentina Trabajar 2,800 households $350,000 Source: Blomquist 2003

15. Superemonos- IMAS list Not from list number of total observations 746 1032 average age of child 12.90 12.95 percentage female 50.27 48.55 percentage in San Jose 60.19 56.88 percentage in Alajuela 6.03 4.94 percentage in Cartago 33.78 38.18 percentage of mothers with incomplete primary education 36.46 35.76 percentage of households lacking working electricity 4.29 3.88 Summary Information on Survey Samples

16. Survey design and implementation • Tailor-made survey • Pilot tested • Power tested • Sample size: 746 beneficiaries of Superémonos 1,042 non-beneficiaries • Data collected in 3 urban centers: San Jose, Alajuela and Cartago • Collected information on labor force participation, school attendance, school performance, and a host of household characteristics • Approx. cost: $15 per survey (no addresses)

17. Duryea Morrison Strategy: Ex-Post EvaluationInformation System Used to Identify Beneficiaries Only We found that beneficiaries ages 12-15 were 5 percentage points more likely to attend school than non-beneficiaries. In this program we have no reason to think there are serious selection problems. Academic record not considered for eligibility. Ex-post methodology may be appropriate.

18. Alternative Strategy: What we might have doneDouble Difference: Information Systems Used to Identify Treatment and Control Groups • Create comparison group from those who fall just above the threshold on SIPO for Superemonos participation • How: Match 3 non-beneficiaries to each beneficiary (based on SIPO score within same geographic unit) • Follow beneficiaries and non-beneficiaries over time (double difference estimation)

19. Hypothetical Example: Effect of Conditional Transfer Program on Hours of School Activities Double Difference Measures Positive Effect of Program Ex-Post Measure Misses Positive Effect of Program 40 A Double Difference = C - B = 27 - 18 =9 35 amount of change by beneficiaries is larger B 30 A = Ex- Post Non- Difference 25 Beneficiaries C Hours Spent on School Activities 20 15 10 Beneficiaries 5 Before After 0 0 1 2 3 4 5 6 7 8 9 Months of Execution of Project

20. Policy Conclusions and Recommendations • Information systems are critical, both for targeting as well as for evaluation. • Information systems can facilitate cost-effective ex-post evaluations. • Possible to modify information systems to provide more rigorous evaluations at reasonably low cost. Monitoring non-beneficiaries in addition to beneficiaries is the most important modification. Part of evaluation strategy for Chile Solidario. • Encourage stricter enforcement of the implementation of scoring thresholds.