From impact evaluations to cost-benefit analysis?

1. From impact evaluations to cost-benefit analysis? DEC Course Poverty and Inequality - Module 6: Evaluating the impacts of assigned programs March 28, 2008 Jed Friedman Development Research Group

2. Outline • Conceptual framework • Challenge of accounting for ‘full’ benefits and costs • Evidence C/B education programs • Broader comparison across programs: C/B nutrition interventions

3. Cost benefit analysis • Traditional cost-benefit analysis focuses on ‘efficiency’ arguments: determine the effect of a program φ on total output or income And compare it to its costs C(φ)

4. Cost benefit analysis • Note that: • Distribution of program impact are irrelevant (costless lump-sum transfers/taxes) • Ignore problem of aggregation of individual valuation of the outcomes

5. Cost benefit analysis • Generally ignore distributional aspects of policies (imp. for programs with redistributive objectives or with altruistic motivations) Initial position in the distribution does not matter • Social justice/positive political economy might be interested in: • Gains to specific groups of population at base state (ex. poor) • Proportion of people who gain relative to their base state • Proportion of people who reach an absolute level of outcome (ex. social values of merit goods, health and education)

6. Cost benefit analysis • Full social costs of the program should include the welfare cost of raising funds, generally ignored.

7. General framework • Consider a vector of policy variables φ, with social cost c(φ) • N1(φ) and N0(φ) is the number of people in each state {1, 0} • Assume constant relative prices • Total output: • Now consider a marginal change in φ. What is the corresponding change in output?

8. Conceptual framework (1) persons induced to change state/sector: “entry effects” (2) change outcomes within state/sector (direct + indirect) (3) marginal social cost

9. What does the conventional evaluation parameter measure? • Entry effects generally ignored • Indirect effects are also generally ignored (spillover/substitution, GE on prices) • Does not incorporate costs The conventional parameter ATT does not correspond to a marginal change and includes the gains accruing to participants…

10. What does the conventional evaluation parameter measure? • Entry effects generally ignored (Moffitt 1992, 2003, Heckman 1992): • effect of the program on the rate at which individuals apply or gain admittance to the program. • Direct: policy changes the way applicants are handled • Indirect: policy reform changes the attractiveness of the program

11. What does the conventional evaluation parameter measure? • Indirect effects are also generally ignored (spillover/substitution, GE on prices) • Program evaluations poorly designed to estimate the effect of system-wide structural reform which alters the entire environment.

12. What does the conventional evaluation parameter measure? • However, evaluation parameter still informative under special conditions ..: • No GE effects, with partial participation • When benchmark/counterfactual state is ‘no program’ • .. about the “gross” gains from the existence of a program at a given level φ1relative to shutting it down (all or nothing) • Not appropriate for ‘fine tuning’/marginal changes, nor for economy-wide reforms → “Policy Relevant Treatment Parameters”

13. Challenges in getting good estimates of benefits • Impact may accrue over the life-cycle • Place value on all relevant impacts • Pricing of impact might be distorted by market failures • Distinguishing between private and social costs (central for the efficiency motive for policies) • Selecting the appropriate discount rates to obtain the net present value of future stream of benefits

14. Challenges in getting good estimates of costs • Account for true private and public cost, including distortions costs (and not nec. budget costs) • Assessing the cost when a project is scaled-up (marginal vs average in the presence of economies of scale). Reasonable to assume they are similar in the long run • Distinguish private vs public costs

15. 1. C/B analysis of a voucher education program in Colombia • Vouchers for private secondary schooling in Colombia (PACES) – Angrist et al 2002 • Offered to low income families residing in poor neighborhoods and attending public schools • Value voucher: US$190 per year, ages 13-15, equal annual installments • average monthly fee for private schools US$340 • Average per pupil cost public secondary school slot US$350 • Assignment by lottery: natural policy experiment • Enrolment primary almost universal, enrollment in secondary 73% overall, 55% in poorest quintile

16. 1. voucher education program:impact estimates • Randomized assignment → compare lottery winners/losers (OLS, OLS with controls) recover an intention-to-treat (ITT) parameter • Take-up: model scholarship use. • Lottery losers were awarded other scholarships: alternative financing (24%) • 90% lottery winners used the scholarship • Model selective compliance with 2SLS and recover the average treatment-on-the-treated (ATT) parameters

17. 1. voucher education program:impact estimates • Lottery winners: • No effect on enrollment • 15% more likely to attend a private school • 0.12-0.16 additional grades completed (mainly due to reduced repetition (-5/6%)) • No effects on drop-out rates • 0.2SD higher test scores • 10% more likely to have completed 8th grade

18. 1. C/B voucher program Estimated cost on hh’ld exp. • Total increase in hh’ld expenditures: • Increase in gross fees for winners $52 (greater likelihood of attending private school + spend voucher on more expensive private schools) • Increased opportunity cost: 1.2 fewer hours working, average h/wage 71c: 1.2x.71x48 weeks=$41 lost earnings • Lottery winners receive on average $74 more aid than losers extra net expenditure per lottery: 52+41-74=$19

19. 1. C/B voucher program Estimated cost on gov. budget • Lottery winners receive on average $74 more aid than losers. • Reduced public schools costs: • lottery winners no difference in likelihood of attending school, but 0.14 less likely to attend public school $350*.14~$50 extra public expenditure per lottery: 74-50=$24

20. 1. C/B voucher program Estimated total cost • Estimated public + private costs $19+$24=$43 per year • Administrative + distortionary costs assumed 30% increase in gov.expenditure • Costs incurred over three years

21. 1. C/B voucher program Estimated gains • Gains to lottery winners: • 0.12-0.16 additional grades completed • 0.2SD higher test scores • 10% more likely to have completed 8th grade • Valuing gains (1): • Estimated return to 1 year schooling ~ 10%  program increases winner wages by 1.2% per year • Annual earnings parents $2,400 per worker (with lower education levels) ~ $3,000 • Annual benefits occur at ages 16-60

22. 1. C/B voucher program • Net present value (Berhman, Knowles 2004): • discount rate is 3%, 5% or 10% per annum. • Benefits (of gains in grade completed) and costs over life-cycle discounted to age 13 • Benefit to cost ratios: • at 3% discount: 3.8 • at 5% discount: 2.7 • at 10% discount: 1.4 • Note: • Conservative: Adding gains from completing 8th grade + increased tests scores brings B/C ratio from 2.7 to 3.9 (at 5%) • Heavy discounting (benefits spread over large period)

23. 2. C/B analysis ECD intervention in Bolivia • Daycare, nutritional and educational services 6-72 months old –poor urban areas Bolivia (Berhman, Cheng, Todd 2004) • Child care centers at women’s houses (training +loans/grants to upgrade homes) • Up to 15 children • Longitudinal information on participants + representative sample children same age group in program and non-program areas • Cross-sectional, matching, marginal PSM

24. 2. C/B analysis ECD intervention in Bolivia: costs • Cost: estimated $43/month child enrolled (516/year) • Costs presented with/without 25% upward adjustment for distortionary costs of government financing • Costs calculating assuming that children take part to the program for 3 years (age 2-5)

25. 2. C/B analysis ECD intervention in Bolivia: benefits • Benefit: use impact estimate of: • height gains (2%) • Shorten length time to complete education (1year) • Increase cognitive skills by 5% • to calculate long-life earnings via: • Increasing adult physical stature (1% increase height →2.4% increase wages (urban Brazil, Strauss Thomas 1997) • Increased number grades completed • Increased cognitive skills (conditional on grades completed) (1% increase cognitive skills →0.233% increase earnings, Alderman et al 1996)

26. 2. C/B analysis ECD intervention in Bolivia • Implied benefit to cost ratio range from 1.7 to 3.7 • Largest contribution gains is increasing the number of grades completed

27. 2. C/B analysis across programs:hunger and malnutrition in LDCs • Copenhagen Consensus 2004 • Objective MDG1: halving prevalence underweight rates by 2015 • Income alone insufficient: assuming a stable annual economic growth rate of 5%, growth would result in 0.5% yearly reduction in malnutrition • Not a single intervention: broad overview of different groups of interventions aiming at: • Reducing prevalence low birth weight • Infant and child nutrition and exclusive breastfeeding • Reducing the prevalence of micronutrient deficiencies (iron, vitamin A, iodine, zinc) • Investing in agriculture technology

28. 3. C/B analysis hunger and malnutrition in LDCs • Attach a value to the benefit of averting 1 child from being LBW from not LBW • Ex. Adverse outcomes for low birth weight (LBW: birth weight<2.5 kg) • Excess mortality (40% more likely to die in the neonatal period) • Higher likelihood of being stunted • Cognitive and neurological impairment → lower earnings and productivity as adults • Higher vulnerability to diseases (infant and lifetime) • Intergenerational transmission (LBW more likely to give birth to LBW children)

29. 3. hunger and malnutrition:Accounting for ‘full’ benefits • Ex. 1: Estimating benefits from reducing incidence of LBW on infant mortality • Do not attach value life saved • Study from Guatemala, India: risk neonatal mortality for LBW 4 times higher than 2-2.9kg, 10 times higher than 3-3.5kg • Value: $800 cost save life through measles immunization in early 1990s adjusted for inflation and distortionary taxation=$1,250 • Value to be adjusted for risk death for LBW gives estimated gain of about $93

30. 3. hunger and malnutrition:Accounting for ‘full’ benefits • Ex. 2: Estimating benefits increased lifetime productivity through decreased stunting • UK long-term follow-up study estimates 1,000 gr less than normal birth →loss 0.5SD height (1.6 cm or 1% studies identical twins in the US) • 1% increase height →2-2.4% increase wages (urban Brazil) • Assumed annual earnings $500 a year in constant prices over a work life 15-60 years old • PDV at 5% discount rate of about $99

31. 3. hunger and malnutrition:overview PDV different benefits

32. 3. hunger and malnutrition:overview PDV different benefits

33. 3. hunger and malnutrition:review various types of intervention • Overall benefit per LBW averted: 580-986$ • 60% benefits accounted for increased productivity • Review costs of different types of interventions addressing LBW (ex. treatment women with bacterial infections, treatment women with STD, drugs pregnant women with poor obstetrics history) • Calculate implied benefit to cost ratios ranges

34. 3. hunger and malnutrition:

35. Conclusions • Essential to account for full benefits and costs • Awareness of the assumptions underlying cost-benefit analysis and impact estimates • Parameter generally estimated in the evaluation literature do not lend themselves to marginal changes in policies • Recent literature on distributional treatment effects • Ex-ante simulations. And structural modeling