The impact evaluation of the progresa oportunidades program of mexico
1 / 85

The Impact Evaluation of the PROGRESA/Oportunidades program of Mexico - PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

The Impact Evaluation of the PROGRESA/Oportunidades program of Mexico. Emmanuel Skoufias The World Bank PRMPR April 2007. Why are CCT programs attractive?-1. CCT can provide the foundation of a comprehensive Poverty Alleviation and Social Protection Policy

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

The Impact Evaluation of the PROGRESA/Oportunidades program of Mexico

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

The Impact Evaluation of the PROGRESA/Oportunidades program of Mexico

Emmanuel Skoufias

The World Bank


April 2007

Why are CCT programs attractive?-1

  • CCT can provide the foundation of a comprehensive Poverty Alleviation and Social Protection Policy

    • Induce investments by poor on human capital

    • Can mitigate short-run macroeconomic shocks

    • Can facilitate the phase-out of price subsidies and/or other less effective programs

    • Can serve as a basic social safety net system available to ALL households (complementary to the safety net system accessible through employment in formal sector)

Why are CCT programs attractive?-2

  • CCT can achieve a significant redistribution of income in favor of the poor under tight fiscal conditions

    • Oportunidades was initiated in the context of a short-run economic crisis (1994-95) and was designed as medium-term response to the crisis

    • Oportunidades gradually replaced generalized food subsidies with direct monetary transfers

Why are CCT programs attractive? -3

  • CCT exploit the complementarities among health, education, and nutrition.

    • Coordination: Promote coordination of poverty alleviation efforts among Gov’t ministries (education, health, nutrition)

    • Synergy: simultaneous provision of health, education and nutrition benefits to all the beneficiaries.

Why are CCT programs attractive?-4

  • Co-responsibility: The beneficiaries need to take specific actions or else they do not receive benefit

  • CCT have potential of leading to lasting improvements in the well being of the poor

    • Evidence from rigorous impact evaluation of Oportunidades in Mexico Familias en Accion in Colombias and other countries

Dual Objectives of CCT

  • Long-run poverty alleviation through investment in human capital (i.e., education, health and nutrition)

    • Early Interventions have much higher returns over life-cycle

  • Short-run effect on poverty through cash transfers

How CCT try to achieve their objectives?-1

  • Targeting (geographic/household-level)

    • Oportunidades combines geographic/village-level with household level targeting within villages

  • Simultaneous intervention in 3 key sectors (synergy)

How CCT try to achieve their objectives?-2

  • Conditioning cash transfers to regular school attendance and visits to health centers

  • Cash transfers given to mothers

  • Parallel support on Supply Side (schools & health centers)

Controversial aspects of Oportunidades-1

  • Why grant direct monetary transfers instead of food in-kind, vouchers, or improving supply side of services.

    Distribution of large volumes of food free of charge can inhibit the development of private commercial channels and create unfair competition with marginal producers in the area

  • Why target on the extreme/structurally poor and not include all?

  • Setting new selection criteria: Why not than start from beneficiary lists of existing programs or obtaining the roster of beneficiaries from community proposals

Controversial aspects of Oportunidades-2

  • Creating a single national roster of beneficiaries

  • Giving transfers directly to individuals rather than to communities

  • Having unique, non-discretionary rules for the whole country rather than allowing flexibility for local initiatives and conditions in each state

Controversial aspects of Oportunidades-3

  • Granting benefits to women, given potential family conflicts

  • Having possible impact on fertility (since benefits are linked to family demographics)

  • Size of cash transfer

  • the definition of family co-responsibilities and their certification (might generate additional workload for teachers and medical personnel)

How the controversial aspects of Oportunidades were managed

  • Piloting

  • Expansion of the program in phases

  • Independent and rigorous evaluation (targeting, impact of the program on health, education, nutrition, social relations, women’s status etc.)

  • Monitoring

  • Operational evaluation of the program

  • Cost analysis

Why Evaluation?

  • Economic Reasons

    • Improve design and effectiveness of the program

    • Comparing program impacts allows G to reallocate funds from less to more effective programs and thus to an increase in Social Welfare

  • Social Reasons (increases transparency & accountability)

  • Political Reasons

    • Credibility/break with “bad” practices of past

CCT programs (like Oportunidades) Expanding

  • Brazil: Bolsa Familia=Bolsa Escola, Bolsa Alimentacao & Programa de Erradicaçao do Trabalho Infantil (PETI)

  • Colombia: Familias en Acción

  • Honduras: Programa de Asignación Familiar (PRAF)

  • Jamaica: Program of Advancement through Health and Education (PATH)

  • Nicaragua: Red de Protección Social (RPS)

  • Turkey

  • Ecuador: Bono Solidario

  • Argentina:

  • Bangladesh: Food for Education


  • Large program covering rural and marginal urban areas

    • In 2004: 5 million families or 25 million individuals

    • In 2004: budget of US$ 2.5 billion or 0.3% of GDP

Program Description & Benefits

  • Education component

    • A system of educational grants (details below)

    • Monetary support or the acquisition of school materials/supplies

      (The above benefits are tied to enrollment and regular (85%) school attendance)

    • Improved schools and quality of educations (teacher salaries)

Program Description & Benefits

  • Health and Nutrition Component

    • basic package of primary heath-care services

    • Food support (cash)

    • nutritional supplements: 6 packs/child/mo; 20% of caloric requirements and 100% of necessary micronutrients)

      (The above benefits are tied to regular visits to health-service centers).

    • Information and training

    • Improved supply and quality of health services (medicine availability etc.)

Program Description & Benefits

  • Average benefit received by beneficiary households: or 20% of the value of consumption expenditure before program

  • About half transfer is the cash transfer for food and the rest from the school-related cash transfer


  • EXPERIMENTAL DESIGN: Program randomized at the locality level (Pipeline experimental design)

  • IFPRI not present at time of selection of T and C localities

  • Report examined differences between T and C for more than 650 variables at the locality level (comparison of locality means) and at the household level (comparison of household means)

  • Sample of 506 localities

    –186 control (no program)

    – 320 treatment (receive program)

  • 24, 077 Households (hh)

    • 78% beneficiaries

    • Differences between eligible hh and actual beneficiaries receiving benefits

    • Densification (initially 52% of hh classified as eligible)


  • Step 1: geographical targeting

    • Identify localities of highest marginality

  • Step 2: Household-level targeting

    • Within the localities identified in step 1, conduct a household census and use that census to collect socio-demographic data and information on housing characteristics that is consistent and standard nationwide.

    • Use discriminant analysis on census data to classify household as poor or nonpoor

Table: A Decomposition of the Sample of All Households in Treatment and Control Villages


E (Y)







Using regressions to get 2DIF estimates:

Limit sample to eligible households in treatment and control and run regression:

  • Y(i,t) denotes the value of the outcome indicator in household (or individual) i in period t,

  • alpha, beta and theta are fixed parameters to be estimated,

  • T(i) is an binary variable taking the value of 1 if the household belongs in a treatment community and 0 otherwise (i.e., for control communities),

  • R2 is a binary variable equal to 1 for the second round of the panel (or the round after the initiation of the program) and equal to 0 for the first round (the round before the initiation of the program),

  • X is a vector of household (and possibly village) characteristics;

  • last term is an error term summarizing the influence random disturbances.

Evaluation Tools

Formal surveys

(Semi)-structured observations and interviews

Focus groups with stakeholders (beneficiaries, local leaders, local PROGRESA officials, doctors, nurses, school teachers, promotoras)

PROGRESA Evaluation Surveys/Data

PROGRESA Evaluation Surveys

Additional Info Sources

School & clinic survey

School and clinic administrative data

Nutrition survey conducted independently by Min. of Health and INSP

Student achievement test scores by Min of Education

Record of payments distributed to beneficiary households

Topics of PROGRESA’s Evaluation

Targeting accuracy and impact on poverty

School enrollment, attendance, child labor, achievement scores

Health and utilization of health facilities

Child Nutrition

Household Consumption & Nutrition

Topics of PROGRESA’s Evaluation cont’d

Operation of the program and perceptions of stakeholders

Cost-Analysis and Cost Effectiveness

Status of women, community relations

Adult labor supply, leisure

Impact on short-run poverty

intrahousehold transfers

Conditional Cash Transfers, Adult Work Incentives, and Poverty

Emmanuel Skoufias

The World Bank

Vincenzo di Maro

University College London

November 3, 2006

LACEA meetings Mexico City


  • Means-tested CCT programs increasingly popular

  • Their success at reducing current poverty depends on whether, and the extent to which, cash transfers reduce adult work incentives.

  • In a CCT a cash transfer can be considered as having offsetting income and substitution effects.

    • Income effect: The cash transfer increases household income which in turn increases both Consumption and Leisure (decreases adult labor supply).

    • Substitution effects: associated with meeting the conditionalities of the program. Direct and indirect time costs that depend on the substitution possibilities in the time of family members used in household production.


  • Thus, the response of adult labor supply to CCTs can only be determined empirically.

  • Most welfare programs in the US, have explicit disincentives to work.

  • In most CCT programs the level of benefits received is not affected by the work decisions of hh members or the income level of the hh (once the hh is determined as eligible for the program).

  • However, there are other sources of negative incentives on adult labor supply for both eligible and non-eligible households.

    • Eligible hh may work/earn less so as not to get an income level above the poverty threshold used by the program.

    • Non-eligible hh may work/earn less so as to qualify for the program


  • Empirical evidence on how labor supply responds to transfers is scarce. Sahn & Alderman (1996): the labor supply effect of a rice subsidy in Sri Lanka is significantly large.


Use data from the PROGRESA 1997-99 evaluation sample to investigate:

  • whether eligibility for program affect adult labor force participation (in the treatment villages)

  • whether the presence of the program affects the labor force participation of adults from non-eligible households (in the treatment villages)

  • The effect of eligibility on adult leisure time

  • The impact of the program on poverty measures based on household income.

Participation in Labor Force


Leisure (from time allocation survey in June 1999:

Table 1: Impact (2DIF estimates) on the probability of working among eligible (E=1) adults

Table 2: Impact (2DIF estimates) on the probability of working among NON-Eligible (E=0) adults

Tables 1 & 2 results 2DIF (1):

  • PROGRESA has no significant impact on the labor force participation of adult males and/or females

  • There are no significant/lasting impacts on the allocation of time between salaried and self-employment activities,

  • though the program appears to be associated with a negative, albeit non-significant, effect on self-employment among males

Tables 1 & 2 results 2DIF (2):

  • The presence of PROGRESA has no significant impact on the labor force participation of adult males and/or females from noneligible households (in the treatment villages)

Table 3: Impact (CSDIF estimates) on Leisure of Eligible (E=1) adults

Table 3 CSDIF (1):

  • PROGRESA has no significant impact on the leisure time of adult males and/or females

Impact of PROGRESA on Poverty (1):

  • Results so far: program has no adverse effects on labor income

  • Effects on total hh income and thus poverty depends on the direct and indirect costs associated with participation in PROGRESA.

  • Participation in PROGRESA

    • (a) income losses form children’s work

    • (b) give up benefits from other programs (DIF, Ninos de Solidaridad, Abasto Social de Leche) in additional to the elimination of the Tortilla subsidy

Impact of PROGRESA on Poverty (2):

  • Figures 1 & 2: the effects of PROGRESA on hh income and poverty may not be adequately summarized by the size and incidence of the cash transfers

  • Econometric analysis based on individual and hh income in each round from a variety of sources: labor income, income from self-employment, other income (pensions, rent, and community profits) and government transfers (Ninos de Solidaridad, ININ, PROBECAT, PET, PROCAMPO) +PROGRESA Cash Transfers (from program admin records)

Figure 1 – Mean household income from children

Among beneficiary households with children aged 8-17

Panel a

Mean household TOTAL income from Children

(excluding PROGRESA cash transfer)

Figure 1 – Mean household income from children

Among beneficiary households with children aged 8-17

Panel b

Mean household LABOR income from Children

Figure 1 – Mean household income from children

Among beneficiary households with children aged 8-17

Panel c

Mean household Other Income from Children

Figure 2

Impact on Poverty


Table 4 results: (1)

  • PROGRESA had a significant impact in reducing poverty between November 1997 and November 1999.

    • E.g. using the 50th percentile of the value of consumption per capita as a poverty line, suggests that the headcount poverty rate declined by around 4.88% between November 1997 and November 1998 and by 18.11% in the November 1999 in treatment areas (using as base the 55.44% headcount poverty rate in treatment localities in November 1997).

    • Over the same period, and using as base the corresponding value of the poverty gap and squared poverty gap indices in treatment areas in November 1997, the poverty gap measure declined by 37.40%, and the severity of poverty measure (squared poverty gap) declined by 47.42%.

    • The higher impacts of the program in reducing poverty over time are consistent with the findings of Gertler et al. (2006), who demonstrated that rural households increased their investments in microenterprises and agricultural activities which, improved the ability of households to generate income.

Table 4 results: (2)

  • Estimates are remarkably in line with the estimates obtained using ex-ante simulations.

    • E.g. simulations based on the predicted consumption of each household in the evaluation sample in November 1997 and adding the maximum amount of PROGRESA cash transfers an eligible household could receive assuming full compliance with the program’s requirements (see Skoufias et al., 2001).

Table 4 results: (3)

  • The poverty reduction effects are stronger for the poverty gap and severity of poverty measures, which put greater weight on the poorest of the poor, and our evidence suggests that these estimated poverty effects are robust to the choice of different poverty lines.

  • Figure 3 and Appendix A

Concluding Remarks

  • PROGRESA does not have any significant effect on adult labor supply choices.

  • There is not much evidence to support the hypothesis that PROGRESA beneficiaries use their transfers to “buy” more leisure.

  • Results are reinforced further by the result that PROGRESA leads to a substantial reduction in current poverty.

The Contributions of Program Evaluation-1

  • Program continued and improved

    • Fox administration (begun in 2001) kept and expanded the program

    • Early operations reports in PROGRESA identified implementation issues to be analyzed further (food supplements, intra-household conflict, targeting views)

    • Decision to maintain household targeting in PROGRESA expansion, but to add “self-selection” to administrative selection in urban areas

The Contributions of Program Evaluation-2

  • Program Design improved: Program expanded to urban areas

    • Benefits extended to Preparatoria Secondary level

    • Jovenes con Oportunidades- aims to create income generating opportunities for poor households through preferential access to microcredit, housing improvements, adult education and access to social/health insurance.

Critical Issues to be Resolved on CCT

  • Do CCT programs break the intergenerational transmission of poverty? Need long time-series

  • What is the minimal CCT that may be paid?

    • Oportunidades: size of transfer based on opportunity cost of children (child wage/value of children’s contribution to family…)

Critical Issues to be Resolved:

  • Impact on Children’s Achievement & Learning?

    • Do CCT increase achievement or induce teachers to lower grade-passing standards?

  • Teacher & health worker incentives

  • Quality of Services

Critical Issues to be Resolved:

  • Do CCT generate Program/Welfare Dependency?

    • So far no negative incentive effects on adultwork

    • Transfers & Income generation

  • Exit Rule?

    • Lack built-in flexibility to expand coverage to households falling below poverty during crisis

Final Issue

  • Long-Run Sustainability of Program Budget & Political Economy of Program Support

Thank you

Evaluation Results-Targeting

  • Geographic targeting of the program in rural areas is good

  • Method of selecting poor households within localities is generally accurate (undercoverage of 7% )

  • PROGRESA’s targeting decreases the poverty gap P(1) by 30% and the severity of poverty P(2) by 45%


  • Are more children attending school because of PROGRESA?

  • How much can schooling be expected to increase?

  • Are there more cost effective ways of bringing children to school?

  • Does PROGRESA have more impact in certain grades?

  • Any effects on drop-out rates, grade progression, repetition, reentry?

Evaluation Results: Education

  • PROGRESA has a positive effect the school attendance of both boys and girls in primary and secondary school

    • Boys in secondary: increase by 8 %

    • Girls in secondary: increase by 14%

  • Negative impact on children’s labor market participation (especially boys)

  • No observed increase in the attendance rate (frequency) of children in school.

  • PROGRESA increases overall educational attainment by 10% (and 8% higher earnings)

Evaluation Results: Education

  • Program effective in keeping children in school especially during the critical transition from primary to secondary

  • Less effective in bringing back to and keeping in school children who were out.

  • Earlier entry ages, less grade repetition, better grade progression

  • PROGRESA more cost-effective than increasing access to junior secondary education


  • Does it increase visits to public health clinics?

  • Does PROGRESA have an effect on child health?

  • On the health of adults?

Evaluation results: Health

  • Significant increase in visit rates in PROGRESA communities

    • Increasing in nutrition monitoring visits, immunization rates and prenatal care in 1st trimester (8% increase)

  • No substitution between private and public facilities

Evaluation results: Health

  • 12% lower incidence of illness in children between ages 0-5.

  • Significantly positive effects on adult health


  • Does PROGRESA impact of child growth?

  • Household consumption and food diet?

Evaluation results: Nutrition

  • Significant effect at increasing child growth (1cm higher growth) and reducing the probability of stunting among children 12-36 mo.

  • Household total consumption increases

  • PROGRESA households “eat better” (higher expenditures on fruits, vegetables, meats & animal products)

Cost Analysis

  • Are the administrative costs of PROGRESA high?

  • What are the private costs associated with participation in the program?

  • What might be the indirect effects of the program on the national economy? (e.g. financing of the program)?

Evaluation Results: Cost Analysis

  • For every 100 pesos allocated to the program, 8.2 pesos are administration/program costs.

    • Very low compared to LICONSA (40 pesos per 100 pesos) and TORTIVALES (14 pesos per 100)

  • Targeting and conditioning of the program makeup 56% of program costs (4.6 pesos out of 8.2 pesos)

  • Private costs (3.8 pesos out of 8.2 pesos)

Evaluation Results: Cost Analysis

  • Eliminating distortionary food subsidies and using funds to finance a program like PROGRESA leads to substantial welfare gains.

  • Login