November 5 2009
1 / 34

November 5, 2009 - PowerPoint PPT Presentation

  • Uploaded on

The PNPM- Generasi Project One-Year Impact Evaluation Preliminary Findings Presented by: Susan Wong, EASER, The World Bank Ben Olken, M.I.T. Department of Economics. November 5, 2009. Structure of Today’s Presentation. Indonesian context Description of PNPM-Generasi project design

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'November 5, 2009' - tekla

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
November 5 2009

The PNPM-Generasi ProjectOne-Year Impact Evaluation Preliminary Findings Presented by:Susan Wong, EASER, The World BankBen Olken, M.I.T. Department of Economics

November 5, 2009

Structure of today s presentation
Structure of Today’s Presentation

  • Indonesian context

  • Description of PNPM-Generasi project design

  • PNPM-Generasi implementation update

  • Impact evaluation design and analysis

  • Preliminary findings of one-year PNPM-Generasi implementation

    • Impact on health indicators

    • Impact on education indicators

    • Effect of community incentives

    • Heterogeneity

  • Conclusion, Further Analysis, and Next Steps

Indonesian context
Indonesian Context

  • Remarkable progress in poverty reduction and key human development indicators over past few decades.

  • Economic growth, increased access to education and health services, expanded anti-poverty programs, and improvements in infrastructure have all helped to reduce poverty.

  • Poverty headcount is 14.1% in 2009.

  • However, 32.5 m Indonesians still live below poverty line & one-half of all HHs remain clustered around national poverty line. High vulnerability.

  • Regional disparities with Eastern Indonesia lagging behind other parts of country, esp. Java.

  • Human development areas require more attention: child malnutrition, infant and maternal mortality, primary to secondary school transition, access to safe water and sanitation. Quality of services also a major concern.

Two pilot projects
Two Pilot Projects

In 2007, GoI started two pilot projects:

  • Household CCT – the traditional model

    • Quarterly tranches of cash transfers

    • Statistically identified 633,000 poor households with children

    • Currently in 13 provinces, 70 districts, 629 municipalities

    • Annual budget of IDR 1.2 trillion (@USD 120 m)

  • PNPM-GenerasiCommunity Block Grants

    • Addresses the same health and education indicators, but at the community level

    • 5 provinces, 21 districts, 178 subdistricts

    • Covering approx. 3.1 million beneficiaries

    • Total budget from 2007-2009 of @USD 107 m

Wb support role
WB Support Role

  • Collaboration between PREM, HD, and Social Development Units in Indonesia

  • Provide TA for design, implementation and evaluations of two pilots.

  • Portion of KDP/PNPM WB loan funds support the PNPM Generasi pilot in 5 provinces.

The pnpm generasi project
The PNPM-Generasi Project

  • Objectives: Accelerate the achievement of MDGs

    • Reduce child mortality

    • Reduce maternal mortality, and

    • Ensure universal coverage of basic education

  • Conditionalities: Places incentives on communities to identify problems and seek solutions to improving 12 health and education indicators

The pnpm generasi project1
The PNPM-Generasi Project

Community incentives:

Version A: 20% of year 2 allocation depends on previous year’s village performance

Version B: Village performance not linked to fund allocation. Otherwise identical to Version A.

Implemented through KDP/PNPM-Rural with:

Same management structures at the national, provincial, and district levels as PNPM-Rural

Facilitated by 2 subdistrict facilitators

Pnpm generasi project design
PNPM-Generasi Project Design

12 indicators: communities are required to work on the same indicators as HH-CCT (Program Keluarga Harapan)


  • Four prenatal care visits during pregnancy

  • Taking iron tablets during pregnancy

  • Delivery assisted by trained professional

  • Two postnatal care visits

  • Complete childhood immunization

  • Ensuring monthly weight increases for infants

  • Regular weighing for under-fives

  • Taking Vitamin A twice a year for under-fives


  • Primary school enrolment (7-12 year olds)

  • Regular primary school attendance >85%

  • Junior secondary school enrolment (13-15 year olds)

  • Regular secondary school attendance >85%

Pnpm generasi project implementation
PNPM-Generasi Project Implementation

  • Geographical coverage:

    • 178 subdistricts in 21 districts, five provinces

    • Approx 3.1 million beneficiaries

  • Block grant amounts:

    • 2007 average per village amount USD 8,400

    • 2008 average per village amount USD 11,600

    • 2009 average expected per village amount USD 14,400

  • Timeframe:

    • First block grant disbursed to villages in Oct-Dec 2007

    • Second year disbursement to villages in Oct-Dec 2008

    • Third year disbursement to villages in Oct-Dec 2009

Village fund allocation in 2007
Village Fund Allocation in 2007

  • 56% of block grants on education:

    • School materials, equipment and uniforms (59%)

    • Financial assistance and school fees (31%)

    • Infrastructure (satellite classrooms and access roads) (5%)

    • Financial incentives for honorarium teachers (4%)

    • Training and behavior change communication (1%)

  • 44% of block grants on health activities:

    • Supplementary feeding activities (40%)

    • Financial assistance for pregnant mothers to use services (30%)

    • Infrastructure (13%)

    • Facilities and equipment (11%)

    • Training and behavior change communication (3%)

    • Incentives for health workers (3%)

Impact evaluation design
Impact Evaluation Design

  • Uses a randomized evaluation

  • Subdistricts allocated by lottery into three groups:

    • with performance incentives,

    • without performance incentives, and

    • controls

  • Subdistrict level randomization addresses spillovers and crowding out

Impact evaluation design1
Impact Evaluation Design

  • Three rounds of surveys:

    - Baseline/Wave I (2007): PNPM-Generasi & PKH (HH CCT)

    • Wave II (2008): PNPM-Generasi only

    • Wave III (scheduled to begin Nov 2009): PNPM-Generasi & PKH

  • Survey design:

    • 12,000 households per wave spread over 300 subdistricts including

      • Anthropometric measurements of children <3

      • Math and Indonesian tests administered to school-aged children (Baseline and Wave III)

    • School and health provider interviews to track supply-side effects

    • Qualitative studies to understand bottlenecks in use and provision of services (Baseline and Wave III)

Impact evaluation design2
Impact Evaluation Design

Sample size per survey for PNPM-Generasi:

33,000 total respondents

- 12,000 households

  • 10,800 married women in reproductive age

    • 4,850 pregnancies (2 years prior to the survey)

  • 9,500 school-aged children (7-15 years old)

  • 4,750 children under-three

  • 2,313 villages

  • 300 subdistrict health centers (puskesmas)

  • 1,157 midwives

  • 2,391 village health posts (posyandu) (Waves II & III only)

  • 847 junior secondary schools

  • 1,065 primary schools (Waves II & III only)

  • Mid term impact analysis
    Mid-Term Impact Analysis

    • Regressions run for:

      • PNPM-Generasi vs No PNPM-Generasi

      • Incentivized (version A) vs Non-Incentivized (version B)

    • Regression specifications:

      • Uses baseline data as control variables

        • Subdistrict average

        • Individual baseline values for panel respondents (0 for non-panel)

      • District fixed effects

      • Province * previous KDP experience fixed effects

      • HH sampling category dummies

    Overview of preliminary findings
    Overview of Preliminary Findings

    Substantial improvements in health indicators

    No impact on education indicators

    Performance-based incentives lead to consistently better outcomes

    Substantial regional heterogeneity with strongest improvements in Sulawesi

    Provider effort, especially for midwives in incentivized locations, increased substantially.

    Greater community engagement particularly through service provision at the village health posts.

    Preliminary findings health
    Preliminary Findings: Health

    • Strongest improvements on health services coverage:

      • Participation in growth monitoring

      • Deliveries assisted by doctors or midwives, particularly in Java and Sulawesi

      • Large increase in village health post participation

    • Long-term health outcomes:

      • Large reductions in neonatal and infant mortality (although some small differences noted at baseline)

      • Some reductions in malnutrition (<2SD weight-for-age) among children under-three in NTT and Sulawesi

    November 5 2009

    Notes on the figures

    Each bar represents the percentage

    change in the indicator in Generasi

    treatment areas compared to control


    Key messages of the figure

    Bars in


    are not



    Control avg.

    Control group


    Bars in solid

    colors are



    Zero percent is the control group average. Bars to the

    right indicate increase while bars to the left indicate reduction.

    November 5 2009

    • Large increase in Village Health Post participation :

      • Numbers of children weighed; receiving supplementary feeding; immunized; receiving Vit A

      • Numbers of pregnant mothers receiving antenatal care; iron pills

      • No increase in non-targeted village health post services

    Control avg.

    November 5 2009

    Control avg.

    Preliminary findings education
    Preliminary Findings: Education (although some small differences noted at baseline)

    • No overall project impact on education

      • Negative impact on enrollment and attendance of 13-15 year olds, primarily among those 13-15 year olds who would have been in primary

      • No impact on primary or jr. secondary net enrollment

    • Overall jr. secondary school enrollments increased in both treatment and control over this period

    November 5 2009

    Control avg. (although some small differences noted at baseline)

    Some hypotheses on education findings
    Some Hypotheses on (although some small differences noted at baseline)Education Findings

    • Primary school enrollment already high at 95%.

    • Jr. secondary gross enrollment increased overall in treatment and control areas.

    • Great deal of fluctuation over last couple of years.

    • Generasi targets only 13-15 years old for jr. secondary, so communities may have interpreted this age conditionality strictly.

    • Program missed the school enrollment period of June-July.

    • Communities seemed to be favoring more assistance towards children already in school vs. focusing on out-of-school children. “Help the greatest number vs. the few” mentality.

    Preliminary findings community incentives
    Preliminary Findings: (although some small differences noted at baseline)Community Incentives

    • Version A, incentivized version outperformed non-incentivized version in improving health service coverage

      • Prenatal care

      • Postnatal care

      • Growth monitoring

    • Version A had larger impact on long-term health outcomes:

      • Acute morbidity (ARI or diarrhea)

      • Malnutrition

    • Version A was more effective in increasing midwives’ work efforts in:

      • Outreach services

      • Time spent seeing patients in their public capacity (reduced time for private practice)

    November 5 2009

    Control avg.

    November 5 2009

    Control avg.

    Preliminary findings heterogeneity
    Preliminary Findings: Heterogeneity improving health outcomes

    • Regional heterogeneity

      • Sulawesi - strongest and consistent effects

      • Java - some positive impacts

      • NTT – smallest effects

    November 5 2009





    November 5 2009

    Negative impact on education largely seen in Sulawesi outcomes overall and in NTT





    Conclusions outcomes overall and in NTT

    PNPM-Generasi has:

    Improved health service coverage mainly through increased village health post (posyandu) participation

    Reduced infant mortality, acute morbidity and malnutrition

    Increased number of hours midwives spend on outreach and services in their public capacity

    PNPM-Generasi has not yet improved formal education indicators

    Community incentives ensures better outcomes with the same project funds and design

    Certain heterogeneity in outcomes were observed:

    Regional: Sulawesi strongest, positive in Java, small in NTT

    Increased community participation and engagement, especially through service provision at village health posts.

    Future analysis
    Future Analysis outcomes overall and in NTT

    Cost-benefit analysis

    Where/for whom PNPM-Generasi works best


    Details of community incentives

    Changes in prices


    What communities spent their funds on

    Next steps
    Next Steps outcomes overall and in NTT

    PNPM-Generasi will likely expand in 2010 to an additional 1-2 provinces.

    Adjustment of some of the education indicators

    Conduct wave III survey in 660 kecamatan jointly with HH CCT (Nov 2009– Jan 2010)

    Finalize operations paper on lessons learned thus far