IMPACT EVALUATION OF PERFORMANCE BASED CONTRACTING FOR

1. IMPACT EVALUATION OF PERFORMANCE BASED CONTRACTING FOR A collaboration between the Ministry of Health, CNLS, SPH, INSP-Mexico and World Bank GENERAL HEALTH AND HIV/AIDS SERVICES IN RWANDA

2. Presentation plan Country Profile Impact evaluation design Results from the general health baseline and Follow up Results from HIV baseline and Follow up studies Discussion 2 9/1/2014

3. SECTION 2 :Impact Evaluation Design

4. Study Rationale No examples of rigorously evaluated bonus payment schemes to public sector health care providers in developing countries No distinction between the incentive effect and the effect of an increase in resources for the health facilities Link between worker motivation programs and quality of care 8 9/1/2014

5. TRADITIONAL FINANCING VS PBF Traditional Way of Health financing in Rwanda was based on 3 sources of finance: GoR / Founders: Investment cost (Infrastructures & Equipment) GoR / Private / Population: Recurrent cost (salaries, drugs, materials, trainings,..) PBF: funds received for remuneration can be derived for enhancing human capacity building, HF materials, or any new strategy aiming at increasing quality.

6. Hypotheses For both general health services and HIV/AIDS services, we will test whether PBC: Increases the quantity of contracted health services delivered Improves the quality of contracted health services provided Does not decrease the quantity or quality of non-contracted services provided, Decreases average household out-of-pocket expenditures per service delivered Improves the health status of the population 10 9/1/2014

7. Evaluation Design Make use of expansion of PBC schemes over time The rollout takes place at the District level Treatment and control facilities were allocated as follows: Identify districts without PBC in health centers in 2005 Group the districts in “similar sets” based on characteristics: rainfall population density livelihoods Flip a coin to assign districts within each “similar” to treatment and control groups. 11 9/1/2014

9. Roll-out plan Phase 0 districts (white) are those districts in which PBF was piloted Cyangugu = Nyamasheke + Rusizi districts Butare = Huye + Gisagara districts BTC = Rulindo + Muhanga + Ruhango + Bugesera + Kigali ville Phase 1 districts(yellow) are districts in which PBF is being implemented in 2006, following the ‘roll-out plan’ Phase 2 districts (green) are districts in which PBF is not yet phased in; these are the so-called ‘Phase 2’ or ‘control districts’ following the roll-out plan. According to plan, PBF will be introduced in these districts by 2008. 13 9/1/2014

10. Baseline Health Facility Sample

11. Program Implementation Timeline 16 9/1/2014

12. More money vs. More incentives Incentive based payments increase the total amount of money available for health center, which can also affect services Phase II area receive equivalent amounts of transfers average of what Phase I receives Not linked to production of services Money to be allocated by the health center Preliminary finding: most of it goes to salaries

13. The baseline has 4 surveys General Health facility survey (166 centers) General Health household survey (2,016 HH) HIV/AIDS facility survey (64 centers) HIV/AIDS household survey (1994 HH)

14. Baseline Field Sampling: GH HH • Randomly selected FOUR CELLS per HF • Done in SPH using Epi Info and given to team leaders • Randomly selected THREE ZONES per cell • Done by GH HH team leader in field • Obtained household lists for each of the zones and randomly selected ONE HOUSEHOLD per zone • Done by GH HH team leader in field with zone/village leader • Random sample of 12-13 households with children < 6 years old per HF • 2,159 households.

15. Baseline Field Sampling: GH HH

16. Validity of Sample • Require two different validations: • Validate the sampling for the evaluation design • Diff in means tests between Phase I and Phase II to determine if intervention and comparison groups balanced at baseline • Validate the quality of data • Compare descriptive stats to other sources of national data (i.e.: 2005 & 2007 DHS, MOH data)

17. Analysis Plan All analyses will be clustered at the district level Compare the average outcomes of facilities and individuals in the treatment group to those in the control group 24 months after the intervention began. Use of multivariate regression (or non-parametric matching) : control confounding factors Test for differential individual impacts by: Gender, poverty level Parental background (If infant : maternal education, HH wealth) 23 9/1/2014

18. Difference in differences models To test the robustness of the analysis Control sample (both observed and unobserved) heterogeneity A two-way fixed effect linear regression: 24 9/1/2014

19. SECTION 3 :Descriptive Results from General Health Baseline

20. GENERAL HEALTH FACILITIES BASELINE SURVEY

21. General Health Centers Survey:Content All 166 centers in General Health General characteristics Human resources module: Skills, experience and motivations of the staff Services and pricing Equipment and resources Vignettes: Pre-natal care, child care, adult care VCT, PMTCT, AIDS detection services Exit interviews: Pre-natal care, child care, adult care, VCT, PMTCT

22. Classification of facility

23. Baseline Health Facility:Main sources of revenue

24. Baseline Health Facility:Human Resources On average: 1 doctor for every 31,190 individuals, 1 nurse for every 4,835 individuals 30

25. Baseline Health Facility: Services

26. Baseline Health Facility: Lab Tests

27. Baseline Health Facility:Overall Patient Satisfaction

28. Baseline Household Data: Education 37

29. Baseline Household Data: Assets

30. Baseline Household Data: Prenatal Care 40

31. Baseline Household Data: Child Immunization 42

32. Baseline Household Data: Child Recent Illness and Symptoms (<6 years)

33. Baseline Household Data:Child Biomarkers (<6 years)

34. Validity of Sample and Data 46 • Evaluation design • Of 110 key characteristics and output variables of HF, the sample is balanced on 104 of the indicators. • Of 80 key HH output variables, the sample is balanced on 73 of the variables. • Quality of data • HH Results comparable to the 2005 DHS, MOH data

35. Follow-up Field Sampling: GH HH • Result: • In total 2159 were suppose to be surveyed in the catchment area of 167 facilities. • 1888 (87%) were interviewed in 2006 and 2008 • 267 (12%) were replaced • 4 (0.2%) were not found and not replaced • % of replacement by region: • South (24%), North (14%), East (4%) and West (13 %)

36. HIV HEALTH FACILITIES BASELINE SURVEY

37. Baseline Health Facility Sample

38. Baseline Household Sample • 38 ARV facilities for household sampling • 13 MAP; 10 Phase I; 15 Phase II ARV sites for HH sampling • Original sample of 1,487 patients from health facilities and associations • Final sample consists of 1,961 households and 7,494 individuals

39. Household Level: Sample Size of Patients • We find there is an additional sample of individuals in our household sample who self-report HIV+ status • Based on this, final distribution of patients:

40. Baseline Health Facility:Main sources of revenue

41. Baseline Health Facility: Services

42. Baseline Health Facility:Overall Patient Satisfaction

43. HIV Patients vs. Non-Patients:Mortality in the Household

44. Patients vs. Non-Patients:Mortality in the Household in the last 5 years

45. HIV Patients vs. Non-Patients:Health Care Utilization

46. Patients vs. Non-Patients:Sexual Behavior

47. Follow-up Field Sampling: HIV HH • Objective: • Return to the same households to create panel data set (2006-2008) • Print baseline roster (names, codes) and keep consistent across waves • Account for household members who left and new arrivals from 2006-2008

48. Follow-up Field Sampling: HIV HH • Result: • In total 1996 were to be surveyed in the catchment area of 38 ARV facilities. • 1,716 (87.06%) were interviewed in 2006 and 2008 • 280 (12.94%) were not found and not replaced • % of missing by region: • South (12.6%), North (12.7%), East (7%) and West (16.4%)