Targeted Evaluation of Five Programs Supporting Orphans and Vulnerable Children: Background and Methods

1. Targeted Evaluation of Five Programs Supporting Orphans and Vulnerable Children:Background and Methods Florence Nyangara, PhD MEASURE Evaluation/Futures Group Dissemination Meeting, September 3rd, 2009 Washington, DC

2. The number of OVC and their corresponding programs increasing Background • SSA - 12 million orphans (0-17 yrs), 2003 • 2010 - over 18 million orphans (O) • Other millions are made vulnerable – HIV/AIDS, dire poverty, war, etc (V) • Response – increased attention to the plight of OVC (funds, programs) Sub-Saharan Africa’s population of children orphaned by AIDS increasing Children on the Brink, 2004

3. Response to the OVC Crisis OVC programs – emergency response to areas most HIV-affected Strategies used were based on existing cultural support systems, conventional wisdom, and lessons learned from other program areas; Support community-based responses (capacity & resources) Household/family support (capacity & resources) Direct support to families & OVC (access to essential services) Gap – lack of evidence to guide OVC programs Call for evidence based programming 2006 - USAID funded MEASURE evaluation to conduct targeted evaluations to fill this evidence gap Background

4. Evaluation Goals • Find out “what works” in terms of • intervention models and program components • cost effectiveness, and • outcomes (benefits) for OVC and their caregivers in resource poor settings • Provide evidence to guide program decisions such as; • Scaling-up of best practices (models, strategies), and • Modify & improve interventions - to make them effective

5. Research Preparation Activities Funds were made available – USAID/PEPFAR/PHE (4) and USAID/Tanzania mission funded (1) program evaluation. Research team formed - MEASURE Evaluation Extensive literature reviews (early 2006) Consultation meetings with stakeholders Identified OVC programs to be evaluated Research protocol developed Ethical approvals obtained – US, Kenya, and Tanzania Identified local research partners (PSRI – KE; AXIOS - TZ)

6. Programs Evaluated Evaluated Five programs : 2 in Kenya & 3 in Tanzania They have different intervention models with varied combinations of child, family/household, and/or community centered approaches (multi-faceted). *** Although, the approaches vary, the goal for all of these programs is to improve the well-being of OVC and their families.

7. Programs Evaluated Kenya (2) Kilifi OVC Project,Catholic Relief Services (CRS) Operating in Kilifi District for two years Community Based HIV/AIDS Prevention, Care and Support Program (COPHIA), Pathfinder & Integrated AIDS Program (IAP) Operating in Thika District for 4 years Tanzania (3) CARE Tumaini Project, Allamano, CARE, FHI (Allamano) Operating in Iringa Region for five years Mama Mkubwa & Kids Club, Salvation Army (TSA) Operating in Mbeya Region for 2-years Jali Watoto, Pact/SAWAKA (Jali Watoto) – Field funded Operating in Karagwe, Kagera Region for four years

8. Overview of Programs & Strategies Evaluated

9. Key Research Questions Impact of indirect support: How do efforts targeted at the structural systems surrounding children– household and community– affect: Children well-being Caregivers well-being Community attitudes and support of OVC & families? Impact of direct support on child outcomes What is the impact of educational, health, legal support, and other direct services on child & families?

10. Methods Case studies (2006 and 2007) Site visits, interviews, program document review Provide understanding of program strategies, components, goals, and expected outcomes Document lessons learned from implementation Case Studies available http://www.cpc.unc.edu/measure/ovc Program expenditures (2006) Expense data collected and social costs estimated Quantified the costs corresponding to specific intervention (e.g. food supplementation, psychosocial service, educational support)

11. Methods Outcome evaluation (2007 and 2008) Post-test study design with intervention & comparison groups Exposed Vs. Non-exposed Surveyed children age 8-14 “or 7-15” & their caregivers Up to 2 children per household Four questionnaires were applied in each household: Q1: Household Questionnaires Q2: Parent/Guardian/Caregiver Questionnaire Q3: Parent/Guardian/Caregiver Regarding Child Questionnaire Q4: Child (age 8-14 “or 7-15”) Questionnaire

12. Ideal Impact Assessment R

13. Post-test Study Design Used The Groups are Not Randomly Assigned Exposed X O1 O2 Non-exposed ** Jali Watoto – Study compared intact groups of intervention versus comparison

14. Research Designs and Sampling

15. Principles Guiding Questionnaires Development Capture multiple measures for each domain Use existing standardized scales where possible (PSS, SES) Intervention Exposure questions to be specific to each program To facilitate comparison across countries and program models, same survey instruments were used except intervention modules Multi-faceted programs necessitated sufficient questions across multiple domains Multiple perspectives on child well-being (child and caregiver) Measures of caregiver, household & community well-being

16. Outcomes Examined Psychosocial well-being – multiple measures – standard scales used where appropriate (child and caregivers) Education – enrollment & attendance (child) Health – self-reported health status and access to health services (child and caregiver) HIV-prevention – HIV-knowledge (child) & HIV-testing (caregiver) Legal protection – birth registration, alternate caregiver Community support – stigma and in-kind support (child & caregiver)

17. Study Strengths and Limitations Strengths: Yielded immediate data on program effects Results can be used to improve current programs Ethical - not withholding services for experiment sake Limitations: Post-test design – no baseline data - impossible to make conclusions concerning change in outcomes resulting from program exposure Selection bias - self-selection to participate and those who did not -makes it difficult to conclude with certainty that the interventions are responsible for the observed differences

18. Analyses Plan Who are the OVC/MVC program beneficiaries? Effects of community level interventions i.e. Community care and support meetings/sensitization Effects of household or caregiver level interventions i.e. Community volunteer or Health Worker home visits Caregivers participation in OVC care seminars Effects of child level interventions i.e. Kids clubs Basic needs support (e.g., education, health, legal)

19. Statistical Analyses Descriptive analysis (Univariate) Bivariate analysis (ANOVA and Chi-square) Multivariate (logistics, and linear regression) Control variable: non-program factors e.g. socio-demographic Child Level – Age, sex, orphan status, relationship to caregiver, and number of different homes the child had lived in the past year. Caregiver level – Age, sex, marital status, education, illness, SES, and # children

20. Description of OVC Program Beneficiaries • Who is enrolled in OVC programs (MVC Profile):- • Although, these programs targeted geographic areas most affected by HIV/AIDS, MVC were identified and assisted regardless of specific causes of vulnerability • Majority of children enrolled in OVC programs are vulnerable in several fronts & not just orphanhood

21. OVC Profile • Orphans (over 66% across programs) • Living in food insecure HH (over 80% across programs) • Poorest households (< 2assets) – over 40% • Living with chronically ill primary caregiver (over 20%) • Living with caregiver aged 50+ (about 23%) • Lived in two or more households in past year (14%)

22. Thank YOU! Key Findings are presented next….

23. MEASURE Evaluation is funded by the U.S. Agency for International Development through Cooperative Agreement GHA-A-00-08-00003-00 and is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership with Futures Group International, ICF Macro, John Snow, Inc., Management Sciences for Health, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government. Visit us online at http://www.cpc.unc.edu/measure.