Targeted Evaluation of Five Programs Supporting Orphans and Vulnerable Children: Background and Methods. Florence Nyangara, PhD MEASURE Evaluation/Futures Group Dissemination Meeting, September 3 rd , 2009 Washington, DC. The number of OVC and their corresponding programs increasing.
Florence Nyangara, PhD
MEASURE Evaluation/Futures Group
Dissemination Meeting, September 3rd, 2009
Sub-Saharan Africa’s population of children orphaned by AIDS increasing
Children on the Brink, 2004
OVC programs – emergency response to areas most HIV-affected
Strategies used were based on existing cultural support systems, conventional wisdom, and lessons learned from other program areas;
Support community-based responses (capacity & resources)
Household/family support (capacity & resources)
Direct support to families & OVC (access to essential services)
Gap – lack of evidence to guide OVC programs
Call for evidence based programming
2006 - USAID funded MEASURE evaluation to conduct targeted evaluations to fill this evidence gap
Funds were made available – USAID/PEPFAR/PHE (4) and USAID/Tanzania mission funded (1) program evaluation.
Research team formed - MEASURE Evaluation
Extensive literature reviews (early 2006)
Consultation meetings with stakeholders
Identified OVC programs to be evaluated
Research protocol developed
Ethical approvals obtained – US, Kenya, and Tanzania
Identified local research partners (PSRI – KE; AXIOS - TZ)
Evaluated Five programs : 2 in Kenya & 3 in Tanzania
They have different intervention models with varied combinations of child, family/household, and/or community centered approaches (multi-faceted).
*** Although, the approaches vary, the goal for all of these programs is to improve the well-being of OVC and their families.
Kilifi OVC Project,Catholic Relief Services (CRS)
Operating in Kilifi District for two years
Community Based HIV/AIDS Prevention, Care and Support Program (COPHIA), Pathfinder & Integrated AIDS Program (IAP)
Operating in Thika District for 4 years
CARE Tumaini Project, Allamano, CARE, FHI (Allamano)
Operating in Iringa Region for five years
Mama Mkubwa & Kids Club, Salvation Army (TSA)
Operating in Mbeya Region for 2-years
Jali Watoto, Pact/SAWAKA (Jali Watoto) – Field funded
Operating in Karagwe, Kagera Region for four years
Overview of Programs & Strategies Evaluated increasing
Impact of indirect support:
How do efforts targeted at the structural systems surrounding children– household and community– affect:
Community attitudes and support of OVC & families?
Impact of direct support on child outcomes
What is the impact of educational, health, legal support, and other direct services on child & families?
Case studies (2006 and 2007)
Site visits, interviews, program document review
Provide understanding of program strategies, components, goals, and expected outcomes
Document lessons learned from implementation
Case Studies available http://www.cpc.unc.edu/measure/ovc
Program expenditures (2006)
Expense data collected and social costs estimated
Quantified the costs corresponding to specific intervention (e.g. food supplementation, psychosocial service, educational support)
Outcome evaluation (2007 and 2008)
Post-test study design with intervention & comparison groups
Exposed Vs. Non-exposed
Surveyed children age 8-14 “or 7-15” & their caregivers
Up to 2 children per household
Four questionnaires were applied in each household:
Q1: Household Questionnaires
Q2: Parent/Guardian/Caregiver Questionnaire
Q3: Parent/Guardian/Caregiver Regarding Child Questionnaire
Q4: Child (age 8-14 “or 7-15”) Questionnaire
The Groups are Not Randomly Assigned
** Jali Watoto – Study compared intact groups of
intervention versus comparison
Capture multiple measures for each domain
Use existing standardized scales where possible (PSS, SES)
Intervention Exposure questions to be specific to each program
To facilitate comparison across countries and program models, same survey instruments were used except intervention modules
Multi-faceted programs necessitated sufficient questions across multiple domains
Multiple perspectives on child well-being (child and caregiver)
Measures of caregiver, household & community well-being
Psychosocial well-being – multiple measures – standard scales used where appropriate (child and caregivers)
Education – enrollment & attendance (child)
Health – self-reported health status and access to health services (child and caregiver)
HIV-prevention – HIV-knowledge (child) & HIV-testing (caregiver)
Legal protection – birth registration, alternate caregiver
Community support – stigma and in-kind support (child & caregiver)
Yielded immediate data on program effects
Results can be used to improve current programs
Ethical - not withholding services for experiment sake
Post-test design – no baseline data - impossible to make conclusions concerning change in outcomes resulting from program exposure
Selection bias - self-selection to participate and those who did not -makes it difficult to conclude with certainty that the interventions are responsible for the observed differences
Who are the OVC/MVC program beneficiaries?
Effects of community level interventions i.e.
Community care and support meetings/sensitization
Effects of household or caregiver level interventions i.e.
Community volunteer or Health Worker home visits
Caregivers participation in OVC care seminars
Effects of child level interventions i.e.
Basic needs support (e.g., education, health, legal)
Descriptive analysis (Univariate)
Bivariate analysis (ANOVA and Chi-square)
Multivariate (logistics, and linear regression)
Control variable: non-program factors e.g. socio-demographic
Child Level – Age, sex, orphan status, relationship to caregiver, and number of different homes the child had lived in the past year.
Caregiver level – Age, sex, marital status, education, illness, SES, and # children
Key Findings are presented next….
International Development through Cooperative Agreement
GHA-A-00-08-00003-00 and is implemented by the Carolina
Population Center at the University of North Carolina at
Chapel Hill, in partnership with Futures Group International,
ICF Macro, John Snow, Inc., Management Sciences for
Health, and Tulane University. The views expressed in this
presentation do not necessarily reflect the views of USAID or the United States government.
Visit us online at http://www.cpc.unc.edu/measure.