Dr Geoff Setswe, Dr Mark Ottenweller & Roger Dickinson SAMEA Evaluation in Action conference

1. Methods for the evaluation of the Abstinence and Faithfulness among Youth (ABY) interventions in South Africa Dr Geoff Setswe, Dr Mark Ottenweller & Roger Dickinson SAMEA Evaluation in Action conference 30 March 2007

2. Structure of the presentation • Introduction to the ABY project • ABY interventions at Hope Worldwide • Evaluation methods • Survey methods • Qualitative methods – Focus Group Discussions • Some baseline demographic results • Recommendations

3. Introduction • The goal of the ABY project is to enhance local responses to prevent HIV infection through encouraging abstinence, faithfulness and avoidance of unhealthy sexual behaviors among youth. • ABY interventions are implemented by Hope Worldwide in selected sites in 5 cities in South Africa. • Project is funded by PEPFAR/USAID over five years (2005-2009) in 5 African countries (SA, Botswana, Kenya, Nigeria and Ivory Coast). • We present methods used for the evaluation of ABY interventions in South Africa.

4. ABY interventions at Hope Worldwide • Abstinence interventions encourage unmarried individuals to abstain or delay sex as the best way to prevent exposure to HIV and STIs. Targeted at primary school learners aged 10-14 years. • Be faithful interventions encourage individuals to practice fidelity in marriage and in other sexual relationships as a critical way to reduce the risk of exposure to HIV. Targeted at youth aged 15-24 years. • Both interventions could be part of ABC or DRC programs

5. ABY interventions at Hope Worldwide • Interventions are designed and implemented by local people and adapted to fit local conditions; • Implemented in age-appropriate and culturally relevant groups, in targeted areas. Include life skills programs with components of Men As Partners (MAP) and VCT for youth; • For 10 -14 and 15 – 19 year-olds, interventions are school and/or church-based. For youth aged 20-24, interventions are church or community-based. • Parents and adult church members have interventions specifically targeted to them.

6. Methods ABY interventions are evaluated using mixed methodologies - quasi-experimental and qualitative designs with 3 strategies: • Pre-intervention or baseline surveys for learners and youth; just before the interventions were implemented in 2005/6. • Post-intervention surveys for learners and youth (who participated in the interventions); when the interventions have been completed in 2008/9. • Focus Group Discussions (FGDs) for parents and church members who participated in the interventions.FGDs will be conducted once – during the middle of the project, in 2007.

7. Pre- and Post intervention surveys One-group, Pretest/Posttest design • A single group i.e. learners and youth, is studied before and after the ABY intervention. No control group. • Layout O1 X O2 O1 = Baseline or pre-intervention survey in 2005/6 X = ABY intervention in 2006/7 to selected participants O2 = Post-test survey in 2008/2009

8. Sampling and sample size estimation

9. Sample size estimation • 11 sites were randomly selected from 20 intervention sites. Then 27 evaluation points were identified. • The evaluation points comprised: • primary schools (for learners aged 10-14 years), • high schools (for learners aged 15-19 years) • sites for out-of-school youth (aged 20-24 years). • Data were collected from 60 randomly selected learners or youth at each of the 27 evaluation points = 1,620. • Grades 11 and 12 were not evaluated as they cannot be followed up in post-intervention surveys • Data were collected by trained fieldworkers.

10. Survey questionnaires • Two types of questionnaires were used: • for learners aged 10-14 years and • for learners/youth aged 15-24 years. • Key questions were on: • Awareness/knowledge of HIV/AIDS and attitudes towards PLWHA • Views on dating, relationships, sexuality and pregnancy • Views on abstinence and faithfulness • The role of the media, role models and workshops in encouraging abstinence and faithfulness • The role of peer-group pressure and violence on abstinence and faithfulness.

11. Some baseline demographic resultsGender distribution of respondents 43.5% (570) 56.5% (740) n = 1310. Response rate = 81%

12. Some baseline demographic resultsDistribution of respondents according to cities 33.1% 25.6% 20% 13.1% 8.2%

13. Some baseline demographic results 12.5% (164) 2.4% (31) 38.5% (505) 46.6% (610) n = 1310

14. Qualitative methods - FGDs • 15 FGDs planned – 3 in each of the 5 cities: • 5 FGDs - adults who have participated in the ABY interventions provided through FBOs in the 5 cities • 5 FGDs - adults or parents of children at ABY intervention schools in the five cities. • 5 FGDs - persons aged 25 years and above from the general population in each of the 5 cities. • FGDs will be used to gather in-depth information on parents and church members' opinions and views on abstinence and faithfulness in relationships. • Each FGD will have 8 - 12 participants who will be asked to provide verbal consent to participate and permission to record the interviews. • The training for facilitators of FGDs has been scheduled and the FGD guide is ready.

15. Schedule of ABY FGDs

16. Recommendations • To ensure credibility: • evaluation should be integrated into the initial planning of the project; • all evaluation preparations and training should be done before the interventions start. • To ensure effective evaluation of ABY interventions, it is essential to: • use various designs and strategies in different target groups; • involve team members in the design of the evaluation; • train team members on the methods of evaluation, data capturing and the importance of data quality and management.

17. Acknowledgements • ABY intervention and evaluation teams at Hope Worldwide South Africa • Learners, youth, parents and adults who participated in the evaluations and FGDs • Funding received from USAID/PEPFAR