Tracer Study Approach in Evaluation of BTVET

1. Tracer Study Approach in Evaluation of BTVET Presented To Uganda Evaluation Association & ECD Project Kampala Evaluation Talk Thursday, 2nd October 2014

2. Presenter • Muhwana Wilberforce, MWSES CONSULT • Ntinda, Steka House, 1st Floor • Plot 3122 Ntinda – Kiwatule road • wamuhwana@gmail.com; mwssconsult@rocketmail.com; 0772429679; 0701242652 • Social Sector Development Consultancy • Education, Health, Community Livelihoods • Policy/project formulation, Baseline surveys, Mid and End Line Evaluation, Impact Assessments, Tracking studies • Government ministries, Institutions, Donor agencies, NGOs

3. Study team • MUHWANA WILBERFORCE (LOCAL CONSULTANT) – Kampala, Uganda • CLAUDIA MUMMENTHEY (INTERNATIONAL CONSULTANT) – Cape Town, South Africa • Sourced by BTC separately • Teamed up to harmonise the methodology for the evaluation • Delivered a single inception report • Supported by 7 RAs and 3 data entrants

4. Background to the study • The Business, Technical, Vocational Education and Training (BTVET) Subsector is under the Ministry of Education and Sports (MoES), BTVET Department and was established under TVET Act, 2008. • BTVET offers training under an overlapping three- tier system: • (i) P7 leavers to enroll in technical schools & community polytechnics, • (ii) ‘O’ leavers enroll in technical institutes & vocational training institutes, • (iii) ‘A’ leavers enroll in technical colleges; and • (iv) recently non-formal short courses across the various levels.

5. Background to the study – cont’d • The sub-sector fell short of the overall goals and objectives regarding the production of a ‘critical mass’ of graduates that could readily be absorbed into the existing labour market. • Internally the sector and subsector was grappling with issues of low quality of education and irrelevant curricula (UNESCO, 2005). Externally the BTVET system was not corresponding with the needs of the private sector (ESAPR, 2010). • The private sector was not involved in curriculum development thus there was no realignment of knowledge and skills acquisition to the real needs of the labour market. • There was widespread unfavourable perception and recognition of the BTVET programs among the public

6. Purpose of the study • MoES embarked on developing a 10-year BTVET Strategic Plan in 2011 • Six studies had been conducted ex- ante • The Tracer Study conducted belatedly • To generate data and information about the effectiveness of different BTVET programmes in Uganda as an input into the strategic planning process of identifying potential improvement areas for the overall system.

7. Objectives of the study • Overall objective: To provide an in-depth analysis of the employment destinations of graduates BTVET programmes. • To undertake a comparative analysis of factors affecting external effectiveness and relevance of training provided by the public and private sectors, • To identify employment destinations/career paths of completers of BTVET programmes and factors determining post – training careers

8. Objectives of the study – cont’d • Accurately assess the potential significant differences in the external effectiveness of different BTVET programmes

9. Tracer study • Tracing of a “reachable” (i.e. within the given restrictions of data availability and time constraints of the assignment) and representative cohort of BTVET graduates from the existing BTVET system using a scientific approach. • Effectiveness – ability to equip learners with the skills and knowledge required in the labour market thus leading to employment readily.

10. Design of the Study • Consensus on study expectations • Methodology • Industry model • Scope • Sample Structure • Data collection tools • Data collection • Data recording and analysis • Data Quality

11. Consensus on Study Expectations • Desk –based review of reports by strategy development group • Debriefing by Strategy development group • Debriefing/interviews with key stakeholders – MoES, WB, BTC, GTZ/PIVOT • On-site visits – BTVET institutions – public, private, rural & urban • Helped to understand expectations, quality of BTVET programmes, available data; planning for the study

12. Methodology • Industry model: • Input process output, with focus on: • Training competence and infrastructure (input) • training processes (processes) • Training outcome (output/outcome) • Overall satisfaction and image/reputation of BTVET

13. Methodology • Scope: • GRADUATES (i.e. learners who had passed final examination; learners who had completed the training but not yet received certificate) • Current EMPLOYERS of graduates. • Contextual elements: • Established the readiness of BTVET graduates being absorbed into in the labour market (i.e. formally employed or self-employed) and/or accessing further education and training opportunities within the system. • Identified and examined the factors and circumstances that enhance and/or inhibit the placement of graduates into employment and/or further education and training opportunities. • Included the evaluation of BTVET delivery processes, interaction between BTVET institutions and employers and expectations of graduates and employers, among others.

14. Sample structure • BTVET Graduate Cohort • Multi dimensional & stratified: (i) By institutions – public + private = 650 • arbitrary 10% representative sample – 65; • Public – 33 + private – 32 = 65 (ii) By regional distribution - UGAPRIVI Classification 7 regions (i.e. Central (incl. Kampala), East, North, South, South West, West Nile and Western

15. Sample structure

16. Sample structure- cont’d • BTVET Graduate Cohort • (iii) By year of completion – distribution across different years of completion (2007-2010) • (iv) By learning pathways –formal, informal, non-formal • (v) By exit examinations- UNEB, DIT • (vi) By different occupational training areas • (vii) By gender distribution – male- 80%, female – 20% • Target sample – telephone contacts for at least 60 graduates in each institution were compiled; 3629 contacts compiled; client’s target was 1500; 1415 graduates were interviewed by phone

17. Sample structure- cont’d • Employers • UBOS data on sector growth – agriculture, industry, services (potential for employment) • UBOS data on distribution of workforce by industry • Kampala & suburbs were covered – good concentration of businesses/companies • Formal & informal • Size of the enterprises • Target sample – contact details for 60 employers; 100 contacts compiled (more than 60 within Kampala); 63 interviewed, 12 not responded, 25 not contacted

18. Data collection tools • Structured questionnaires for graduates and employers • Inclusive of industry model focal areas • 5- point Likert-type scales • Qualitative responses

19. Data collection • 7 RAs deployed in 7 regions to collect bio data (especially phone contacts) on all target graduates in the selected institutions; used a fixed template • 3 RAs deployed to collect profile data on employer organisations (especially phone & e-mail contacts) • 7 RAs administered the questionnaires to the graduates using telephonic interviews (English and in local languages) • 7 weeks, 50 interviews per week/ 10 per day • 2 RAs administered questionnaires to employers physically, collected completed questionnaires, sent via email – based on respondents’ preference • 3 weeks, 20 interviews per week/4 per day

20. Data recording and analysis • M&E web-based survey platform- Metior Solutions. • 3 data recorders trained for ½ day • Each data recorder was given account on the platform • Entered each questionnaire and uploaded on the platform • Quantitative and qualitative data was analysed using platform’s own instant analysis tools • Descriptive statistics on quantitative data and summary reports on qualitative data provided instantly

21. Data quality • Training of RAs for 3 days including simulation & feedback • Initial supervised/observed telephonic interviews – 5 interviews after training • Confirmation of interviews – 5 in every 50 • Paper questionnaires thoroughly checked • Instant platform-based feedback on data entry – incomplete questionnaires, inconsistencies, errors in entries, etc

22. Key findings • Overall satisfaction and BTVET image • 59% of graduates were satisfied • 48% of employers were satisfied • Training quality and delivery (efficiency) • 55% of graduates provided a good rating • 35% of employers rated private institutions as excellent • 38% of employers rated public institutions as good • Overall BTVET outcome (effectiveness) • 64% of graduates felt that skills acquired made them ready for the labour market • 63% of employers felt that graduates were ‘prepared’ for applying the skills and knowledge but not up to the required levels of employment

23. Recommendations • Establish effective BTVET Management Information System • Upgrade, modernise and sufficiently resource existing training infrastructure • Intensify and strengthen cooperation/partnerships between training institutions and industry • Enhance female participation • Focused BTVET programmes on local skills demands

24. Challenges of the evaluation • Lack of baseline data on both graduates and employers • A base population frame of contacts details had to be established before sampling • EMIS BTVET data on graduates not disaggregated by training programmes • Called for prior field visit to public and private institutions • Lack of up-to-date contact data on public and private institutions across 7 regions • individual research approaches on institutions had to be conducted • Obtaining employer interviews in the formal sector was cumbersome as it involved formal approval • Many were left out

25. Challenges of the evaluation • Some employers required repetitive follow –ups and/or solicitation for nominal incentives • Discarded from the list after failure through persuasion • Less eagerness for participation among graduates in the central region • opted for extra interviews in South and South West to compensate

26. Rationale for evaluation approach • Tracer study periodically reaffirms quality, relevance and effectiveness of BTVET sub sector/related services • Tracer study adequately informs direction for prioritisation • Telephonic interviews minimise time consumed in ‘snowballing’ • Metior platform facilitates quick delivery of analysed data particularly in restrictive ‘time bound’ situations