1 / 13

Impact evaluation

Impact evaluation. REAL (working with Singizi) Presentation to NSA 3 rd /4 th August. Defining impact evaluation. T he positive and negative, intended and unintended, direct and indirect, primary and secondary effects produced by an intervention.

cnovak
Download Presentation

Impact evaluation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Impact evaluation REAL (working with Singizi) Presentation to NSA 3rd/4th August

  2. Defining impact evaluation The positive and negative, intended and unintended, direct and indirect, primary and secondary effects produced by an intervention. OECD Development Assistance Committee definition

  3. Extending the definition When understanding impact there is a need to go beyond “what works and for which groups” and answer other equally important and related questions, such as “why do certain programmeswork for some groups and not for others”. Martin and Grubb, What works and for whom:A review of OECD countries’ experiences with active labour market policies, Swedish Economic Policy Review, 8 (2001) 9-56

  4. Possible Impact Evaluations Designs

  5. Approach for this study • Undertaking a meta-review of impact studies undertaken during the period of NSDSIII • Completing a tracer study (where sectors have not undertaken a tracer study) • Will not have a control group but instead is a longitudinal study (though possibility of comparing with administrative data available being explored to provide a point of comparison for key issues such as length of time to transition to the labour market where programmes are offered to unemployed)

  6. Rationale for this design selection Its practical and … • country-led planning and evaluation (often covers whole country) • Has horizontal and vertical partnerships • Aim of long-term changes, but a need for short-term results • Multiple stakeholders with different perspectives • Non linear • Many components or services • Different theories of change evident across projects • Multiple donors and agencies • Context is critical and challenge has multiple and contested causes Large, complex Complex interventions • May include a number of projects and wider scope • Often involves several blueprint approaches • Defined objectives but often broader and less precise and harder to measure • Often not time-bound • Context important • multiple donors and national agencies Complicated programs Simple projects Small, simple • “blue print” producing standardized product • relatively linear • Limited number of services • Time-bound • defined and often small target population • Defined objectives Adapted from Michael Bambeger, June 28, 2011 and Singizi Theory of Change and complex programmes

  7. Implications for methodology • Need to consider quantitative and qualitative data - Participatory, developmental and utilisation approach – these consider change from the perspective of the people/communities/institutions undergoing the intervention • Which can be collected in many various ways, like surveys, focus group discussions, interviews, etc

  8. Tracer focus • Programme details – field, level, type • Status in programme (if still in – STOP, if dropped out – one set of questions, if completed – another set of questions) • Demographics – ace, gender, age, disability etc • Entry into programme – motivation, selection, employment status when entering the programme (will route from here)

  9. Tracer focus (continued) • If unemployed/pre-employed before the programme – • Employment status now and/or education status • Extent to which programme enabled you to access this employment (support measures in place and recognition of programme) • Whether the programme enabled the learner to succeed in this work (quality and relevance of programmeand opportunities that the learner have had access to in the workplace) • Possibilities of navigating further pathways of learning or employment

  10. Tracer focus (continued) • If employed before the programme • Status now - Whether the individual is still in the same workplace • Promotion? Increase? New responsibilities? Access to training? • Whether the programme enabled this (recognition or new skills – linked to quality or relevance of the programme, etc) • Any changes in the workplace because of their increased skill

  11. Tracer focus (continued) • If self-employed before the programme • Status now - whether still self-employed/employed/studying/unemployed • Extent to which the individual has increased skills to run and grow their business (if still self-employed) • If programme linked to other support as relevant (finance, market, etc)

  12. Tracer focus (continued) • If unemployed now – • Whether still looking and if so, where - • Whether the programme helped them to look • What they think the programme could have done more of to assist them?

  13. Process • We are collecting data: • Which programmesand providers and which learners (plus contact details) • This has been a challenge but working hard to get these • Plan to send tracer to all sectors that have not completed a tracer (emails, calls, SMS) • Then do sample of qualitative interviews

More Related