1 / 11

Using realist evaluations to design and implement complex interventions

This case study explores the use of realist evaluations in the planning and implementation of a market systems development programme in Ethiopia. It highlights key lessons learned and provides an example of linking the realist approach with planning frameworks. The realist evaluation approach focuses on understanding the mechanisms and contexts that underpin interventions.

louisal
Download Presentation

Using realist evaluations to design and implement complex interventions

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Using realist evaluations todesign and implement complex interventions Case study from a market-systems development programme in Ethiopia Matthew McConnachie and Clarissa Samson

  2. Outline • Background to the realist approach • Case study: The Market-Approaches to Climate Resilience project • Key lessons from the case study • Example of linking realist approach with planning frameworks

  3. The realist evaluation approach • Evaluates what works, for whom and in what contexts • Focused on understanding the mechanisms which underpin interventions and how they interact with the context

  4. The realist approach is well-suited for extrapolating findings to new contexts • Able to provide findings that are linked to specific contexts and target groups, and explain the generative mechanisms (why change happened) • But limited work done on how to use realist evaluations to inform the planning and design of interventions

  5. Project Background: Market Approaches to Climate Resilience in Ethiopia • Part of the DFID Building Resilience and Adaptation to Climate Extremes and Disasters programme (2015 – 2019) • Implemented by two NGOs: Farm Africa and Mercy Corps • Aimed to strengthen the private-sector and community-management systems for lowland Ethiopians (financial services, natural resource management, urban business opportunities) • Complex programme, worked across multiple levels • Systems-based approach, transformational change

  6. Evaluation Background: Market Approaches to Climate Resilience in Ethiopia • All projects conducted realist evaluations, a realist synthesis was used at the programme level (see parallel session 5.1 for details) • LTS did the evaluation work on behalf of the project implementers • Evaluation deliverables included periodic reports (M&E plan, baseline, mid-term, endline and extension evaluation reports)

  7. Project planning stage: could have integrated the realist approach in a more structured way • Proposal and initial-design stages included the evaluation team • We identified evidence gaps but not entirely through a realist lens • Theory of change development was participatory but not done explicitly in a realist way

  8. Using the realist approach during project implementation: what went well • Learning and adaptation between the main evaluation deliverables: mid-term to endline and endline into the extension phase • We used data collection templates that were context and target group specific • Project implementers found it useful to have the evaluation findings structured around specific target groups and contexts • Generated some useful findings for the BRACED programme-level evaluation

  9. Using the realist approach during project implementation: challenges • More ongoing real-time assessments, tightly linked with the monitoring work done by the project staff • Sequencing of quantitative and qualitative data collection work • Finding the balance between project-wide generalisable findings and realist case study deep-dives • Project staff having sufficient time and capacity to support the realist evaluation work • Relatively short-time frame of the project

  10. Key lessons • Situational knowledge will always be important • Integrate the realist approach with planning focussed frameworks (e.g. co-design, adaptive management, developmental evaluation) • Develop the realist program theory concurrently with the design of the program, iterating back and forwards between them • Identify the learning-gaps before designing the project • Establish adequate processes and systems (workplans, adequate resources) • Be participatory (tailored communication, workshops)

  11. Example of linking the realist approach with other approaches • Westhorp, Stevens and Kaye (2016) used realist action research in ‘The Bridgewater Project’, to evaluate different social security payment options • Action research seeks to solve real world problems, trialing solutions until a ‘best fit’ solution is reached. • Also integrated this with a co-design approach to develop service, a collaborative approach to service design. • Benefits: helping project staff make sense of findings and identifying the needs of sub-groups

More Related