Impact Assessment Training 10/11 th September 2013 Oslo

1. Impact Assessment Training 10/11th September 2013 Oslo

2. Workshop Outcomes By the end of these two days you will have: • Developed a common understanding of what is meant by impact assessment; and how it differs from and complements the processes of monitoring and evaluation. • Identified key challenges that you face in conducting impact assessments • Worked with pre-selected case studies  to design an impact assessment process • Considered how the results of impact assessments might be used in learning, reporting and being more accountable  to both donors and stakeholders • Identified ways in which you can take forward learning from this workshop.

3. The vicious circle

4. Impact So what do we mean when we talk about Impact?

5. Impact Think about an event or a person who made a significant impact on your life (a birth or death of a family member or friend, a relationship, a marriage, an accident) • What does impact mean in relation to this? • What concepts or ideas does it include?

6. How do you “assess” this impact? • ... You try to understand the nature of the change that has taken place in you and to determine its significance in your life.

7. Impact Assessment “The systematic analysis of significant and/or lasting change – positive or negative, intended or not – in the lives of target groups, brought about by a given action or a series of actions”

8. Its not that easy... consider Danny..

9. Why assess impact? • To understand the implications of our work • To become more accountable to those we work with (stakeholders) • To support institutional learning and decision making and improve future work • To contribute to policy development and effective advocacy • To help demonstrate organisational performance

10. Key areas of enquiry for impact Five essential questions: • What has changed? • For whom? • How significant was it? • Will it last? • In what ways did we contribute to these changes?

11. The challenge for this workshop To be able to develop Impact Assessment processes which are: • Simple and user-friendly • Build on existing structures and systems • Are useful for accountability both upwards and downwards • Are useful for your organisational learning

12. Session 2 Relationship between M&E and Impact Assessment

13. M&E or Impact Assessment? • How is the assessment of impact different from the processes of monitoring and evaluation? • What to do? • When to do it?

14. Example In a project to build social housing for a local community, for example: • Monitoring would relate to the purchase of materials, and building the houses according to plans that have been drawn up. • Evaluationwouldassess the results of these efforts: how good was the plan? How well were the houses built? Was the project cost effective etc

15. But... • It’s possible to have well-built cost effective housing schemes which are no use to those for whom they were intended (e.g. aborigine “settlements”). • Thousands of “successful projects and programmes” which fail to make a positive impact on the lives of people they aim to serve. Some projects result in negative impacts. Impact Assessment addresses the “So what?” question: as a result of our efforts, what’s changed for whom; and how significant is this for them?

16. Impact – so what?! Thousands of “successful” projects and programmes make no lasting difference to people’s lives. Impact Assessment addresses the “So What?” Question As a result of our efforts, what has changed for whom; and how significant is this for them?

17. Differences in Brief

18. Log Frame Approache.g. Skills training programme - a results chain... Does it always work like this???

19. Small groups’ task Small UK based NGO, working through partners in Ethiopia in Education Goal: All children have access to free basic education • What high level outcomes would ensure that this goal is achieved?

20. Small groups’ task • Select one outcome • Discuss what activities/outputs would lead to the achievement of this outcome • Develop this into a mini project • Develop indicators at each level • Use different colour post its!

21. Small groups’ task

22. Analysing the results To what extent are you able to assess the key questions for impact assessment? • What has changed? • For whom? • How significant/lasting are these changes for different target groups? • What -if anything - did our programme (project) contribute?

23. Measuring impact based on results chain Tends to be limited to assessing what you expect/hope will change... Good for demonstrating results to donors..

24. So... • So how else to assess impact? • What are the challenges? • What different approaches could we use? Answers this afternoon!

25. Session 3 Focus on change and how to assess it

26. Thinking about Change • How does change happen? • How do/can development organisations conceptualise this? • What is their realistic sphere of influence on changes that do take place?

28. Change is.. Complex in that many different changes can take place simultaneously in people’s lives • Continuous in that nothing in society or the environment is ever static • Variable in pace, scale and/or over the course of any intervention • Not necessarily lasting or sustainable.

29. What do we need to know to assess impact? • Who or what was involved in the change? (e.g. individual actors or state institutions) • What strategies were used to bring about the change? (e.g. reform, mass mobilization) • What were the contexts that affected how the change happened? (e.g. urbanization, power relationships) • What was the process or pathway of change? (e.g. demonstration effects, cumulative progress) • How were our efforts connected to this?

30. Organisational response to this? • Many organisations are developing Theories of Change to better support their understanding of how change happens and their role in the process

31. Four Key Elements

32. How do they complement other planning and M&E processes? • Strategic plans? • Log frames? • M&E systems? • Learning loops?

33. 3. ToC doesn’t replace results frameworks - they make them stronger ToC = critical analysis process Logic Model = Accountability Framework which is informed by ToC

34. Summary of some of the differences

35. Examples of Impact Assessment Frameworks:

36. Contributes To change in: Contributes to: Contributes to: • Marginalised • women are empowered and access rights, opportunities and services • including: • Equal access to property and resources • Reduction in harmful traditional practises • Increased representation in local, regional and national leadership positions • Increased economic empowerment • Reduction in violence and exploitation against women? Capacity Building to partners Power holders at local and regional level ensure that all women access rights, opportunities and services MRDF Increased partner capacity Awareness of rights Funds for Capacity Building Marginalised women have the confidence, knowledge and skills to access rights, opportunities and services Funds for partners Access to information Activities proposed by partners Access to support Marginalised women are organised, active and influential and play an active role in society Access to credit Direct sphere of influence Access to resources Marginalised women are supported and empowered by their families, society and culture Long Term Impact Access to training Theory of Change – Women's Empowerment (MRDF as a stone making ripples in the water)

37. Session 4

38. Approaches and strategies • Three approaches: • Post programme: Testing logic of log frame ( impact “evaluation”) • Participatory ToC approach which is used to design monitor and assess efforts • Research: looking back sometime later and assessing changes and their relation to programme efforts • Four strategies: • Build into existing M&E • Tracer and tracker studies • Ensure key moments of critical reflection • Commission a retrospective study

39. Plans for tomorrow Case Study Task

40. Day 2

41. Session 1 Case Study Task 1

42. Reflection on yesterday

43. Main steps in designing an Impact Assessment • Define purpose, approach and scope of the assessment • Develop/confirm theory of change and/or dimensions of change • Develop a list of areas of enquiry which will help you to explore impact • Select tools and methods for gathering and analysing relevant information • Decide on reporting strategies • Make concrete plans and timelines (who, when, how....)

44. The challenges To develop a process which is: • Simple and user-friendly • Build on existing structures and systems • Are useful for accountability both upwards and downwards • Are useful for your organisational learning And the challenges that we identified yesteday

45. ACCOUNTABILITY TO THE POOR

46. Case Study Task 1 In small groups: • Share the details of the programme you are working on (you need to be clear about context, overall goal and top level outcomes)

47. Case Study Task 1 • What will be the main purpose of doing this impact assessment? Organisational learning? To meet donor demands? Accountability to stakeholders? For advocacy • Which approach (or combination of approaches?) to Impact Assessment would be most appropriate in this Why? • What will be the scope and scale of this assessment?

48. Session 2 Case Study Task 2

49. Case Study Task 2: • What is your realistic “scope of influence”? Which what areas of impact will you realistically be able to “assess”, and which areas of change will you be able to “illustrate contributions to change”? • Based on this, what “Dimensions of Change” will you be looking to assess? • Develop a menu of areas of enquiry which will enable you to set baselines and track progress in relation to impact

50. Being clear about your scope of influence • Sphere of indirect influence • (there are other factors/actors • which influence changes • you want to see) • Assessment process less • rigorous – • illustrations of impact Sphere of direct influence (you are working directly with target groups on specific changes) Need to assess impact rigorously