Methods for Assessing Policy Impact

1. Methods for Assessing Policy Impact Process and Partnerships for Pro-Poor Policy Change, Project Initiation Workshop 1 ILRI, 21st February 2005

2. Workshop Outline Introduction to the RAPID Framework and ILRI/ODI Project Case Study Approach Episode Study Approach Outcome Mapping Approach RAPID Outcome Assessment (ROA) Approach Lunch Practical Sessions

3. Workshop Purpose & Objectives Objectives By the end of the workshop, participants will: understand the Process and Partnership for Pro-Poor Policy Change project�s purpose and general approach have the opportunity to contribute their own suggestions to improve the project; understand, and have had the chance to try out the three key methods which will be used in the project; assess the usefulness of the approaches in their own work.

4. An introduction to the RAPID Framework and ILRI/ODI Project

5. Definitions Research: �any systematic effort to increase the stock of knowledge� Policy: a �purposive course of action followed by an actor or set of actors� Agendas / policy horizons Official statements documents Patterns of spending Implementation processes Activities on the ground We define both research and policy very broadly. By research we do not just mean classical scientific research. It include any systematic learning process - from theory building and data collection to evaluation action research. Similarly, policy is not just narrowly defined as a set of policy documents or legislation; it is about setting a deliberate course of action and then implementing it. It includes the setting of policy agendas, official policy documents, legislation, changes in patterns of government spending to implement policies, and the whole process of implementation. It is also about what happens on the ground: a policy is worth nothing unless it results in actual change. These are all relevant if we want to try to make policy more evidence-based and see the results of our research adopted in policy and practice. We define both research and policy very broadly. By research we do not just mean classical scientific research. It include any systematic learning process - from theory building and data collection to evaluation action research. Similarly, policy is not just narrowly defined as a set of policy documents or legislation; it is about setting a deliberate course of action and then implementing it. It includes the setting of policy agendas, official policy documents, legislation, changes in patterns of government spending to implement policies, and the whole process of implementation. It is also about what happens on the ground: a policy is worth nothing unless it results in actual change. These are all relevant if we want to try to make policy more evidence-based and see the results of our research adopted in policy and practice.

6. Policy Processes - Identify a policy problem - Commission research - Assess the results - Select the best policy - Establish the policy framework - Implement the policy - The problem is solved

7. Reality� �The whole life of policy is a chaos of purposes and accidents. It is not at all a matter of the rational implementation of the so-called decisions through selected strategies 1� �Most policy research on African agriculture is irrelevant to agricultural and overall economic policy in Africa2� Policy-making used to be widely thought of as a linear and logical process, in which policy-makers identified a problem, commissioned research, took note of the results and made sensible policies which were then implemented. Clearly that is not the case. Policy-making is a dynamic, complex, chaotic process. Clay and Schaffer�s book �Room for Manoeuvre� in 1984 described �the whole life of policy is a chaos of purposes and accidents. It is not at all a matter of the rational implementation of decisions through selected strategies�. That is increasingly recognised as a more realistic description of the policy process than the linear rational model � though the truth is probably somewhere in the middle. Furthermore, as Steve Omamo pointed out in a recent report on policy research on African agriculture: �Most policy research on African agriculture is irrelevant to agricultural and overall economic policy in Africa�. It is not really surprising that the link between research and policy is tenuous and difficult to understand if policy processes are complex and chaotic and much research is not very policy relevant. Policy-making used to be widely thought of as a linear and logical process, in which policy-makers identified a problem, commissioned research, took note of the results and made sensible policies which were then implemented. Clearly that is not the case. Policy-making is a dynamic, complex, chaotic process. Clay and Schaffer�s book �Room for Manoeuvre� in 1984 described �the whole life of policy is a chaos of purposes and accidents. It is not at all a matter of the rational implementation of decisions through selected strategies�. That is increasingly recognised as a more realistic description of the policy process than the linear rational model � though the truth is probably somewhere in the middle. Furthermore, as Steve Omamo pointed out in a recent report on policy research on African agriculture: �Most policy research on African agriculture is irrelevant to agricultural and overall economic policy in Africa�. It is not really surprising that the link between research and policy is tenuous and difficult to understand if policy processes are complex and chaotic and much research is not very policy relevant.

8. Existing theory Linear model Percolation model, Weiss Tipping point model, Gladwell �Context, evidence, links� framework, ODI Policy narratives, Roe Systems model (NSI) External forces, Lindquist �Room for manoeuvre�, Clay & Schaffer �Street level bureaucrats�, Lipsky Policy as social experiments, Rondinelli Policy Streams & Windows, Kingdon Disjointed incrementalism, Lindquist The �tipping point�, Gladwell Crisis model, Kuhn �Framework of possible thought�, Chomsky Variables for Credibility, Beach The source is as important as content, Gladwell Linear model of communication, Shannon Interactive model, Simple and surprising stories, Communication Theory Provide solutions, Marketing Theory I Find the right packaging, Marketing II Elicit a response, Kottler Translation of technology, Volkow Epistemic communities Policy communities Advocacy coalitions etc, Pross Negotiation through networks, Sebattier Shadow networks, Klickert Chains of accountability, Fine Communication for social change, Rockefeller Wheels and webs, Chapman & Fisher There is a vast amount of existing theory on this subject - you can read all about many of them on our website. But most of it is from developed, OECD countries and there is very little systematic research on the interface of research and policy in developing countries. This is a serious problem given the massive diversity of cultural, economic, and political contexts in the developing world � particularly given the weaker research and policymaking capabilities and democratic deficits that characterize some developing countries. Furthermore, international actors have an exaggerated impact on research and policy processes in the South. This makes it difficult to draw valid generalizations and lessons from existing experience and theory.There is a vast amount of existing theory on this subject - you can read all about many of them on our website. But most of it is from developed, OECD countries and there is very little systematic research on the interface of research and policy in developing countries. This is a serious problem given the massive diversity of cultural, economic, and political contexts in the developing world � particularly given the weaker research and policymaking capabilities and democratic deficits that characterize some developing countries. Furthermore, international actors have an exaggerated impact on research and policy processes in the South. This makes it difficult to draw valid generalizations and lessons from existing experience and theory.

9. Existing theory � a short list Malcolm Gladwell�s book �The Tipping Point� describes how social epidemics spread. It is about the different types of people who are involved in the policy process: connectors, who know a lot of people; mavens, who hoover up and digest information; and salesmen who are very good at �selling� ideas. He describes research into US news anchors in the run-up to the elections in the United States, which showed how very small differences in the way they behave on screen can give very strong messages to the public. He talks about how the context affects how people behave. In another experiment in the US, researchers sent student on errands all over the campus, and arranged for them to pass somebody in distress who clearly needed help and anaysed the factors which influenced whether the students stopped to help or not. The most important factor seemed to be whether the student was in a hurry or not. He describes how some ideas seem to be �sticky� - the factors that determine whether people remember specific bits of information. Gladwell describes how the conjunction of these factors create the �tipping points� when ideas suddenly spread and are adopted. Malcolm Gladwell�s book �The Tipping Point� describes how social epidemics spread. It is about the different types of people who are involved in the policy process: connectors, who know a lot of people; mavens, who hoover up and digest information; and salesmen who are very good at �selling� ideas. He describes research into US news anchors in the run-up to the elections in the United States, which showed how very small differences in the way they behave on screen can give very strong messages to the public. He talks about how the context affects how people behave. In another experiment in the US, researchers sent student on errands all over the campus, and arranged for them to pass somebody in distress who clearly needed help and anaysed the factors which influenced whether the students stopped to help or not. The most important factor seemed to be whether the student was in a hurry or not. He describes how some ideas seem to be �sticky� - the factors that determine whether people remember specific bits of information. Gladwell describes how the conjunction of these factors create the �tipping points� when ideas suddenly spread and are adopted.

10. An Analytical Framework The aim of our framework is to simplify the complexity of how evidence contributes to the policy process so that policy makers and researchers can make decisions about how they do their work to maximise the chance that policies are evidence-based, and that research does have a positive impact on policy and practice. It is based on a thorough review of the literature and a wide range of case studies at international, regional and national level across the developing world. It identifies four broad groups of factors. We call the first external influences. These are the factors outside a particular country which affect policy makers and policy processes within the country. Even in big countries such as India, international economic, trade and even cultural issues matter a great deal. In smaller, heavily indebted countries, World Bank and Bilateral Donor policies and practices can be very influential. At national level the factors fall into three main areas. The political context includes the people, institutions and processes involved in policy making. The evidence arena is about the type and quality of research and how it is communicated. The third arena links is about the mechanisms affecting how evidence gets into the policy process or not. The aim of our framework is to simplify the complexity of how evidence contributes to the policy process so that policy makers and researchers can make decisions about how they do their work to maximise the chance that policies are evidence-based, and that research does have a positive impact on policy and practice. It is based on a thorough review of the literature and a wide range of case studies at international, regional and national level across the developing world. It identifies four broad groups of factors. We call the first external influences. These are the factors outside a particular country which affect policy makers and policy processes within the country. Even in big countries such as India, international economic, trade and even cultural issues matter a great deal. In smaller, heavily indebted countries, World Bank and Bilateral Donor policies and practices can be very influential. At national level the factors fall into three main areas. The political context includes the people, institutions and processes involved in policy making. The evidence arena is about the type and quality of research and how it is communicated. The third arena links is about the mechanisms affecting how evidence gets into the policy process or not.

11. Case Studies Sustainable Livelihoods: The Evolution of DFID Policy The PRSP Initiative: Research in Multilateral Policy Change The adoption of Ethical Principles in Humanitarian Aid post Rwanda Animal Health Care in Kenya: Evidence fails to influence Policy

12. A Practical Framework An interesting thing about the framework is how well it maps onto real-life activities. The political context sphere maps onto politics and policy making, evidence onto the processes of research, learning and thinking, and links onto networking, the media and advocacy. Even the overlapping areas map onto recognisable activities. The intersection of the political context and evidence represents the process of policy analysis � the study of how to implement and the likely impact of specific policies. The overlap between evidence and links is the process of academic discourse through publications and conferences, and the area between links and political context is the world of campaigning and lobbying. The area in the middle � the bulls-eye � where convincing evidence providing a practical solution to a current policy problem, that is supported by and brought to the attention of policymakers by actors in all three areas is where there is likely to be the most immediate link between evidence and policy. An interesting thing about the framework is how well it maps onto real-life activities. The political context sphere maps onto politics and policy making, evidence onto the processes of research, learning and thinking, and links onto networking, the media and advocacy. Even the overlapping areas map onto recognisable activities. The intersection of the political context and evidence represents the process of policy analysis � the study of how to implement and the likely impact of specific policies. The overlap between evidence and links is the process of academic discourse through publications and conferences, and the area between links and political context is the world of campaigning and lobbying. The area in the middle � the bulls-eye � where convincing evidence providing a practical solution to a current policy problem, that is supported by and brought to the attention of policymakers by actors in all three areas is where there is likely to be the most immediate link between evidence and policy.

13. What you need to know The external environment: Who are the key actors? What is their agenda? How do they influence the political context? The political context: Is there political interest in change? Is there room for manoeuvre? How do they perceive the problem? The evidence: Is it there? Is it relevant? Is it practically useful? Are the concepts familiar or new? Does it need re-packaging? Links: Who are the key individuals? Are there existing networks to use? How best to transfer the information? The media? Campaigns? So, if you are a researcher, policy maker or development practitioner with the desire to promote a particular policy you need to know about: the external environment which might influence how people think or behave: who are the key external actors? what is their agenda? And how do they influence the political context? the political context you are working in: is there political interest in change? is there room for manoeuvre? how do policy makers perceive the problem? the evidence you have, or could get: is there enough of it? is it convincing? is it relevant? is it practically useful? are the concepts familiar or new? does it need re-packaging? and the links that exist to bring the evidence to the attention of policy makers: who are the key organisations and individuals? are there existing networks to use? What�s the best way to transfer the information: face-to-face or through the media or campaigns?So, if you are a researcher, policy maker or development practitioner with the desire to promote a particular policy you need to know about: the external environment which might influence how people think or behave: who are the key external actors? what is their agenda? And how do they influence the political context? the political context you are working in: is there political interest in change? is there room for manoeuvre? how do policy makers perceive the problem? the evidence you have, or could get: is there enough of it? is it convincing? is it relevant? is it practically useful? are the concepts familiar or new? does it need re-packaging? and the links that exist to bring the evidence to the attention of policy makers: who are the key organisations and individuals? are there existing networks to use? What�s the best way to transfer the information: face-to-face or through the media or campaigns?

14. What researchers need to do For researchers wishing to influence policy and practice, understanding the context, evidence and links is just the first part of the process. Our case studies also identify a number of practical things that researchers need to do to influence policy and practice, and how to do it. In the political context arena you need to get to know the policymakers, identify friends and foes, prepare for regular policy opportunities and look out for policy windows. One of the best ways is to work with them through commissions, and establish an approach that combines a strategic focus on current issues with the ability to respond rapidly to unexpected opportunities. Make sure your evidence is credible. This has much more to do with your long term reputation than the scientific credibility of an individual piece of research. Provide practical solutions to policy problems in familiar language and concepts. Action-research using pilot projects to generate legitimacy seems to be particularly powerful. Make the most of the existing links by getting to know the other actors, working through existing networks and building coalitions and partnerships. Identify the key individuals who can help. You need people who can network with others, mavens to absorb and process information, and good salesmen who can convince the sceptics. You may also need to use informal �shadow networks� as well as more formal channels. For researchers wishing to influence policy and practice, understanding the context, evidence and links is just the first part of the process. Our case studies also identify a number of practical things that researchers need to do to influence policy and practice, and how to do it. In the political context arena you need to get to know the policymakers, identify friends and foes, prepare for regular policy opportunities and look out for policy windows. One of the best ways is to work with them through commissions, and establish an approach that combines a strategic focus on current issues with the ability to respond rapidly to unexpected opportunities. Make sure your evidence is credible. This has much more to do with your long term reputation than the scientific credibility of an individual piece of research. Provide practical solutions to policy problems in familiar language and concepts. Action-research using pilot projects to generate legitimacy seems to be particularly powerful. Make the most of the existing links by getting to know the other actors, working through existing networks and building coalitions and partnerships. Identify the key individuals who can help. You need people who can network with others, mavens to absorb and process information, and good salesmen who can convince the sceptics. You may also need to use informal �shadow networks� as well as more formal channels.

15. Policy entrepreneurs Doing all of these things requires a wide range of skills. Researchers who want to be good policy entrepreneurs also need to be: Storytellers: Practitioners, bureaucrats and policy-makers often articulate and make sense of complex realities through simple stories. Though sometimes profoundly misleading there is no doubt that narratives are incredibly powerful. Networkers: Policy-making usually takes place within communities of people who know each other and interact. If you want to influence policymakers, you need to join their networks. Engineers: There is often a huge gap between what politicians and policy-makers say they are doing and what actually happens on the ground. Researchers need to work not just with the senior level policy-makers, but also with the 'street-level bureaucrats'. Fixers: Policy making is essentially a political process. Although you don�t need to be a Rasputin or Machiavelli, successful policy entrepreneurs need to know how to operate in a political environment - when to make your pitch, to whom and how. Try ODI�s Policy Entrepreneur Questionnaire to find out whether you tend to favour, or to avoid any of these activities. If so, you may need to develop new skills in these areas, or work with others who have these skills.Doing all of these things requires a wide range of skills. Researchers who want to be good policy entrepreneurs also need to be: Storytellers: Practitioners, bureaucrats and policy-makers often articulate and make sense of complex realities through simple stories. Though sometimes profoundly misleading there is no doubt that narratives are incredibly powerful. Networkers: Policy-making usually takes place within communities of people who know each other and interact. If you want to influence policymakers, you need to join their networks. Engineers: There is often a huge gap between what politicians and policy-makers say they are doing and what actually happens on the ground. Researchers need to work not just with the senior level policy-makers, but also with the 'street-level bureaucrats'. Fixers: Policy making is essentially a political process. Although you don�t need to be a Rasputin or Machiavelli, successful policy entrepreneurs need to know how to operate in a political environment - when to make your pitch, to whom and how. Try ODI�s Policy Entrepreneur Questionnaire to find out whether you tend to favour, or to avoid any of these activities. If so, you may need to develop new skills in these areas, or work with others who have these skills.

16. Practical Tools Influencing policy change is an art as much as a science, but there are a wide range of well known and often straightforward tools that can provide powerful insights and help to maximize your chances of impact on policy. We�ve already seen how ODI�s RAPID Framework can help you to understand the context you are working in and how you could use the Policy Entrepreneur Questionnaire to figure out what you are good at. Other useful tools to help to understand the policy context include Stakeholder Analysis, Forcefield Analysis, Writeshops, Policy Mapping and Political Context Mapping. This is vital in terms of developing an influence strategy. There is a wide set of research tools � from case studies to action research � that can help generate new or better evidence to support your case. The key communications questions are: Who do I want to convince? What do I want them to do? What will convince them? What relevant material do I have? A SWOT analysis can help to focus a communications strategy on the key messages and targets, and using the media can help you to reach a wide audience. Many tools have also been developed by organisations involved in lobbying, advocacy and campaigning for pro-poor change.Influencing policy change is an art as much as a science, but there are a wide range of well known and often straightforward tools that can provide powerful insights and help to maximize your chances of impact on policy. We�ve already seen how ODI�s RAPID Framework can help you to understand the context you are working in and how you could use the Policy Entrepreneur Questionnaire to figure out what you are good at. Other useful tools to help to understand the policy context include Stakeholder Analysis, Forcefield Analysis, Writeshops, Policy Mapping and Political Context Mapping. This is vital in terms of developing an influence strategy. There is a wide set of research tools � from case studies to action research � that can help generate new or better evidence to support your case. The key communications questions are: Who do I want to convince? What do I want them to do? What will convince them? What relevant material do I have? A SWOT analysis can help to focus a communications strategy on the key messages and targets, and using the media can help you to reach a wide audience. Many tools have also been developed by organisations involved in lobbying, advocacy and campaigning for pro-poor change.

17. Practical Application Within ODI Workshops for researchers, policy makers and activists. Advice to a DFID forest/ground water research project in India: Less research More communication Developing champions in regional and national government Local, Regional & National advocacy campaign

18. Further Information / Resources ODI Working Papers Bridging Research and Policy Book Meeting series Monograph Tools for Policy Impact RAPID Briefing Paper www.odi.org.uk/rapid Malcolm Gladwell�s book �The Tipping Point� describes how social epidemics spread. It is about the different types of people who are involved in the policy process: connectors, who know a lot of people; mavens, who hoover up and digest information; and salesmen who are very good at �selling� ideas. He describes research into US news anchors in the run-up to the elections in the United States, which showed how very small differences in the way they behave on screen can give very strong messages to the public. He talks about how the context affects how people behave. In another experiment in the US, researchers sent student on errands all over the campus, and arranged for them to pass somebody in distress who clearly needed help and anaysed the factors which influenced whether the students stopped to help or not. The most important factor seemed to be whether the student was in a hurry or not. He describes how some ideas seem to be �sticky� - the factors that determine whether people remember specific bits of information. Gladwell describes how the conjunction of these factors create the �tipping points� when ideas suddenly spread and are adopted. Malcolm Gladwell�s book �The Tipping Point� describes how social epidemics spread. It is about the different types of people who are involved in the policy process: connectors, who know a lot of people; mavens, who hoover up and digest information; and salesmen who are very good at �selling� ideas. He describes research into US news anchors in the run-up to the elections in the United States, which showed how very small differences in the way they behave on screen can give very strong messages to the public. He talks about how the context affects how people behave. In another experiment in the US, researchers sent student on errands all over the campus, and arranged for them to pass somebody in distress who clearly needed help and anaysed the factors which influenced whether the students stopped to help or not. The most important factor seemed to be whether the student was in a hurry or not. He describes how some ideas seem to be �sticky� - the factors that determine whether people remember specific bits of information. Gladwell describes how the conjunction of these factors create the �tipping points� when ideas suddenly spread and are adopted.

19. Yes, but: It this its role? �Global Public Good� Research vs Policy Advocacy Probably needs to do both: How? Understand the political context Get the evidence & package it well Strategic networking / lobbying / campaigning Collaboration�. Can ILRI do it?

21. Project Leaders: ODI / ILRI Key collaborators: ECAPAPA Case study collaborators in Kenya: MoLFD / KARI Range of NGOs & other SDP partners

22. Not all research is expected or intended to lead to policy change, but there may be; Specific cases where research is expected to; provide evidence for policy change identify potential policies (or impact of) influence the policy making process (advocacy) Cases where speculative research becomes relevant because of changes in circumstance

23. Ideas for methods and approaches Lessons learnt from earlier activities Identification of appropriate communication tools

24. Three case studies in three DIFFERENT countries A project considered to have influenced policy change A stream of research addressing a particular policy area A clear policy change; New policy statement New law Irrefutable change in way something is done

25. Three case studies SDP and impact on changed view of informal milk trade ???? ????

26. Discussion: Can ILRI hope to influence pro-poor policy through research? Any good case studies?

27. Case Study Approach

28. What is a Case Study? Definition: " A systematic inquiry into an event or a set of related events which aims to describe and explain the phenomenon of interest" Bromley (1990)

29. Why is it useful? An ideal methodology when a holistic, in-depth investigation is needed Designed to bring out the details from the viewpoint of the participants by using multiple sources of data

30. Types of Case Study Types of case studies: Exploratory, Explanatory, Descriptive (Yin, 1993) Stake (1995) included three others: Intrinsic - when the researcher has an interest in the case; Instrumental - when the case is used to understand more than what is obvious to the observer; Collective - when a group of cases is studied.

31. Issues The unit of analysis is a critical factor Typically a system of action rather than an individual or group of individuals Tend to be selective, focusing on one or two issues that are fundamental to understanding the system being examined Case studies are multi-perspectival analyses The researcher considers not just the voice and perspective of the actors, but also of the relevant groups of actors and the interaction between them They give a voice to the powerless and voiceless.

32. Triangulation Data source triangulation, when the researcher looks for the data to remain the same in different contexts; Investigator triangulation, when several investigators examine the same phenomenon; Theory triangulation, when investigators with different view points interpret the same results; and Methodological triangulation, when one approach is followed by another, to increase confidence in the interpretation.

33. Applications To explain complex causal links between research and policy To describe the real-life context in which policy has been influenced by research To describe the policy influencing process itself To explore those situations in which the policy intervention being evaluated has no clear set of outcomes.

34. Process Design the case study protocol: determine the required skills develop and review the protocol Conduct the case study: prepare for data collection distribute questionnaire conduct interviews Analyze case study evidence: analytic strategy Develop conclusions, recommendations, and implications based on the evidence

35. Episode Study Approach

36. What is an Episode Study �A study that focuses on a clear policy change and tracks back to assess what impact research had among the variety of issues that led to the policy change�.

37. an excellent way of investigating the influence of research on policy Can focus on a single episode or comparative episodes. What is the purpose?

38. The process of working backwards in time gives a more realistic view of the broad range of factors � other than research � that influence policy Tracking forward probably overemphasizes the importance of research Advantage

39. Policy processes are complex, multi-layered and change over time Often difficult to isolate the impact of research from other factors Actors may �re-write history� Important to seek the views of a wide range of informed stakeholders The process of preparing an episode study is iterative Key facts and / or inconsistencies need to be cross-checked with key informants Issues

40. Identify a clear policy change. Identify key Research Questions (draw on RAPID framework) Explore how and why those policy decisions and practices took place Assess the relative role of research in that process by drawing on the framework. Process

41. Apply the RAPID Framework An interesting thing about the framework is how well it maps onto real-life activities. The political context sphere maps onto politics and policy making, evidence onto the processes of research, learning and thinking, and links onto networking, the media and advocacy. Even the overlapping areas map onto recognisable activities. The intersection of the political context and evidence represents the process of policy analysis � the study of how to implement and the likely impact of specific policies. The overlap between evidence and links is the process of academic discourse through publications and conferences, and the area between links and political context is the world of campaigning and lobbying. The area in the middle � the bulls-eye � where convincing evidence providing a practical solution to a current policy problem, that is supported by and brought to the attention of policymakers by actors in all three areas is where there is likely to be the most immediate link between evidence and policy. An interesting thing about the framework is how well it maps onto real-life activities. The political context sphere maps onto politics and policy making, evidence onto the processes of research, learning and thinking, and links onto networking, the media and advocacy. Even the overlapping areas map onto recognisable activities. The intersection of the political context and evidence represents the process of policy analysis � the study of how to implement and the likely impact of specific policies. The overlap between evidence and links is the process of academic discourse through publications and conferences, and the area between links and political context is the world of campaigning and lobbying. The area in the middle � the bulls-eye � where convincing evidence providing a practical solution to a current policy problem, that is supported by and brought to the attention of policymakers by actors in all three areas is where there is likely to be the most immediate link between evidence and policy.

42. Key Questions The external environment: Who are the key actors? What is their agenda? How do they influence the political context? The political context: Is there political interest in change? Is there room for manoeuvre? How do they perceive the problem? The evidence: Is it there? Is it relevant? Is it practically useful? Are the concepts familiar or new? Does it need re-packaging? Links: Who are the key individuals? Are there existing networks to use? How best to transfer the information? The media? Campaigns? So, if you are a researcher, policy maker or development practitioner with the desire to promote a particular policy you need to know about: the external environment which might influence how people think or behave: who are the key external actors? what is their agenda? And how do they influence the political context? the political context you are working in: is there political interest in change? is there room for manoeuvre? how do policy makers perceive the problem? the evidence you have, or could get: is there enough of it? is it convincing? is it relevant? is it practically useful? are the concepts familiar or new? does it need re-packaging? and the links that exist to bring the evidence to the attention of policy makers: who are the key organisations and individuals? are there existing networks to use? What�s the best way to transfer the information: face-to-face or through the media or campaigns?So, if you are a researcher, policy maker or development practitioner with the desire to promote a particular policy you need to know about: the external environment which might influence how people think or behave: who are the key external actors? what is their agenda? And how do they influence the political context? the political context you are working in: is there political interest in change? is there room for manoeuvre? how do policy makers perceive the problem? the evidence you have, or could get: is there enough of it? is it convincing? is it relevant? is it practically useful? are the concepts familiar or new? does it need re-packaging? and the links that exist to bring the evidence to the attention of policy makers: who are the key organisations and individuals? are there existing networks to use? What�s the best way to transfer the information: face-to-face or through the media or campaigns?

43. � review of the literature; � interviews with key actors; � capturing the authors� own experience; and � discussions at workshops. Methods

44. Episode Study Examples

45. 1970s 1980s 1990s 2000s Paravets in Kenya Shortly after setting up its first decentralised animal health projects, ITDG organised the first, of what were to become annual �vets workshops� which became the focus of a network of people involved in paravet projects. A conscious effort was made at the start to invite senior government veterinary staff to participate to convince them of the value of the approach. Many NGO and bilateral project staff who were already involved in, or wanted to start decentralised animal health projects, were keen to join the network, and it increasingly focused on practical issues. While this contributed to the rapid spread of the approach across northern Kenya, it neglected to involve senior government policy makers. Dr Kajume, then Provincial Head of Veterinary Services heard about one of these workshops by accident, checked with the Director of Veterinary Services in Nairobi, and was instructed to attend the workshop, tell the participants it was illegal, and close it down. But instead, he became convinced of the value of the approach and persuaded the Director to allow the projects to continue. So paravet projects continued to spread across northern Kenya, deliberately ignored by the Director of Veterinary Services, until the publication of the Kenya Vet Board letter brought matters to a head, and he was forced to do something about it. At which point, Dr Kajume, now Deputy Director of Veterinary Services persuaded him to support the process of multi stakeholder workshops and commission the Hubl study which led to the development of a new policy framework. Shortly after setting up its first decentralised animal health projects, ITDG organised the first, of what were to become annual �vets workshops� which became the focus of a network of people involved in paravet projects. A conscious effort was made at the start to invite senior government veterinary staff to participate to convince them of the value of the approach. Many NGO and bilateral project staff who were already involved in, or wanted to start decentralised animal health projects, were keen to join the network, and it increasingly focused on practical issues. While this contributed to the rapid spread of the approach across northern Kenya, it neglected to involve senior government policy makers. Dr Kajume, then Provincial Head of Veterinary Services heard about one of these workshops by accident, checked with the Director of Veterinary Services in Nairobi, and was instructed to attend the workshop, tell the participants it was illegal, and close it down. But instead, he became convinced of the value of the approach and persuaded the Director to allow the projects to continue. So paravet projects continued to spread across northern Kenya, deliberately ignored by the Director of Veterinary Services, until the publication of the Kenya Vet Board letter brought matters to a head, and he was forced to do something about it. At which point, Dr Kajume, now Deputy Director of Veterinary Services persuaded him to support the process of multi stakeholder workshops and commission the Hubl study which led to the development of a new policy framework.

46. PRSPs � Political Context Widespread awareness of a �problem� with international development policy in late 90s Failure of SAPs (and Asian financial crisis) Mounting public pressure for debt relief Stagnation of Comprehensive Development Framework idea Diverging agendas (UK � Poverty, US � Governance) WB/IMF Annual General Meeting, Sept 1999

47. PRSPs � Evidence Long-term academic research informing new focus on poverty, participation, ownership, aid effectiveness etc Applied policy research: ESAF reviews HIPC review SPA Working Groups NGO research on debt Uganda�s PEAP

48. PRSPs � Links WB, IMF, SPA, Bilaterals, NGOs all involved Formal an informal networks �None of the players was more than two handshakes away from any of the others�

49. Outcome Mapping

50. What is it? an integrated PM&E tool a system to think holistically & strategically about how we intend to achieve result an approach that focuses on changes in the behaviour, relationships or actions of partners (as outcomes) a methodology that characterizes and assesses the program�s contributions to the achievement of outcomes an approach for designing in relation to the broader development context but assessing within your sphere of influence

51. Focus: On Behavioural Change

52. For a program to tell its performance story in outcome terms by: articulating its goals and designing its activities designing a monitoring system for assessing internal performance and outcomes of partners setting a use-oriented evaluation plan How can it be used?

53. Focussing on changes in partners� behaviour, relationships, or actions allows a program to: measure results within its sphere of influence obtain feedback about its efforts in order to improve its performance take credit for its contributions to the achievement of outcomes show progress towards outcomes Why use it?

54. Outcomes: changes in behaviours, relationships, activities and/or actions of the people, groups and organisations with whom we work Vision: the broad human, social and environmental betterment we desire Mission: how we intend to contribute towards the achievement of the vision Boundary partners: individuals, groups and organisations with whom we interact directly to effect change Outcome challenges: changes behaviours of the boundary partners as identified by the vision Terminology

55. The Three Stages

56. Intentional Study Design

57. Performance Monitoring

58. Evaluation Planning

59. Main Elements

60. RAPID Outcome Assessment

61. What is it? A Visual Tool Combines the outcome mapping concept within a case study & episode study approach Systematic approach to collecting information about changes in behaviour of key project partners that contributed to the policy change Assessment of the contribution of the project (programme, strategy, etc.) to observed changes in behaviour �and ultimately to the policy change

62. Approach Describe policy environment at end Describe policy environment at the beginning Identify the key policy actors Identify key boundary partners Describe boundary partner behaviour at end Describe boundary partner behaviour at beginning Describe changes in BP behaviour Describe changes in project (strategic/opportunistic) Describe external influences Determine level of impact of changes in project Determine level of impact of external influences Check through external interviews Write report

63. Sources & Outputs Literature review - Project background, progress, (published) achievements Participatory workshop with staff (and BP) Gather detailed information Identify issues for further investigation Interviews with key informants to: Triangulate the result of the workshop, Fill the gaps of information Clarify causality Report Writing Visual and Narrative

64. ROA Terms and Definitions Boundary partners: individuals, groups and organisations with whom we interact directly to effect change. Outcomes: changes in behaviours, relationships, activities and/or actions of the people, groups and organisations with whom we work. Behaviours: the way we or our boundary partners do or think about things.

65. The key steps of the ROA framework Describe the policy environment at the end of the project

66. The key steps of the ROA framework 2. Describe the policy environment at the beginning of the project

67. The key steps of the ROA framework

68. The key steps of the ROA framework 5. Describe the behaviours of the boundary partners that contributed to the change in the policy environment or policy

69. The key steps of the ROA framework Describe the behaviours of the boundary partners at the beginning of the project

70. The key steps of the ROA framework 7. Map the key changes in behaviour for each boundary partner from the start of the project

71. The key steps of the ROA framework 8. Map the key changes in the project including organisational changes, outputs and changes in behaviour during the same period.

72. The key steps of the ROA framework Map the external influences including the actions f strategic partners and other exogenous factors during the same period

73. The key steps of the ROA framework 10. Determine the level of impact/influence of the project on the changes in behaviour of the boundary partners

74. The key steps of the ROA framework 11. Determine the level of impact/influence of external influences on the changes in behaviour of the boundary partners and the project

75. The key steps of the ROA framework 12. Refine conclusions with in-depth interviews and assess the real contribution of the project on the policy change