methods for assessing policy impact n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Methods for Assessing Policy Impact PowerPoint Presentation
Download Presentation
Methods for Assessing Policy Impact

Loading in 2 Seconds...

play fullscreen
1 / 75

Methods for Assessing Policy Impact - PowerPoint PPT Presentation


  • 119 Views
  • Uploaded on

Methods for Assessing Policy Impact. Process and Partnerships for Pro-Poor Policy Change, Project Initiation Workshop 1 ILRI, 21st February 2005. Workshop Outline. Introduction to the RAPID Framework and ILRI/ODI Project Case Study Approach Episode Study Approach Outcome Mapping Approach

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Methods for Assessing Policy Impact' - jerry


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
methods for assessing policy impact

Methods for Assessing Policy Impact

Process and Partnerships for Pro-Poor Policy Change, Project Initiation Workshop 1

ILRI, 21st February 2005

workshop outline
Workshop Outline
  • Introduction to the RAPID Framework and ILRI/ODI Project
  • Case Study Approach
  • Episode Study Approach
  • Outcome Mapping Approach
  • RAPID Outcome Assessment (ROA) Approach

Lunch

  • Practical Sessions
workshop purpose objectives
Workshop Purpose & Objectives

Objectives

By the end of the workshop, participants will:

  • understand the Process and Partnership for Pro-Poor Policy Change project’s purpose and general approach
  • have the opportunity to contribute their own suggestions to improve the project;
  • understand, and have had the chance to try out the three key methods which will be used in the project;
  • assess the usefulness of the approaches in their own work.

Purpose:

To familiarise the participants with the general approach and specific methods to be used in the SDP case study

definitions
Definitions
  • Research: “any systematic effort to increase the stock of knowledge”
  • Policy:a “purposive course of action followed by an actor or set of actors”
    • Agendas / policy horizons
    • Official statements documents
    • Patterns of spending
    • Implementation processes
    • Activities on the ground
policy processes
Policy Processes

- Identify a policy problem

- Commission research

- Assess the results

- Select the best policy

- Establish the policy framework

- Implement the policy

- The problem is solved

reality
Reality…
  • “The whole life of policy is a chaos of purposes and accidents. It is not at all a matter of the rational implementation of the so-called decisions through selected strategies1”
  • “Most policy research on African agriculture is irrelevant to agricultural and overall economic policy in Africa2”

1 - Clay & Schaffer (1984), Room for Manoeuvre; An Exploration of Public Policy in

Agricultural and Rural Development, Heineman Educational Books, London

2 – Omamo (2003), Policy Research on African Agriculture: Trends, Gaps, and Challenges,

International Service for National Agricultural Research (ISNAR) Research Report No 21

existing theory
Linear model

Percolation model, Weiss

Tipping point model, Gladwell

‘Context, evidence, links’ framework, ODI

Policy narratives, Roe

Systems model (NSI)

External forces, Lindquist

‘Room for manoeuvre’, Clay & Schaffer

‘Street level bureaucrats’, Lipsky

Policy as social experiments, Rondinelli

Policy Streams & Windows, Kingdon

Disjointed incrementalism, Lindquist

The ‘tipping point’, Gladwell

Crisis model, Kuhn

‘Framework of possible thought’, Chomsky

Variables for Credibility, Beach

The source is as important as content, Gladwell

Linear model of communication, Shannon

Interactive model,

Simple and surprising stories, Communication Theory

Provide solutions, Marketing Theory I

Find the right packaging, Marketing II

Elicit a response, Kottler

Translation of technology, Volkow

Epistemic communities

Policy communities

Advocacy coalitions etc, Pross

Negotiation through networks, Sebattier

Shadow networks, Klickert

Chains of accountability, Fine

Communication for social change, Rockefeller

Wheels and webs, Chapman & Fisher

Existing theory

www.odi.org.uk/rapid/lessons/theory

existing theory a short list
Existing theory – a short list
  • Policy narratives, Roe
  • Systems of Innovation Model, (NSI)
  • ‘Room for manoeuvre’, Clay & Schaffer
  • ‘Street level bureaucrats’, Lipsky
  • Policy as social experiments, Rondene
  • Policy streams and policy windows, Kingdon
  • Disjointed Incrementalism, Lindblom
  • Social Epidemics, Gladwell
  • The RAPID Framework
an analytical framework

The political context – political and economic structures and processes, culture, institutional pressures, incremental vs radical change etc.

The links between policy

and research communities – networks, relationships, power, competing discourses, trust, knowledge etc.

The evidence – credibility, the degree it challenges received wisdom, research approaches and methodology, simplicity of the message, how it is packaged etc

An Analytical Framework

External Influences

Socio-economic and

cultural influences,

donor policies etc

case studies
Case Studies
  • Sustainable Livelihoods: The Evolution of DFID Policy
  • The PRSP Initiative: Research in Multilateral Policy Change
  • The adoption of Ethical Principles in Humanitarian Aid post Rwanda
  • Animal Health Care in Kenya: Evidence fails to influence Policy
a practical framework

External Influences

Campaigning, Lobbying

Policy analysis, & research

Scientific information exchange & validation

A Practical Framework

political context

Politics and Policymaking

Media,

Advocacy, Networking

Research, learning & thinking

evidence

links

what you need to know
What you need to know
  • The external environment: Who are the key actors? What is their agenda? How do they influence the political context?
  • The political context: Is there political interest in change? Is there room for manoeuvre? How do they perceive the problem?
  • The evidence: Is it there? Is it relevant? Is it practically useful? Are the concepts familiar or new? Does it need re-packaging?
  • Links: Who are the key individuals? Are there existing networks to use? How best to transfer the information? The media? Campaigns?
what researchers need to do
What researchers need to do
  • Work with them – seek commissions
  • Strategic opportunism – prepare for known events + resources for others
  • Get to know the policymakers.
  • Identify friends and foes.
  • Prepare for policy opportunities.
  • Look out for policy windows.
  • Who are the policymakers?
  • Is there demand for ideas?
  • What is the policy process?
  • Establish credibility
  • Provide practical solutions
  • Establish legitimacy.
  • Present clear options
  • Use familiar narratives.
  • Build a reputation
  • Action-research
  • Pilot projects to generate legitimacy
  • Good communication
  • What is the current theory?
  • What are the narratives?
  • How divergent is it?
  • Build partnerships.
  • Identify key networkers, mavens and salesmen.
  • Use informal contacts
  • Get to know the others
  • Work through existing networks.
  • Build coalitions.
  • Build new policy networks.
  • Who are the stakeholders?
  • What networks exist?
  • Who are the connectors, mavens and salesmen?
policy entrepreneurs
Policy entrepreneurs

Networkers

Storytellers

Fixers

Engineers

practical tools
Practical Tools

Overarching Tools

- The RAPID Framework

- Using the Framework

- The Entrepreneurship Questionnaire

Context Assessment Tools

- Stakeholder Analysis

- Forcefield Analysis

- Writeshops

- Policy Mapping

- Political Context Mapping

Communication Tools

- Communications Strategy

- SWOT analysis

- Message Design

- Making use of the media

Research Tools

- Case Studies

- Episode Studies

- Surveys

- Bibliometric Analysis

- Focus Group Discussion

Policy Influence Tools

- Influence Mapping & Power Mapping

- Lobbying and Advocacy

- Campaigning: A Simple Guide

- Competency self-assessment

practical application
Practical Application
  • Within ODI
  • Workshops for researchers, policy makers and activists.
  • Advice to a DFID forest/ground water research project in India:
    • Less research
    • More communication
    • Developing champions in regional and national government
    • Local, Regional & National advocacy campaign
further information resources
Further Information / Resources
  • ODI Working Papers
  • Bridging Research and Policy Book
  • Meeting series Monograph
  • Tools for Policy Impact
  • RAPID Briefing Paper
  • www.odi.org.uk/rapid
can ilri do it
Can ILRI do it?

Yes, but:

  • It this its role?
  • “Global Public Good” Research vs Policy Advocacy
  • Probably needs to do both:

How?

  • Understand the political context
  • Get the evidence & package it well
  • Strategic networking / lobbying / campaigning
  • Collaboration….
slide20

ILRI

International Livestock Research Institute

Process and partnership for pro-poor policy change

The New DfID funded Project

slide21

ILRI

International Livestock Research Institute

  • Project Leaders: ODI / ILRI
  • Key collaborators: ECAPAPA
  • Case study collaborators in Kenya:
    • MoLFD / KARI
    • Range of NGOs & other SDP partners
slide22

Why would I be interested?

ILRI

International Livestock Research Institute

  • Not all research is expected or intended to lead to policy change, but there may be;
    • Specific cases where research is expected to;
      • provide evidence for policy change
      • identify potential policies (or impact of)
      • influence the policy making process (advocacy)
    • Cases where speculative research becomes relevant because of changes in circumstance
slide23

The project …

ILRI

International Livestock Research Institute

  • Ideas for methods and approaches
  • Lessons learnt from earlier activities
  • Identification of appropriate communication tools
slide24

What will we be doing?

ILRI

International Livestock Research Institute

  • Three case studies in three DIFFERENT countries
    • A project considered to have influenced policy change
    • A stream of research addressing a particular policy area
    • A clear policy change;
      • New policy statement
      • New law
      • Irrefutable change in way something is done
slide25

What will we be doing?

ILRI

International Livestock Research Institute

  • Three case studies
    • SDP and impact on changed view of informal milk trade
    • ????
    • ????
slide26

ILRI

International Livestock Research Institute

  • Discussion:
    • Can ILRI hope to influence pro-poor policy through research?
    • Any good case studies?
what is a case study
What is a Case Study?

Definition:

" A systematic inquiry into an event or a set of related events which aims to describe and explain the phenomenon of interest" Bromley (1990)

why is it useful
Why is it useful?
  • An ideal methodology when a holistic, in-depth investigation is needed
  • Designed to bring out the details from the viewpoint of the participants by using multiple sources of data

Goal :

to describe as accurately as possible the fullest, most complete description of the case.

types of case study
Types of Case Study

Types of case studies:

  • Exploratory,
  • Explanatory,
  • Descriptive (Yin, 1993)

Stake (1995) included three others:

  • Intrinsic - when the researcher has an interest in the case;
  • Instrumental - when the case is used to understand more than what is obvious to the observer;
  • Collective - when a group of cases is studied.
issues
Issues
  • The unit of analysis is a critical factor
  • Typically a system of action rather than an individual or group of individuals
  • Tend to be selective, focusing on one or two issues that are fundamental to understanding the system being examined
  • Case studies are multi-perspectival analyses
  • The researcher considers not just the voice and perspective of the actors, but also of the relevant groups of actors and the interaction between them
  • They give a voice to the powerless and voiceless.
triangulation
Triangulation
  • Data source triangulation, when the researcher looks for the data to remain the same in different contexts;
  • Investigator triangulation, when several investigators examine the same phenomenon;
  • Theory triangulation, when investigators with different view points interpret the same results; and
  • Methodological triangulation, when one approach is followed by another, to increase confidence in the interpretation.
applications
Applications
  • To explain complex causal links between research and policy
  • To describe the real-life context in which policy has been influenced by research
  • To describe the policy influencing process itself
  • To explore those situations in which the policy intervention being evaluated has no clear set of outcomes.
process
Process
  • Design the case study protocol:
    • determine the required skills
    • develop and review the protocol
  • Conduct the case study:
    • prepare for data collection
    • distribute questionnaire
    • conduct interviews
  • Analyze case study evidence:
    • analytic strategy
  • Develop conclusions, recommendations, and implications based on the evidence
what is an episode study
What is an Episode Study

“A study that focuses on a clear policy change and tracks back to assess what impact research had among the variety of issues that led to the policy change”.

what is the purpose
What is the purpose?
  • an excellent way of investigating the influence of research on policy
  • Can focus on a single episode or comparative episodes.

Tracking backwards from policy change to any particular research which influence policy

advantage
Advantage
  • The process of working backwards in time gives a more realistic view of the broad range of factors – other than research – that influence policy
  • Tracking forward probably overemphasizes

the importance of research

issues1
Issues
  • Policy processes are complex, multi-layered and change over time
  • Often difficult to isolate the impact of research from other factors
  • Actors may ‘re-write history’
  • Important to seek the views of a wide range of informed stakeholders
  • The process of preparing an episode study is iterative
  • Key facts and / or inconsistencies need to be cross-checked with key informants
process1
Process
  • Identify a clear policy change.
  • Identify key Research Questions (draw on RAPID framework)
  • Explore how and why those policy decisions and practices took place
  • Assess the relative role of research in that process by drawing on the framework.
apply the rapid framework

External Influences

Campaigning, Lobbying

Policy analysis, & research

Scientific information exchange & validation

Apply the RAPID Framework

political context

Politics and Policymaking

Media,

Advocacy, Networking

Research, learning & thinking

evidence

links

key questions
Key Questions
  • The external environment: Who are the key actors? What is their agenda? How do they influence the political context?
  • The political context: Is there political interest in change? Is there room for manoeuvre? How do they perceive the problem?
  • The evidence: Is it there? Is it relevant? Is it practically useful? Are the concepts familiar or new? Does it need re-packaging?
  • Links: Who are the key individuals? Are there existing networks to use? How best to transfer the information? The media? Campaigns?
methods
Methods

• review of the literature;

• interviews with key actors;

• capturing the authors’ own experience; and

• discussions at workshops.

Steps 3 and 4 can be done through a variety of methods:

paravets in kenya

International Research

The Hubl Study

Paravets in Kenya
  • Professionalisation of Public Services.
  • Structural Adjustment → Collapse
  • Paravet projects emerge.
  • ITDG projects.
  • Privatisation.
  • ITDG Paravet network.
  • Rapid spread in North.
  • KVB letter (January 1998).
  • Multistakeholder WSs → new policies.
  • Still not approved / passed!
  • Professionalisationof Public Services.
  • Structural Adjustment
  • Privatisation
  • ITDG Paravet networkand change of DVS.
  • KVB letter(January 1998).
  • Multistakeholder WSs → new policies.

1970s

1980s

1990s

2000s

ITDG projects – collaborative research.

Dr Kajume

prsps political context
PRSPs – Political Context
  • Widespread awareness of a “problem” with international development policy in late 90s
  • Failure of SAPs (and Asian financial crisis)
  • Mounting public pressure for debt relief
  • Stagnation of Comprehensive Development Framework idea
  • Diverging agendas (UK – Poverty, US – Governance)
  • WB/IMF Annual General Meeting, Sept 1999
prsps evidence
PRSPs – Evidence
  • Long-term academic research informing new focus on poverty, participation, ownership, aid effectiveness etc
  • Applied policy research:
    • ESAF reviews
    • HIPC review
    • SPA Working Groups
    • NGO research on debt
  • Uganda’s PEAP
prsps links
PRSPs – Links
  • WB, IMF, SPA, Bilaterals, NGOs all involved
  • Formal an informal networks
  • “None of the players was more than two handshakes away from any of the others”
what is it
What is it?
  • an integrated PM&E tool
  • a system to think holistically & strategically about how we intend to achieve result
  • an approach that focuses on changes in the behaviour, relationships or actions of partners (as outcomes)
  • a methodology that characterizes and assesses the program’s contributions to the achievement of outcomes
  • an approach for designing in relation to the broader development context but assessing within your sphere of influence
how can it be used
How can it be used?
  • For a program to tell its performance story in outcome terms by:
    • articulating its goals and designing its activities
    • designing a monitoring system for assessing internal performance and outcomes of partners
    • setting a use-oriented evaluation plan
why use it
Why use it?
  • Focussing on changes in partners’ behaviour, relationships, or actions allows a program to:
    • measure results within its sphere of influence
    • obtain feedback about its efforts in order to improve its performance
    • take credit for its contributions to the achievement of outcomes
    • show progress towards outcomes
terminology
Terminology
  • Outcomes: changes in behaviours, relationships, activities and/or actions of the people, groups and organisations with whom we work
  • Vision: the broad human, social and environmental betterment we desire
  • Mission: how we intend to contribute towards the achievement of the vision
  • Boundary partners: individuals, groups and organisations with whom we interact directly to effect change
  • Outcome challenges: changes behaviours of the boundary partners as identified by the vision
performance monitoring
Performance Monitoring
  • Provides a framework for a continuous monitoring of the initiative as a tool to achieve its outcomes.
  • The program uses progress markers, a set of graduated indicators of behavioural change, identified in the intentional design stage to clarify direction with its primary partners and to monitor outcomes
evaluation planning
Evaluation Planning
  • Helps identify the evaluation priorities assessing the strategy at greater depth than the performance monitoring stage
what is it1
What is it?
  • A Visual Tool
  • Combines the outcome mapping concept within a case study & episode study approach
  • Systematic approach to collecting information about changes in behaviour of key project partners that contributed to the policy change
  • Assessment of the contribution of the project (programme, strategy, etc.) to observed changes in behaviour –and ultimately to the policy change
approach
Approach
  • Describe policy environment at end
  • Describe policy environment at the beginning
  • Identify the key policy actors
  • Identify key boundary partners
  • Describe boundary partner behaviour at end
  • Describe boundary partner behaviour at beginning
  • Describe changes in BP behaviour
  • Describe changes in project (strategic/opportunistic)
  • Describe external influences
  • Determine level of impact of changes in project
  • Determine level of impact of external influences
  • Check through external interviews
  • Write report
sources outputs
Sources & Outputs
  • Literature review

- Project background, progress, (published) achievements

  • Participatory workshop with staff (and BP)
    • Gather detailed information
    • Identify issues for further investigation
  • Interviews with key informants to:
    • Triangulate the result of the workshop,
    • Fill the gaps of information
    • Clarify causality
  • Report Writing
    • Visual and Narrative
roa terms and definitions
ROA Terms and Definitions
  • Boundarypartners: individuals, groups and organisations with whom we interact directly to effect change.
  • Outcomes: changes in behaviours, relationships, activities and/or actions of the people, groups and organisations with whom we work.
  • Behaviours: the way we or our boundary partners do or think about things.
the key steps of the roa framework
The key steps of the ROA framework
  • Describe the policy environment at the end of the project
the key steps of the roa framework1
The key steps of the ROA framework

2. Describe the policy environment at the beginning of the project

the key steps of the roa framework2
The key steps of the ROA framework

3/4. Identify key policy actors and boundary partners (that were influential at end)

the key steps of the roa framework3
The key steps of the ROA framework

5. Describe the behaviours of the boundary partners that contributed to the change in the policy environment or policy

the key steps of the roa framework4
The key steps of the ROA framework
  • Describe the behaviours of the boundary partners at the beginning of the project
the key steps of the roa framework5
The key steps of the ROA framework

7. Map the key changes in behaviour for each boundary partner from the start of the project

the key steps of the roa framework6
The key steps of the ROA framework

8. Map the key changes in the project including organisational changes, outputs and changes in behaviour during the same period.

the key steps of the roa framework7
The key steps of the ROA framework
  • Map the external influences including the actions f strategic partners and other exogenous factors during the same period
the key steps of the roa framework8
The key steps of the ROA framework

10. Determine the level of impact/influence of the project on the changes in behaviour of the boundary partners

the key steps of the roa framework9
The key steps of the ROA framework

11. Determine the level of impact/influence of external influences on the changes in behaviour of the boundary partners and the project

the key steps of the roa framework10
The key steps of the ROA framework

12. Refine conclusions with in-depth interviews and assess the real contribution of the project on the policy change