Stephen Powell, Joakim Molander  & Ivona Čelebičić
Download
1 / 24

EASY-ECO Vienna 2008 - PowerPoint PPT Presentation


  • 110 Views
  • Uploaded on

Stephen Powell, Joakim Molander & Ivona Čelebičić. Assessment of Outcome Mapping as a tool for evaluating and monitoring support to civil society organisations. EASY-ECO Vienna 2008. proMENTE social research, Sarajevo. Who are we.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'EASY-ECO Vienna 2008' - darci


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Easy eco vienna 2008

Stephen Powell, Joakim Molander & Ivona Čelebičić

Assessment of Outcome Mapping as a tool for evaluating and monitoring support to civil society organisations

EASY-ECO Vienna 2008

proMENTE social research, Sarajevo


Who are we
Who are we

  • Steve Powell & Ivona ČelebičićproMENTE social research, Sarajevo

  • Joakim Molander: at the time of the evaluation, First Secretary at the Swedish Embassy, Sarajevo

  • Report: www.promente.org/sida2eng


Background
Background

  • Govt. in B&H barely functional

  • Civil society fills the gap?

  •  Support people and networks

  • How to evaluate?

  • OM


Evaluation tor
Evaluation TOR

  • Explorative evaluation of six civil society projects

  • Sida-funded civil society programming: lessons on sustainability

  • Exploration of OM as a tool


Messages
Messages

  • OM worked well for civil society evaluation

  • OM plays well with other approaches

  • Sustainability ↔ focus on “key players”

  • OM can be part of a patchwork, rather than monolithic, approach to M&E


Our project in b h 1 donor 3 framework partners 6 implementing partners 6 projects 2007
Our project in B&H: 1 donor, 3 framework partners, 6 implementing partners, 6 projects: 2007


Om 12 steps
OM: 12 steps implementing partners, 6 projects: 2007

Focus on outcomes

Considering all the dimensions of strategy

Helping the implementing partner to learn

Internal M &E


Method timeframe
Method: timeframe implementing partners, 6 projects: 2007

Baseline

Final assessment


Method om
Method: OM implementing partners, 6 projects: 2007

  • Special use of OM for external evaluation

    • Mission, vision, progress markers, outcome challenge

    • Assessment of planning strategies using strategy maps

    • OM questionnaires

    • OM interview with boundary partners (+ relevance interviews)

    • OM interview with implementing partners (+ relevance interviews)

    • Writing evaluation-start Outcome Journals

    • Confirming evaluation-start Outcome Journals

    • OM questionnaires (same as evaluation-start)

    • OM interview with boundary partners (on the basis of evaluation-start interview) (+ relevance interviews)

    • OM interview with implementing partners (on the basis of evaluation-start interview) (+ relevance interviews)

    • Assessment of management progress using existing strategy maps

    • Mission and vision: still relevant? Changed focus?

    • Gathering contextual information and intervention timeline

Baseline interimfinal assessment


Message 1 om worked well with civil society evaluation
Message 1: OM worked well with civil society evaluation implementing partners, 6 projects: 2007

  • Most organisations succeeded in redefining changes in a small group of boundary partners as the main component of their vision

  • NGOs enthusiastic!?

  • Focus on contribution is a big relief

  • Gives richer feedback

  • Strategy maps inspire and organise thinking about different dimensions of planning

  • OM standard method and questionnaires, (and interviews?) showed projects making progress towards their vision

  • Consider problem-based rather than vision-based programming. Vision is not always about boundary partner change


Message 2 om plays well with other approaches
Message 2: OM plays well with other approaches implementing partners, 6 projects: 2007


2a qualitative methods
2a: Qualitative methods implementing partners, 6 projects: 2007

  • Gained a lot of additional information with systematic content analysis of independent “relevance interviews”

    • “A good part of the population is used to some sort of humanitarian aid, some sort of social help, mercy, call it what you will. This means that nobody has to work and, at the same time, they get something. We can set our sights lower but we don’t need to invest anything.”


2b quantitative analysis
2b: quantitative analysis implementing partners, 6 projects: 2007

  • For four projects, a customised questionnaire was made on the basis of progress markers.

  • Questionnaire filled in at baseline and again at final assessment

  • Results from around 100 baseline and 100 final assessment questionnaires compared on a per-project basis.

  • Validation of OM evaluation methods?

    • Less clear progress with the least educated


2b quantitative analysis1
2b: quantitative analysis implementing partners, 6 projects: 2007

male female

  • Small but significant overall improvement

  • Corresponds to information from OM interviews & journals

  • Women start lower than men and improve more than men

  • In spite of a possible tendency to “raise the bar” during the project “soft” interpretation

Results for one project: average scores on progress marker questionnaires

 at baseline

 at evaluation end


2c implementing partners and boundary partners do not agree on difficulty of progress markers
2c: implementing partners and boundary partners do not agree on “difficulty” of progress markers

 involve boundary partners in planning!! – especially the poor

Average baseline score for one project

Love to see / like to see / expect to see


2d problems with om rbm
2d: problems with on “difficulty” of progress markersOM + RBM

  • Avoiding a double workload:

  • OM for monitoring OM for planning and reporting

  • Better planning: OM could/should be adopted by donors from the project application stage (or as a hybrid)

  • Donors have to want to help organisations to learn, at the cost of demonstrating effectiveness


Message 3 sustainability focus on real people
Message 3: Sustainability ↔ focus on on “difficulty” of progress markersreal people!


3a differentiate your boundary partners
3a: differentiate your boundary partners on “difficulty” of progress markers

  • Especially in civil society, boundary partner groups can be usefully divided into "less powerful", "potentially powerful" and "powerful".

  • Even within one group, distinguish clearly between partners who are at different levels of development towards, or agreement with,the outcome challenge.


3b focus on key players
3b: focus on on “difficulty” of progress markers“key players”

  • = boundary partners or implementing partners??

  • They believe that change is possible through the efforts of individuals.

  • Civil society not a mass movement! Handful of key (“career”) activists and organisations

  • They have “first name and family name” and are not interchangeable or easily replaceable; yet invisible on paper


3c a taboo motivation
3c: A taboo: motivation on “difficulty” of progress markers

  • Why would these people want to play along? Money a taboo? If we aren’t using sticks, do we have juicy carrots?

  • Careers advice for activists? Business support for NGOs?


3d another taboo personality
3d: Another taboo: personality on “difficulty” of progress markers

  • Effective activists are not necessarily effective bureaucrats (Easterly). Should they be?

  • Personality is hard to change Differential skills are needed (talking to parliament, then villagers)


Message 4 om can be part of a patchwork rather than monolithic approach to m e
Message 4: OM can be part of a on “difficulty” of progress markerspatchwork, rather than monolithic, approach to M&E


4a the monolithic approach
4a: the monolithic approach on “difficulty” of progress markers

Program

Project A

Project B

Impacts … outcomes … outputs …

  • Control: gathering data via a chain in which every link has a vested interest in lying

  • Monitoring: ticking boxes at the expense of strategic thinking and organisational learning

  • Evaluation: Can you calculate impacts by aggregating outputs? Are NGOs competent to measure outcome and impact directly? Is it their job?

Causation

Data aggregation

  • RBM: killing the birds of project and program planning, control, monitoring and evaluation with one stone?


4b om is only part of a patchwork solution
4b: OM is only part of a (patchwork) solution on “difficulty” of progress markers

  • Better control: release NGOs from exhausting box-ticking and implement fair random checking of outputs.

  • Better planning and monitoring: use OM where appropriate

  • Better management: in exchange, require elements of strategic planning and organisational learning.

  • Better evaluation:

    • Demonstrate donor-relevant impact: commission independent, external investigations of society change and why it happens/happened

    • Sharpen the focus: help NGOs to do empirical explorations of a selection of specific issues that really interest them

  • OM isn’t everything