Monitoring and evaluating capacity-building at multiple levels
This presentation is the property of its rightful owner.
Sponsored Links
1 / 26

Monitoring and evaluating capacity-building at multiple levels PowerPoint PPT Presentation


  • 83 Views
  • Uploaded on
  • Presentation posted in: General

Monitoring and evaluating capacity-building at multiple levels. UNFCCC Meeting on monitoring and evaluating CB, Rio, Brazil 6-7 November 2008 David Watson Consultant. Scope of Presentation. Background: perspectives and approach The story so far: M&E of CB under UNFCCC

Download Presentation

Monitoring and evaluating capacity-building at multiple levels

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Monitoring and evaluating capacity building at multiple levels

Monitoring and evaluating capacity-building at multiple levels

UNFCCC Meeting on monitoring and evaluating CB, Rio, Brazil

6-7 November 2008

David Watson

Consultant


Scope of presentation

Scope of Presentation

  • Background: perspectives and approach

  • The story so far: M&E of CB under UNFCCC

  • How other global programmes tackle M&E of CB

  • The big picture: M&E of CB in international development

  • Alternative paradigms: esp. systems thinking

  • Towards a tool-kit for practitioners


Background

Background

  • Sorry about late distribution of paper! (See Section X for a potted summary!)

  • Personal ‘points of departure’: building institutions and governance

  • A touch of scepticism about M&E practices; pragmatism; incrementalism; practicality; process

  • Broadening debate constructively

  • Not ‘selling’ any particular approaches

  • Cognisant of the seriousness of the challenges of slowing climate change, and of enhancing the effectiveness of CB to that end


Levels of capacity and cb

Levels of Capacity and CB

  • May be helpful to bear in mind two complementary dimensions:

  • ‘Horizontal’: from individuals, through organisations, to social / national systems or networks

  • ‘Vertical’ : from local, to regional, to national, to global.


Other global programmes experience with m e of cb

Other Global Programmes’ Experience with M&E of CB

  • PRSPs

  • GAVI / Roll Back Malaria

  • UNAIDS

  • Programme Based Approaches (PBAs)

  • WFP

  • Lessons pertinent to CB to address Climate Change


Lessons from other global progs

Lessons from other global progs.

  • Several (GAVI and RBM) had specialised groups focussed on M&E, plus Task Forces on specialised subjects (e.g. CB)

  • At country level: a Secretariat may have a M&E Cell; use existing M&E structures where they exist; keep it simple; minimise number of indicators.

  • Empirical evidence of enhanced capacities is often scarce

  • ‘Strategic incrementalism’ attracted PBA-ers attention: pursue ‘quick wins’ and ‘peer-learning’


Lessons continued

..lessons continued…

  • Clarify objectives of CB, based on thorough needs assessments, as a basis for a simple results framework

  • Try to avoid a ‘missing middle’: indicators for the steps between outputs and outcomes;

  • Reflect on whether the M&E process should feed more into national political processes, (engaging e.g. with parliaments, media, civil society) AND be a precondition of partnerships nationally and with the international community.

  • Don’t be naïve in your ‘theory of political change’: …’participation’ is not enough.


A poignant quotation

A poignant quotation

‘M&E practices do not provide a framework for re-presenting (or making meaning from) the complex and multiple processes of institutional and individual learning and change that are sought in strengthening institutional capacity’

Source: Unitar ‘Challenges and Constraints in M&E of Capacity Building’ presentation to first (Antigua) workshop Nov 2007


M e of cb recent insights

M&E of CB : recent insights

  • Theme paper for ECDPM Study on Capacity Change and Performance 2003-8 (Baser Morgan + Watson on Theme Paper)

  • On M&E of CD tended to agree with UNITAR!

  • Often, ‘performance improvement’ indicators are used as proxy for capacity increases

  • …not many e.g.s of ‘capacity’ being monitored

  • Accountability (to donors) the main ‘driver’

  • Most public sector CB experience disappointing…political and institutional factors seem important

  • Donors have a poor record on M&E of CB: ..including their own capacities and incentives


Capacitated organisations

‘Capacitated’ organisations:

1. ..carry out tasks effectively

2. ..engage, decide, act

3. ..relate, attract resources & support

4. ..adapt and self-renew

5. ..balance diversity, and build coherence across the organisational system

Capacity: ‘that emergent combination of individual competences and collective capabilities which enables a human system to create public value


And implications for m e of cd

..and implications for M&E of CD..?

  • Agree on the nature of capacity to be monitored!

  • ..need to focus on more complex issues…

  • Information on change and progress should not be sent ‘up’ …but shared internally for purpose of learning

  • Pay more attention to the context…and inter-relationships in capacity processes..

  • to the changes taking place (intended or not) and their contributions to ‘capacity’…

  • ..i.e. need more participative approaches to M&E… and to learning and reflection..


Alternative paradigms to m e of cb

Alternative Paradigms to M&E (of CB)

  • Critical reactions to ‘monitoring abilities to perform…and meeting pre-determined objectives’

  • Wheatley: ‘measurement’ = anathema (when looking at human behaviour)

  • ‘Cause and effect’ logic inapplicable

  • ..change processes are complex..involve inter-dependencies…and are not ‘linear’ or stable, nor necessarily visible

  • ..but may well be long-term, and more political than technical


Systems thinking s relevance

Systems Thinking’s relevance

  • Growth in interest in ST as an analytical framework for development and natural resource management

  • Explicitly examined as part of ECDPM study: helps address implications mentioned

  • Human institutions seen as complex adaptive systems: e.g. climate!

  • See Box 1 for a summary of basic concepts

  • ‘Mess’ = problems unbounded in scope, time and resources; no clear agreement on optimal solution or how to achieve it; goals and strategies are contested; perspectives differ; no clear cause and effect; uncertainty prevails.


What about capacity

What about ‘Capacity’?

  • ‘Capacity is an emergent property of human systems’ (ECDPM Study)

  • (Emergent properties are those which have no meaning in terms of parts making up the whole system. They are the outcome of system behaviour or synergy.)

  • Two examples: ENACT Jamaica (Box 2)

  • And IUCN Asia (Box 3)


Enact iucn

Formal performance monitoring system abandoned

Empowerment of frontline staff for rapid response

Absence of a ‘model’ to assess performance

Let partners adapt and adopt measures

Donor modified its approach..more ‘learning-friendly’ faced with diversity

Unusually diverse membership

Flexibility demonstrated by funding agencies

Permitted experimentation, innovation and creativity

Evolution of IUCN learning processes

‘Teaming’ process

Ownership by governments + IUCN credibility / legitimacy

CB= continuous process ‘no road map, only a goal’

ENACTIUCN


Cases commonalities c f st

Cases’ commonalities c.f. ST

  • Identification and recognition of goals

  • Emphasis on values to be reflected

  • Clarity and awareness of mission amongst clients too

  • Leadership encourages experimentation

  • Opportunities for learning from experience: self-assessment and ‘stories’ of positive experience or changes or errors

  • Flexibility to adapt (thru new skills-building) to new needs / priorities, thru OJT ‘hands-on’

  • Informality of M&E systems: responsive to needs of clients / network members

  • Ability to learn from experience is crucial


But reductionist approaches are still relevant where

BUT ‘Reductionist’ approaches are still relevant…where:

  • It is possible to define required capabilities unambiguously and specifically

  • ..and to assess existing ones = ‘gap’

  • Therefore easy to define indicators

  • Where stakeholders able and willing to define their shortfalls and ‘sign up’

  • Incentives exist to improve performance

  • Leadership, and all above combine into ‘ownership’

  • BUT ..this combination of circumstances is rare! (e.g. public financial management IMF and WB forged consensus)


Towards a tool box for m e of cd in climate change

Towards a ‘Tool Box’ for M&E of CD in Climate Change

  • Principles: pragmatism: acknowledge weaknesses in all M&E systems: main aim = ‘sense-making’

  • Seek out ‘what M&E exists, and what works, already’

  • Acknowledge that the best M&E systems are ‘customised’: adapted by participants, based on local conditions


Some existing frameworks

Some existing frameworks

  • GEF Resource Kit on M&E

  • GEF Indicators for 5 key capacities:

    • Engagement; generate access to and use of information; policy and legislation development; management and implementation; monitor and evaluate

    • Scoring / indicator system for each;

    • Obliges actions / next steps / link to outcomes

  • UNEP ‘Lessons Trees’

    • Trying to improve quality of learning (especially about common problems) and application of learning to future programmes


Self assessment

Self-Assessment

  • Examples from CB in research and development organisations

  • SA workshops after applying qualitative and quantitative tools

  • Managers, staff and stakeholders identify strengths and weaknesses, and set new directions ….Advantages =

    (1) those with knowledge of and interest in the organisation gain in-depth insights: what’s working and why..where improvements needed

    (2) Well-prepared to address the changes needed

  • E.g. M&E of past CB in Mekong Farming R&D Systems Institute (Box 4)

    • Preparation of ‘work stories’ on past CB efforts

    • + interviews with key staff on changes and challenges


A balanced approach to m e of cb

A ‘Balanced Approach’ to M&E of CB

  • Framework generated to address M&E of CB a la ECDPM Study dimensions of ‘Capacity’

  • Piloted in PNG legal and judicial reform

  • Time-consuming and exhaustive: but national practitioners did find it helpful in conceptualising all dimensions of capacities being built up.

  • See extra handout


Appreciative enquiry in formative evaluation

Appreciative Enquiry in Formative Evaluation

  • Evaluators form more of an understanding of the political, cultural and historical ‘landscape’

  • Encouraging organisations to develop their relationships with primary stakeholders

  • Community Development Corporations in US

    • Seeking out ‘what enabled effectiveness, and what hindered it’

    • Evaluator regularly listening to ‘stories’

    • Regular visits and sharing of reports

    • Recipients able to ‘co-create’ the initiative and develop OWN capacity for assessment

    • Evaluator chosen by NGO not donor: NGO employed not as expert but for its ability to learn collectively


Most significant change msc

Most Significant Change (MSC)

  • First applied in evaluation of a complex RD programme in Bangladesh

    • Process managers identify domains of change which are important to evaluate

    • Stories (descriptions of changes deemed significant: with reasons: why significant?) periodically collected from stakeholders

    • Analysed and filtered up thru committees

    • Criteria for choosing stories are collated and fed back to stakeholders;

    • Final selection made (annually?) with reasons

    • Circulated to all

    • Site visits to check; deepen understanding of changes


Msc features

MSC features

  • Focuses attention and direction of work in programme towards valued directions

  • Dialogue and deliberations crucial

  • Takes place over time: responsive to changing contexts

  • Policy makers, funders, field engaged looking at the value of changes

  • Stories help all relate to information

  • Non-experts (the story-tellers) involved in evaluation

  • Dialogue based on real experience and concrete outcomes not abstract indicators

  • …MSC positively evaluated as technique in Laos (Willetts 2004)


Annotated bibliography

Annotated Bibliography

  • ..use as a part of the ‘tool-kit’!

  • Sections on:

    • Climate Change-related sources

    • Broader CB literature and M&E

    • Other Global Programmes’ Materials

    • Civil Society CB and M&E

    • Systems Thinking and Complexity literature and M&E examples


Summary why st has potential for m e of cb in climate change

Summary: why ST has potential for M&E of CB in Climate Change

  • Avoids pitfalls of logical framework in what is an amorphous field: CB

  • Emphasises clarity of objective-setting, AND of learning collectively from reality

  • Generates, relates to, and values ‘stories’

  • Potential to enhance team-work and inter-relationships of hitherto disparate institutions and groups in their ‘ecosystem’ context;

  • Climate change context – politically-charged, formal and informal, amenable to negotiation

  • Evidence indicates ST M&E approaches ‘can work for and strengthen social change’ (Guijt 2007 IDS)


  • Login