Monitoring and evaluating capacity-building at multiple levels
Sponsored Links
This presentation is the property of its rightful owner.
1 / 26

Monitoring and evaluating capacity-building at multiple levels PowerPoint PPT Presentation


  • 92 Views
  • Uploaded on
  • Presentation posted in: General

Monitoring and evaluating capacity-building at multiple levels. UNFCCC Meeting on monitoring and evaluating CB, Rio, Brazil 6-7 November 2008 David Watson Consultant. Scope of Presentation. Background: perspectives and approach The story so far: M&E of CB under UNFCCC

Download Presentation

Monitoring and evaluating capacity-building at multiple levels

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Monitoring and evaluating capacity-building at multiple levels

UNFCCC Meeting on monitoring and evaluating CB, Rio, Brazil

6-7 November 2008

David Watson

Consultant


Scope of Presentation

  • Background: perspectives and approach

  • The story so far: M&E of CB under UNFCCC

  • How other global programmes tackle M&E of CB

  • The big picture: M&E of CB in international development

  • Alternative paradigms: esp. systems thinking

  • Towards a tool-kit for practitioners


Background

  • Sorry about late distribution of paper! (See Section X for a potted summary!)

  • Personal ‘points of departure’: building institutions and governance

  • A touch of scepticism about M&E practices; pragmatism; incrementalism; practicality; process

  • Broadening debate constructively

  • Not ‘selling’ any particular approaches

  • Cognisant of the seriousness of the challenges of slowing climate change, and of enhancing the effectiveness of CB to that end


Levels of Capacity and CB

  • May be helpful to bear in mind two complementary dimensions:

  • ‘Horizontal’: from individuals, through organisations, to social / national systems or networks

  • ‘Vertical’ : from local, to regional, to national, to global.


Other Global Programmes’ Experience with M&E of CB

  • PRSPs

  • GAVI / Roll Back Malaria

  • UNAIDS

  • Programme Based Approaches (PBAs)

  • WFP

  • Lessons pertinent to CB to address Climate Change


Lessons from other global progs.

  • Several (GAVI and RBM) had specialised groups focussed on M&E, plus Task Forces on specialised subjects (e.g. CB)

  • At country level: a Secretariat may have a M&E Cell; use existing M&E structures where they exist; keep it simple; minimise number of indicators.

  • Empirical evidence of enhanced capacities is often scarce

  • ‘Strategic incrementalism’ attracted PBA-ers attention: pursue ‘quick wins’ and ‘peer-learning’


..lessons continued…

  • Clarify objectives of CB, based on thorough needs assessments, as a basis for a simple results framework

  • Try to avoid a ‘missing middle’: indicators for the steps between outputs and outcomes;

  • Reflect on whether the M&E process should feed more into national political processes, (engaging e.g. with parliaments, media, civil society) AND be a precondition of partnerships nationally and with the international community.

  • Don’t be naïve in your ‘theory of political change’: …’participation’ is not enough.


A poignant quotation

‘M&E practices do not provide a framework for re-presenting (or making meaning from) the complex and multiple processes of institutional and individual learning and change that are sought in strengthening institutional capacity’

Source: Unitar ‘Challenges and Constraints in M&E of Capacity Building’ presentation to first (Antigua) workshop Nov 2007


M&E of CB : recent insights

  • Theme paper for ECDPM Study on Capacity Change and Performance 2003-8 (Baser Morgan + Watson on Theme Paper)

  • On M&E of CD tended to agree with UNITAR!

  • Often, ‘performance improvement’ indicators are used as proxy for capacity increases

  • …not many e.g.s of ‘capacity’ being monitored

  • Accountability (to donors) the main ‘driver’

  • Most public sector CB experience disappointing…political and institutional factors seem important

  • Donors have a poor record on M&E of CB: ..including their own capacities and incentives


‘Capacitated’ organisations:

1. ..carry out tasks effectively

2. ..engage, decide, act

3. ..relate, attract resources & support

4. ..adapt and self-renew

5. ..balance diversity, and build coherence across the organisational system

Capacity: ‘that emergent combination of individual competences and collective capabilities which enables a human system to create public value


..and implications for M&E of CD..?

  • Agree on the nature of capacity to be monitored!

  • ..need to focus on more complex issues…

  • Information on change and progress should not be sent ‘up’ …but shared internally for purpose of learning

  • Pay more attention to the context…and inter-relationships in capacity processes..

  • to the changes taking place (intended or not) and their contributions to ‘capacity’…

  • ..i.e. need more participative approaches to M&E… and to learning and reflection..


Alternative Paradigms to M&E (of CB)

  • Critical reactions to ‘monitoring abilities to perform…and meeting pre-determined objectives’

  • Wheatley: ‘measurement’ = anathema (when looking at human behaviour)

  • ‘Cause and effect’ logic inapplicable

  • ..change processes are complex..involve inter-dependencies…and are not ‘linear’ or stable, nor necessarily visible

  • ..but may well be long-term, and more political than technical


Systems Thinking’s relevance

  • Growth in interest in ST as an analytical framework for development and natural resource management

  • Explicitly examined as part of ECDPM study: helps address implications mentioned

  • Human institutions seen as complex adaptive systems: e.g. climate!

  • See Box 1 for a summary of basic concepts

  • ‘Mess’ = problems unbounded in scope, time and resources; no clear agreement on optimal solution or how to achieve it; goals and strategies are contested; perspectives differ; no clear cause and effect; uncertainty prevails.


What about ‘Capacity’?

  • ‘Capacity is an emergent property of human systems’ (ECDPM Study)

  • (Emergent properties are those which have no meaning in terms of parts making up the whole system. They are the outcome of system behaviour or synergy.)

  • Two examples: ENACT Jamaica (Box 2)

  • And IUCN Asia (Box 3)


Formal performance monitoring system abandoned

Empowerment of frontline staff for rapid response

Absence of a ‘model’ to assess performance

Let partners adapt and adopt measures

Donor modified its approach..more ‘learning-friendly’ faced with diversity

Unusually diverse membership

Flexibility demonstrated by funding agencies

Permitted experimentation, innovation and creativity

Evolution of IUCN learning processes

‘Teaming’ process

Ownership by governments + IUCN credibility / legitimacy

CB= continuous process ‘no road map, only a goal’

ENACTIUCN


Cases’ commonalities c.f. ST

  • Identification and recognition of goals

  • Emphasis on values to be reflected

  • Clarity and awareness of mission amongst clients too

  • Leadership encourages experimentation

  • Opportunities for learning from experience: self-assessment and ‘stories’ of positive experience or changes or errors

  • Flexibility to adapt (thru new skills-building) to new needs / priorities, thru OJT ‘hands-on’

  • Informality of M&E systems: responsive to needs of clients / network members

  • Ability to learn from experience is crucial


BUT ‘Reductionist’ approaches are still relevant…where:

  • It is possible to define required capabilities unambiguously and specifically

  • ..and to assess existing ones = ‘gap’

  • Therefore easy to define indicators

  • Where stakeholders able and willing to define their shortfalls and ‘sign up’

  • Incentives exist to improve performance

  • Leadership, and all above combine into ‘ownership’

  • BUT ..this combination of circumstances is rare! (e.g. public financial management IMF and WB forged consensus)


Towards a ‘Tool Box’ for M&E of CD in Climate Change

  • Principles: pragmatism: acknowledge weaknesses in all M&E systems: main aim = ‘sense-making’

  • Seek out ‘what M&E exists, and what works, already’

  • Acknowledge that the best M&E systems are ‘customised’: adapted by participants, based on local conditions


Some existing frameworks

  • GEF Resource Kit on M&E

  • GEF Indicators for 5 key capacities:

    • Engagement; generate access to and use of information; policy and legislation development; management and implementation; monitor and evaluate

    • Scoring / indicator system for each;

    • Obliges actions / next steps / link to outcomes

  • UNEP ‘Lessons Trees’

    • Trying to improve quality of learning (especially about common problems) and application of learning to future programmes


Self-Assessment

  • Examples from CB in research and development organisations

  • SA workshops after applying qualitative and quantitative tools

  • Managers, staff and stakeholders identify strengths and weaknesses, and set new directions ….Advantages =

    (1) those with knowledge of and interest in the organisation gain in-depth insights: what’s working and why..where improvements needed

    (2) Well-prepared to address the changes needed

  • E.g. M&E of past CB in Mekong Farming R&D Systems Institute (Box 4)

    • Preparation of ‘work stories’ on past CB efforts

    • + interviews with key staff on changes and challenges


A ‘Balanced Approach’ to M&E of CB

  • Framework generated to address M&E of CB a la ECDPM Study dimensions of ‘Capacity’

  • Piloted in PNG legal and judicial reform

  • Time-consuming and exhaustive: but national practitioners did find it helpful in conceptualising all dimensions of capacities being built up.

  • See extra handout


Appreciative Enquiry in Formative Evaluation

  • Evaluators form more of an understanding of the political, cultural and historical ‘landscape’

  • Encouraging organisations to develop their relationships with primary stakeholders

  • Community Development Corporations in US

    • Seeking out ‘what enabled effectiveness, and what hindered it’

    • Evaluator regularly listening to ‘stories’

    • Regular visits and sharing of reports

    • Recipients able to ‘co-create’ the initiative and develop OWN capacity for assessment

    • Evaluator chosen by NGO not donor: NGO employed not as expert but for its ability to learn collectively


Most Significant Change (MSC)

  • First applied in evaluation of a complex RD programme in Bangladesh

    • Process managers identify domains of change which are important to evaluate

    • Stories (descriptions of changes deemed significant: with reasons: why significant?) periodically collected from stakeholders

    • Analysed and filtered up thru committees

    • Criteria for choosing stories are collated and fed back to stakeholders;

    • Final selection made (annually?) with reasons

    • Circulated to all

    • Site visits to check; deepen understanding of changes


MSC features

  • Focuses attention and direction of work in programme towards valued directions

  • Dialogue and deliberations crucial

  • Takes place over time: responsive to changing contexts

  • Policy makers, funders, field engaged looking at the value of changes

  • Stories help all relate to information

  • Non-experts (the story-tellers) involved in evaluation

  • Dialogue based on real experience and concrete outcomes not abstract indicators

  • …MSC positively evaluated as technique in Laos (Willetts 2004)


Annotated Bibliography

  • ..use as a part of the ‘tool-kit’!

  • Sections on:

    • Climate Change-related sources

    • Broader CB literature and M&E

    • Other Global Programmes’ Materials

    • Civil Society CB and M&E

    • Systems Thinking and Complexity literature and M&E examples


Summary: why ST has potential for M&E of CB in Climate Change

  • Avoids pitfalls of logical framework in what is an amorphous field: CB

  • Emphasises clarity of objective-setting, AND of learning collectively from reality

  • Generates, relates to, and values ‘stories’

  • Potential to enhance team-work and inter-relationships of hitherto disparate institutions and groups in their ‘ecosystem’ context;

  • Climate change context – politically-charged, formal and informal, amenable to negotiation

  • Evidence indicates ST M&E approaches ‘can work for and strengthen social change’ (Guijt 2007 IDS)


  • Login