Monitoring and evaluating capacity-building at multiple levels

1. Monitoring and evaluating capacity-building at multiple levels UNFCCC Meeting on monitoring and evaluating CB, Rio, Brazil 6-7 November 2008 David Watson Consultant

2. Scope of Presentation • Background: perspectives and approach • The story so far: M&E of CB under UNFCCC • How other global programmes tackle M&E of CB • The big picture: M&E of CB in international development • Alternative paradigms: esp. systems thinking • Towards a tool-kit for practitioners

3. Background • Sorry about late distribution of paper! (See Section X for a potted summary!) • Personal ‘points of departure’: building institutions and governance • A touch of scepticism about M&E practices; pragmatism; incrementalism; practicality; process • Broadening debate constructively • Not ‘selling’ any particular approaches • Cognisant of the seriousness of the challenges of slowing climate change, and of enhancing the effectiveness of CB to that end

4. Levels of Capacity and CB • May be helpful to bear in mind two complementary dimensions: • ‘Horizontal’: from individuals, through organisations, to social / national systems or networks • ‘Vertical’ : from local, to regional, to national, to global.

5. Other Global Programmes’ Experience with M&E of CB • PRSPs • GAVI / Roll Back Malaria • UNAIDS • Programme Based Approaches (PBAs) • WFP • Lessons pertinent to CB to address Climate Change

6. Lessons from other global progs. • Several (GAVI and RBM) had specialised groups focussed on M&E, plus Task Forces on specialised subjects (e.g. CB) • At country level: a Secretariat may have a M&E Cell; use existing M&E structures where they exist; keep it simple; minimise number of indicators. • Empirical evidence of enhanced capacities is often scarce • ‘Strategic incrementalism’ attracted PBA-ers attention: pursue ‘quick wins’ and ‘peer-learning’

7. ..lessons continued… • Clarify objectives of CB, based on thorough needs assessments, as a basis for a simple results framework • Try to avoid a ‘missing middle’: indicators for the steps between outputs and outcomes; • Reflect on whether the M&E process should feed more into national political processes, (engaging e.g. with parliaments, media, civil society) AND be a precondition of partnerships nationally and with the international community. • Don’t be naïve in your ‘theory of political change’: …’participation’ is not enough.

8. A poignant quotation ‘M&E practices do not provide a framework for re-presenting (or making meaning from) the complex and multiple processes of institutional and individual learning and change that are sought in strengthening institutional capacity’ Source: Unitar ‘Challenges and Constraints in M&E of Capacity Building’ presentation to first (Antigua) workshop Nov 2007

9. M&E of CB : recent insights • Theme paper for ECDPM Study on Capacity Change and Performance 2003-8 (Baser Morgan + Watson on Theme Paper) • On M&E of CD tended to agree with UNITAR! • Often, ‘performance improvement’ indicators are used as proxy for capacity increases • …not many e.g.s of ‘capacity’ being monitored • Accountability (to donors) the main ‘driver’ • Most public sector CB experience disappointing…political and institutional factors seem important • Donors have a poor record on M&E of CB: ..including their own capacities and incentives

10. ‘Capacitated’ organisations: 1. ..carry out tasks effectively 2. ..engage, decide, act 3. ..relate, attract resources & support 4. ..adapt and self-renew 5. ..balance diversity, and build coherence across the organisational system Capacity: ‘that emergent combination of individual competences and collective capabilities which enables a human system to create public value

11. ..and implications for M&E of CD..? • Agree on the nature of capacity to be monitored! • ..need to focus on more complex issues… • Information on change and progress should not be sent ‘up’ …but shared internally for purpose of learning • Pay more attention to the context…and inter-relationships in capacity processes.. • to the changes taking place (intended or not) and their contributions to ‘capacity’… • ..i.e. need more participative approaches to M&E… and to learning and reflection..

12. Alternative Paradigms to M&E (of CB) • Critical reactions to ‘monitoring abilities to perform…and meeting pre-determined objectives’ • Wheatley: ‘measurement’ = anathema (when looking at human behaviour) • ‘Cause and effect’ logic inapplicable • ..change processes are complex..involve inter-dependencies…and are not ‘linear’ or stable, nor necessarily visible • ..but may well be long-term, and more political than technical

13. Systems Thinking’s relevance • Growth in interest in ST as an analytical framework for development and natural resource management • Explicitly examined as part of ECDPM study: helps address implications mentioned • Human institutions seen as complex adaptive systems: e.g. climate! • See Box 1 for a summary of basic concepts • ‘Mess’ = problems unbounded in scope, time and resources; no clear agreement on optimal solution or how to achieve it; goals and strategies are contested; perspectives differ; no clear cause and effect; uncertainty prevails.

14. What about ‘Capacity’? • ‘Capacity is an emergent property of human systems’ (ECDPM Study) • (Emergent properties are those which have no meaning in terms of parts making up the whole system. They are the outcome of system behaviour or synergy.) • Two examples: ENACT Jamaica (Box 2) • And IUCN Asia (Box 3)

15. Formal performance monitoring system abandoned Empowerment of frontline staff for rapid response Absence of a ‘model’ to assess performance Let partners adapt and adopt measures Donor modified its approach..more ‘learning-friendly’ faced with diversity Unusually diverse membership Flexibility demonstrated by funding agencies Permitted experimentation, innovation and creativity Evolution of IUCN learning processes ‘Teaming’ process Ownership by governments + IUCN credibility / legitimacy CB= continuous process ‘no road map, only a goal’ ENACT IUCN

16. Cases’ commonalities c.f. ST • Identification and recognition of goals • Emphasis on values to be reflected • Clarity and awareness of mission amongst clients too • Leadership encourages experimentation • Opportunities for learning from experience: self-assessment and ‘stories’ of positive experience or changes or errors • Flexibility to adapt (thru new skills-building) to new needs / priorities, thru OJT ‘hands-on’ • Informality of M&E systems: responsive to needs of clients / network members • Ability to learn from experience is crucial

17. BUT ‘Reductionist’ approaches are still relevant…where: • It is possible to define required capabilities unambiguously and specifically • ..and to assess existing ones = ‘gap’ • Therefore easy to define indicators • Where stakeholders able and willing to define their shortfalls and ‘sign up’ • Incentives exist to improve performance • Leadership, and all above combine into ‘ownership’ • BUT ..this combination of circumstances is rare! (e.g. public financial management IMF and WB forged consensus)

18. Towards a ‘Tool Box’ for M&E of CD in Climate Change • Principles: pragmatism: acknowledge weaknesses in all M&E systems: main aim = ‘sense-making’ • Seek out ‘what M&E exists, and what works, already’ • Acknowledge that the best M&E systems are ‘customised’: adapted by participants, based on local conditions

19. Some existing frameworks • GEF Resource Kit on M&E • GEF Indicators for 5 key capacities: • Engagement; generate access to and use of information; policy and legislation development; management and implementation; monitor and evaluate • Scoring / indicator system for each; • Obliges actions / next steps / link to outcomes • UNEP ‘Lessons Trees’ • Trying to improve quality of learning (especially about common problems) and application of learning to future programmes

20. Self-Assessment • Examples from CB in research and development organisations • SA workshops after applying qualitative and quantitative tools • Managers, staff and stakeholders identify strengths and weaknesses, and set new directions ….Advantages = (1) those with knowledge of and interest in the organisation gain in-depth insights: what’s working and why..where improvements needed (2) Well-prepared to address the changes needed • E.g. M&E of past CB in Mekong Farming R&D Systems Institute (Box 4) • Preparation of ‘work stories’ on past CB efforts • + interviews with key staff on changes and challenges

21. A ‘Balanced Approach’ to M&E of CB • Framework generated to address M&E of CB a la ECDPM Study dimensions of ‘Capacity’ • Piloted in PNG legal and judicial reform • Time-consuming and exhaustive: but national practitioners did find it helpful in conceptualising all dimensions of capacities being built up. • See extra handout

22. Appreciative Enquiry in Formative Evaluation • Evaluators form more of an understanding of the political, cultural and historical ‘landscape’ • Encouraging organisations to develop their relationships with primary stakeholders • Community Development Corporations in US • Seeking out ‘what enabled effectiveness, and what hindered it’ • Evaluator regularly listening to ‘stories’ • Regular visits and sharing of reports • Recipients able to ‘co-create’ the initiative and develop OWN capacity for assessment • Evaluator chosen by NGO not donor: NGO employed not as expert but for its ability to learn collectively

23. Most Significant Change (MSC) • First applied in evaluation of a complex RD programme in Bangladesh • Process managers identify domains of change which are important to evaluate • Stories (descriptions of changes deemed significant: with reasons: why significant?) periodically collected from stakeholders • Analysed and filtered up thru committees • Criteria for choosing stories are collated and fed back to stakeholders; • Final selection made (annually?) with reasons • Circulated to all • Site visits to check; deepen understanding of changes

24. MSC features • Focuses attention and direction of work in programme towards valued directions • Dialogue and deliberations crucial • Takes place over time: responsive to changing contexts • Policy makers, funders, field engaged looking at the value of changes • Stories help all relate to information • Non-experts (the story-tellers) involved in evaluation • Dialogue based on real experience and concrete outcomes not abstract indicators • …MSC positively evaluated as technique in Laos (Willetts 2004)

25. Annotated Bibliography • ..use as a part of the ‘tool-kit’! • Sections on: • Climate Change-related sources • Broader CB literature and M&E • Other Global Programmes’ Materials • Civil Society CB and M&E • Systems Thinking and Complexity literature and M&E examples

26. Summary: why ST has potential for M&E of CB in Climate Change • Avoids pitfalls of logical framework in what is an amorphous field: CB • Emphasises clarity of objective-setting, AND of learning collectively from reality • Generates, relates to, and values ‘stories’ • Potential to enhance team-work and inter-relationships of hitherto disparate institutions and groups in their ‘ecosystem’ context; • Climate change context – politically-charged, formal and informal, amenable to negotiation • Evidence indicates ST M&E approaches ‘can work for and strengthen social change’ (Guijt 2007 IDS)