1 / 28

Monitoring and Evaluation in CBA - The specificity of M&E in the field of adaptation

Monitoring and Evaluation in CBA - The specificity of M&E in the field of adaptation. UNDP M&E framework for adaptation, VRA and IAS C. O. Nyandiga and A.F Wittmann Presented at the UNDP CBA- UNV Workshop, August 2-10, Dakar-Senegal. What is Monitoring and Evaluation?.

emory
Download Presentation

Monitoring and Evaluation in CBA - The specificity of M&E in the field of adaptation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Monitoring and Evaluation in CBA- The specificity of M&E in the field of adaptation UNDP M&E framework for adaptation, VRA and IAS C. O. Nyandiga and A.F Wittmann Presented at the UNDP CBA- UNV Workshop, August 2-10, Dakar-Senegal

  2. What is Monitoring and Evaluation? • Monitoring = A continuous data collection and analysis process implemented to assess a project (or program or policy) and compare it with the expected performance • Evaluation = A systematic and objective measurement of the results achieved by a project, a program or a policy, in order to assess its relevance, its coherence, the efficiency of its implementation, its effectiveness and its impact, as well as its sustainability (source : World Bank)

  3. Why do we do M&E ? • Monitoring and Evaluation = a systematic and on-going process, that is part of project inception, planning and implementation • M&E is necessary to ensure that the OBJECTIVES of the project are met M&E • Provides project management and stakeholders with on-going indications of progress or lack of progress in achievements of objectives • Helps identifying problems and successes of projects, during implementation • Provides a basis for corrective actions • Helps reinforcement of initial positive results • Helps in determining if a project is still relevant

  4. UNDP M&E Adaptation framework ADAPTATION : What do we want to measure ? We want to make sure our objectives are achieved : improving the adaptive capacity / reducing the vulnerability of the communities and the ecosystems on which they rely. What can we measure ? • Coverage : extent to which the project reaches vulnerable stakeholders (individuals, households, businesses, government agencies, policymakers…) • Impact : extent to which the project reduces the vulnerability, through policymaking, capacity building…) • Sustainability : ability of stakeholders to continue the adaptation process beyond project lifetime • Replicability : extent to which projects generate and disseminate results and lessons of value for replication

  5. UNDP M&E Adaptation Measurements at Project Level for all Thematic Areas Coverage • i. Number of households, businesses engaged in vulnerability reduction or adaptive capacity development activities, as a proportion of households in the community or region targeted by the project. • ii. Number of policies introduced or adjusted to incorporate climate change risks. • iii. Number of investment decisions revised or made to incorporate climate change risks. • iv. Number of stakeholders (individuals, households, communities, etc.) served by new or expanded climate information management systems (e.g. early warning systems, forecasting,etc.). Impact • i. Percent change in stakeholders’ behaviours utilizing adjusted practices or resources for managing climate change risks, assessed via QBS. • ii. Percent improvement in stakeholders’ capacities to manage climate change (e.g. communicate climate change risks, disseminate information, or make decisions based on high quality information), as relevant, assessed via QBS. • iii. Percent reduction in perceived vulnerability: • a. Percent improvement in stakeholder perceptions of vulnerability to a recurrence of primary climate change-related threat(s), assessed via QBS.

  6. UNDP M+E Adaptation Measurements for all TA -ctd Sustainability • i. Number of beneficiaries of project receiving training in implementation of specific adaptation measures or decision-support tools. • ii. Local (or spatially appropriate) availability of skills and resources necessary to continue adaptation after conclusion of project, assessed via QBS. • iii. Support for project activities among participating communities as assessed by QBS. • iv. Number of outside programmes, policies or projects incorporating project results into their processes. Replicability • i. Number of “lessons learned” from the project

  7. CBA M&E Adaptation framework • CBA falls within the UNDP Framework for M&E of Adaptation to Climate Change • The specificity of CBAdaptation = it is a challenge to monitor and evaluate it, because : • “Adaptation is not generally an outcome, but rather consists of a diverse suite of ongoing processes that enable the achievement of development objectives under climate change” • “Adaptation cuts across many development objectives” • The results of adaptation can be measured in the long-term, whereas projects have a short duration • It is difficult to decouple climate risks from other drivers • (UNDP Framework for Monitoring and Evaluation of ACC)

  8. M&E in the CBA programme • M&E aims to ensure and show that the CBA objectives are met : • Building adaptive capacity • Reducing community vulnerability • Increasing the resilience of global environmental benefits to climate change (including variability) in the GEF focal areas • M&E applies to the 3 CBA outcomes • Outcome 1 – Local level (projects) • Outcome 2 – National level • Outcome 3 – Global level

  9. CBA M&E system • 4 sets of indicators are monitored • The UNDP Adaptation Indicators • Adaptive capacities • Sustainable community management of natural resources • The SGP’s Impact Assessment System (IAS) • GEB • Livelihood / Empowerment • The Vulnerability Reduction Assessment • Volunteerism and Inclusive Participation indicator

  10. CBA M&E system • 4 sets of indicators are monitored • At the project level • Aggregated at the national level • Then again aggregated at the global level

  11. UNDP adaptation indicators • Adaptive capacity fostered among natural resource dependent communities through awareness building, enhanced access to climate change and scenario information, and improved access to alternative resources • 2.1 Number of stakeholders at community level (e.g. businesses, community representatives, CBOs, NGOs) engaged by project and provided with training in climate change risk management and scenario planning. • 2.2 Population covered by awareness building programmes to increase understanding of risks associated with climate change among general public and key stakeholder groups. • 2.3 Increase in awareness of climate change related risks to natural resources (QBS) • 2.4 Percentage change in natural resource dependent population with access to alternative or supplementary livelihood options (QBS) • Sustainable community management of natural resources in the face of climate change promoted • 4.1 Percentage of population in relevant areas engaged in sustainable community management activities • 4.2 Number of measures deployed as part of sustainable resource management activities. • 4.3 Percentage of area of concern in which sustainable resource management activities are implemented. • 4.4 Number of stakeholders (e.g. families/households) benefiting from sustainable resource management activities (e.g. in terms of increased income or food security). • 4.5 Success of sustainable resource management interventions in securing livelihoods and protecting resources (QBS).

  12. SGP’s Impact Assessment System • Why ? Measure the Global Environmental Benefits and the Livelihood / Empowermentbenefits generated by the project • What ? • GEB / • Biodiversity-(# of species, innovations/new technologies, local/national policies) • Land Degradation- (ha. of land restored, land sustainably managed, tons of soil erosion prevented, # of innovations/new technologies, local/national policies) • Livelihood / Empowerment / • Poverty Reduction –(# of households or individuals who benefited from the project, income generation achieved through the project…) • Capacity Building –(# of NGOs, community groups whose capacities were increased, #of women participating, support ensured from local / governmental institutions…)

  13. SGP’s Impact Assessment System ctd. • Who, How ? And When ? At project conception, chose one GEB indicator and one LIVELIHOOD / EMPOWERMENT indicator During project development, measure the baseline values of the indicators and prepare your monitoring plan (in project proposal) – During project implementation, measure the evolutions of the indicators (project reports) – RESPONSIBILITY OF THE GRANTEE At the end of the project, measure the final indicator (final participatory evaluation / final report) – RESPONSIBILITY OF THE GRANTEE • Monitoring Plan – • Whatyouwillmeasure (indicator) • Whatis the target value of yourindicator • Howyouwillmeasureit • Whenyouwillmeasureit • Whowillmeasureit Don’tforget to report the indicators in the SGP database

  14. Examples of the type of possible indicators for GEB Arid & semi-Arid zone ecosystems • Local population estimates of arid and semi-arid zone species • Changes in extent of undegraded arid and semi-arid habitat. Coastal, Marine and Freshwaters: • Local population estimates of target aquatic species • Changes in extent of healthy coral reef • Changes in extent and quality of mangroves (patch size, maximum tree size). Forest: • Local population estimates of forest species • Changes in natural forest extent • Number of concessionaries in the formulation of timber extraction Mountain: • Local population estimates of montane species • Changes in extent of undegraded montane habitat • Rate of deforestation on slopes >20% in project areas Agrobiodiversity: • Changes in number of local land races & domestic livestock breeds. • Changes in extent of agricultural area using low-input high diversity production methods • Incorporation of local land races and indigenous breeds in national breeding programmes • Certification standards for agricultural products

  15. Ex-post measurements WHAT • Systematically capture and document off-site, indirect and longer-term impacts- “ex-post evaluation”. Generates GEB Measurements HOW • Using proxy and development indicators while project is ongoing • Using structured QBS during and after project closure • External independent consultants WHEN • Minimum 5 years after project completion-Done by UNDP EO

  16. Vulnerability Reduction Assessment (VRA) • What ? The objective of a CBA project is to REDUCE the vulnerability of the community. VRA is a tool that contributes to measure achievement of this objective. VRA is a form of Participatory Impact Assessment, which focuses on the community’s own perception of its vulnerability and adaptation capacities. • Why ? • Measure the community’s perceptions of current and future climate change risks and adaptive capacities (now and in the long-run) • Measure the obstacles to adaptation, and the assets • Evaluate the sustainability of the project • Capture qualitative information / knowledge from the community, essential for project development/management (to make sure the project responds to community priorities) and for knowledge-management • Capture quantitative (thus comparable) information (=scores) that will allow to verify the achievement of CBA objectives (at project / national / global levels)

  17. Vulnerability Reduction Assessment (VRA) • How ? When ? VRA is composed of 4 indicators, that are transformed into 4 questions (tailored to the local context). These questions form the core of a “VRA participatory workshop”, that is organized at least 3 times in the course of a project, in the community (at the beginning, at mid-course of project implementation, at the end).

  18. Vulnerability Reduction Assessment (VRA) • The H-Form = the tool that is used for VRA facilitation and data collection

  19. Organizing a VRA session Where ? Choose an appropriate location in the village, according to what you already know about the community. The location has to be accepted by / accessible to everyone When ? Accommodate the community schedule, including women’s schedule Who ? Facilitating team : 1 or 2 lead facilitators (with leverage and experience ; who know the local language and culture) 1 person to take notes on the H-form 1 person to take extensive notes on the side 1 person to take pictures and coordinate the logistics MOBILIZE the grantee as well as community-members. BE GENDER-SENSITIVE QUICK TIPS -Make sure all the members of the community are invited and informed about the meeting -Create a friendly atmosphere, where everyone is at ease to speak -Promote open discussion (the participants are the “teachers”, because we want to learn from them and capture their perceptions) -Use a language that is appropriate -Prepare your H-forms in advance (as well as all the material you might need) -Inform the authorities in advance

  20. Sequencing a VRA session (example from Morocco) • 1-Introduction • Present the facilitating team • Explain the context of the workshop (CBA project) and the objectives of the workshop • Explain the “rules” : promote open participation (there is no wrong / right answer; everyone is allowed to speak…) • 2-Launch a general discussion about the local environment and climate • Use simple questions and tackle climate aspects one by one • what is the “usual” climate here ? • when does it rain ? when is it hot ? For how long… • Are there any storms, droughts ? When ? • you can draft a season calendar for example • 3-How does the community experience climate change? • Have you noticed any changes in the past years ? • Has it rained more ? less ? When did it rain last year ? for how long ? • Is it hotter than in the past ? is the dry season longer ? • Are there more storms etc. QUICK TIPS • -Adapt to relevant climatic matters • -Be very specific : referring to “climate change” is too vague in most contexts • -Ask simple questions one by one, so people have time to discuss them

  21. Sequencing a VRA session (example from Morocco) 4-What are the impacts of these changes on your livelihood = VRA QUESTION 1 / Fill out H-form 5-What will be the impacts if these changes (more storms / more droughts) double in the future ? = VRA QUESTION 2 / Fill out H-form The impacts are noted in 2 columns (positive impacts / negative impacts) After the discussion, assessment : how serious are these impacts, in the view of the community What are the possible solutions to limit these impacts ? 6-What are the obstacles / assets for adaptation ? = VRA QUESTION 3 / Fill out H-form What prevents you from implementing the solutions that you mention ? After the discussion, assessment : are the obstacles stronger than the assets What are the possible solutions to overcome the obstacles and build on the assets ? 7-Do you think that with the project, you will be able to adapt further in the future ? = VRA QUESTION 4 / Fill out H-form (Assess the sustainability of the project, and how people will contribute) After the discussion, assessment : are the participants optimistic about the outcomes of the projects ? are they willing to actively participate ? 8-Conclusion Thank people for their participation Present the next steps of the project

  22. Vulnerability Reduction Assessment (VRA) • What are the results of a VRA session • QUANTITATIVE DATA = SCORING Generate a score for each question (noted on 5; 1 shows that the vulnerability is high / 5 shows that the vulnerability is low Calculate the average from each question Multiply the total average by 2 to obtain a score noted on 10 • QUALITATIVE DATA = all the information captured during the session, and classified in the different parts of the H-form • Negative impacts • Positive impacts • Ideas, Solutions • How to use the results of a VRA session • The initial VRA results • Are used for project development an incorporated in the project proposal (provides data for baseline assessment / local knowledge / and ideas for project outcomes/outputs) • Provide the baseline vulnerability measurement, reference for project monitoring • The 2nd and 3rd VRA results are to be compared with baseline, in order to monitor the project. QUICK TIPS • Always make sure you thoroughly document the VRA sessions • Take detailed notes on your own H-form, • Note your personal side-comments –how many people ? Where ? When ? Who facilitated ? What went right / wrong ? What can be improved ? … • Take pictures Don’tforget to report the indicators in the SGP database

  23. Vulnerability Reduction Assessment (VRA) • Lessons learned from implementation • VRA is a powerful way to capture a lot of information from the community • It gives an opportunity for mobilization and discussion, and contributes to fostering community ownership of the project • Language and communication are critical • Gender appropriateness is essential (in the choice of the location, scheduling, language, facilitators) : in Morocco, for example, we do specific women’s workshops • VRA is a flexible tool : its organization can be tailored to the specific context (for ex. In Morocco, we don’t always use the big H-form, and we added some questions/discussions to prepare for the VRA questions) – PROVIDED ALL THE INFO IS COLLECTED FROM THE DISCUSSION • VRA can be incorporated in a larger workshop (information about CC, training, rural assessment…) so that the communities are prepared and so that the discussions are to the point. • The main challenge is the QUANTITATIVE DATA (for discussion) How can they be better understood (scale of value is not necessarily translated in numbers) ? How can they be better captured (voting/scoring might block participants; people don’t want to stand out)? How can they be more reliable for monitoring (composition of the group changes from one VRA to the next ; participants might over/underestimate the scores in order to ensure the funding… ? There is a potential for improvement, in order to be able to prove the impacts of our projects.

  24. Volunteerism / Inclusive participation BASELINE indicators (project proposal) • What are the mechanisms for volunteerism that already exist in the community before the CBA project (for example, traditional mechanisms for mutual assistance, associations, etc.)? • Number of volunteers in the community already engaged in climate change adaptation activities before the CBA project • What are the opportunities or obstacles that could facilitate or impede people from engaging in voluntary activities? The CBA project aims at reinforcing Volunteerism : these indicators should be monitored over the course of the project Be sure to capture quantitative / qualitative data on community volunteers and participation during the project : in the activity reports / follow-up on the Volunteerism chart Inclusive participation : monitor the vulnerable group’s participation (esp. women, youth, people with a handicap, elders, property-less farmers…). In the activity reports / follow-up on the Volunteerism chart

  25. M&E timeline

  26. Conclusion and discussion • More and more, we are asked to prove our results. • M&E is ESSENTIAL to show that our projects achieve their objectives. • This is why we need to assess the challenges of our system, and try to improve it.

  27. Pleaseshare … • How do you implement our M&E ? • How do you explain it to the proponents / communities, and support them in implementing ? • What are the challenges you are facing ? • What do you need to overcome these challenges ?

  28. QUESTIONS • Quantitative aspect of VRA ? • Consolidate our 4 set of indicators into 1, without losing any of the information ? • Incorporate project outcome indicators ?

More Related