RESULTS-BASED M&E AND IMPACT EVALUATION CONCEPTS

1. RESULTS-BASED M&E AND IMPACT EVALUATION CONCEPTS by Suganya Hutaserani Lead Evaluation Specialist Independent Evaluation Department, ADB Presentation at SHIPDET Special Topic Courses 14-18 November 2011

2. Contents of Presentation Links between Results-Based Monitoring & Evaluation (M&E) and Performance-Based Planning & Budgeting (PBB), in relation to Managing for Dev. Results (MfDR) How to Carry Out Project “Evaluation”?: Project Evaluation Criteria and Rating System What is Meant by “Impact” in “Project Design & Monitoring Framework (DMF)/Logframe”? What is Meant by “Impact” in “Rigorous Impact Evaluation (RIE or IE)” and How to Include It in Project Evaluation Criteria? Why is RIE or IE Needed? 2

3. Links between Results-Based M&E and PBB • Performance-based planning and budgeting (PBB) means that the entire planning and budgeting framework of a country is policy-oriented and results-based (with performance targets), linking “resources” to “results.” • In countries where M&E systems are well developed (results-based) and integrated into their PBB, the whole process is then considered a driver of success for socioeconomic progresses. • This is because performance targets (e.g., outcome and impact indicators) at various phases in the planning cycle could be monitored and evaluated, with evaluation findings and lessons fed into the PPB process to re-allocate resources to facilitate the achievement of performance targets in subsequent phases. • Thus, results-based M&E and PBB are important tools in support of the country’s MfDR

4. How to Carry Out Project Evaluation?: Project Evaluation Criteria Core Criteria: Relevance (20%) = consistencies of project objectives with the country needs, gov.’t plans, ADB strategies; and design quality (e.g., quality of DMF, M&E design, technical aspects) Efficiency (30%) = project portfolio/implementation performance, resource utilization (including EIRR) Effectiveness (30%) = project achievement of outcomes Sustainability (20%) = prospects for sustaining project outputs/outcomes or “risks” to sustaining outcomes Non-Core Criteria: Impacts = country’s achievement of dev. results (incl. MDGs related to the project and other socioeconomic effects) Borrower/EA Performance = gov.’t ownership, compliance with loan covenants, and provision of counterpart funds/staff ADB Performance = role in aid coordination, extent of supervisions, other value addition (e.g., catalytic role) 4

5. Project Evaluation Criteria and Weights 5

6. Project Evaluation Rating System

7. Sample DMF/Logframe of a Project in the Education Sector 7

8. What is Meant by “Impact” in “Project Design & Monitoring Framework (DMF)/Logframe”? Outputs = amount of work, goods and services produced by a project in a period (e.g., # of schools built, # of teachers trained, new course materials produced; length of roads upgraded) Outcomes = results of outputs of the project (e.g., increased student enrollment and completion rates in schools in project areas; reduced travel time and cost in project areas; increased household income in project areas), compared with those “before” the project Impact = higher-level outcomes “contributed to” by the project (e.g. increased employment rate, NER and literacy rate, and other MDGs), compared with those “before” the project 8

9. What Is Meant by “Impact” in “Rigorous Impact Evaluation (RIE or IE)” and How to Include It in Evaluation Criteria”? Outputs = amount of work, goods and services produced by a project in a period (e.g., # of schools built, # of teachers trained, new course materials produced; length of roads upgraded) Outcomes = results of outputs of the project (e.g., increased student enrollment and completion rates in schools in project areas; reduced travel time and cost in project areas; increased household income in project areas), compared with those “before” the project Impact = the “outcomes” of the project compared to what they’d have been in the absence of the project (i.e., comparison of outcome data b/w “project areas” and “control areas” or b/w “with” and “without” project). This means that “outcome” (rather than “impact”) indicators from DMF are normally used in doing RIE or IE, and the “impact” estimated by IE will be assessed under the “effectiveness” criterion. 9

10. What Is Meant by “Impact” in “Rigorous Impact Evaluation (RIE or IE)”? 10

11. Why is Rigorous Impact Evaluation (RIE or IE) Needed? Generally, when evaluating “effectiveness” of a project, we compare the outcome data b/w “before” and “after” project (e.g., increased HH income from $2/day in 2006 to $8/day in 2011). However, such an increase might not be due solely to our project, but also influenced by assistance of other donors/projects. To find out the extent to which such an increase is due to our project, (“attribution”) as an evidence supporting value for money of our investment, we need to use RIE or IE method by comparing outcome data b/w “project/treatment area” and “counterfactual/control area,” and use rigorous techniques (e.g. randomized control trials, or quasi-experimental technique) to arrive at an unbiased estimate of the “impact.” 11

12. THANK YOU WWW.ADB.ORG/EVALUATION