Project Ratings: Connects and Disconnects

1. Project Ratings:Connects and Disconnects SoniyaCarvalho Lead Evaluation Officer and ICR Review Coordinator, IEGPS Independent Evaluation Group

2. How are Projects Rated at the World Bank? • Project ratings are based on evaluation criteria that are harmonized across IEG and OPCS • Main ratings are: Outcome, Risk to Development Outcome, Bank Performance, and Borrower Performance • Evaluation methodology is objectives-based

3. What are the Harmonized Evaluation Criteria? • Outcome = Relevance + Efficacy + Efficiency • Risk to Development Outcome = Likelihood of detrimental change x Impact on outcome if that change materializes • Bank Performance = Quality at Entry + Quality of Supervision • Borrower Performance = Government Performance + Implementing Agency(ies) Performance

4. Why do ICR and ICR Review Ratings Differ? Reason #1: Disagreement over project objectives or their weighting • Objectives in ICR are not taken from PAD/Financing Agreement • ICR does not discuss how any poorly articulated objectives have been interpreted • “By” and “through” parts of objectives are treated as the intended outcomes • Relative weights given to different objectives in arriving at ICR ratings are not made explicit and justified

5. Why do ICR and ICR Review Ratings Differ (Cont’d) Reason #2: Insufficient evidence on achievement of objectives/efficacy • Too much focus on outputs and inputs to the neglect of outcomes • Reported Key Performance Indicators do not fully capture achievement of objectives • Results chain neglected – attribution/plausible association between outputs and outcomes not adequately explained

6. Why do ICR and ICR Review Ratings Differ? (Cont’d) Reason #3: Differences in treatment of country circumstances • In rating Project Outcome, IEG makes no allowance for difficult country context; objectives and design are supposed to take that into account • However, difficult country contexts do factor into the Bank Performance rating • Project Outcome rating is based on results not effort

7. Why do ICR and ICR Review Ratings Differ (Cont’d) Reason #4: Lack of familiarity with the ratings • ICR rates project outcome based only on achievement of objectives, neglecting relevance and efficiency • Relevance in ICR may ignore current relevance • Harmonized criteria for deriving Bank Performance from its two constituent elements not observed (Ditto for Borrower Performance)

8. How can the disconnect be reduced? • Ensure clarity about the objectives being used as the benchmark for evaluation • Ensure that the ICR contains evidence on the entire results chain, from outputs to outcomes, including for institutional objectives • Explicitly discuss attribution/plausible association, providing contextual information on overlapping government or other donor activities in the sector

9. How can the disconnect be reduced? (Cont’d) • Discuss the relevance of project outcomes to the current CAS • Acknowledge gaps in achieving the intended outcomes, drawing insightful lessons • Acknowledge gaps in M&E, indicating proxy information used to underpin the ICR’s judgments

10. Even an unsatisfactory project with poor M&E can win an IEG Good Practice ICR Quality Award! Thank you