1 / 10

Project Ratings: Connects and Disconnects

Project Ratings: Connects and Disconnects. Soniya Carvalho Lead Evaluation Officer and ICR Review Coordinator, IEGPS Independent Evaluation Group. How are Projects Rated at the World Bank?. Project ratings are based on evaluation criteria that are harmonized across IEG and OPCS

payton
Download Presentation

Project Ratings: Connects and Disconnects

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Project Ratings:Connects and Disconnects SoniyaCarvalho Lead Evaluation Officer and ICR Review Coordinator, IEGPS Independent Evaluation Group

  2. How are Projects Rated at the World Bank? • Project ratings are based on evaluation criteria that are harmonized across IEG and OPCS • Main ratings are: Outcome, Risk to Development Outcome, Bank Performance, and Borrower Performance • Evaluation methodology is objectives-based

  3. What are the Harmonized Evaluation Criteria? • Outcome = Relevance + Efficacy + Efficiency • Risk to Development Outcome = Likelihood of detrimental change x Impact on outcome if that change materializes • Bank Performance = Quality at Entry + Quality of Supervision • Borrower Performance = Government Performance + Implementing Agency(ies) Performance

  4. Why do ICR and ICR Review Ratings Differ? Reason #1: Disagreement over project objectives or their weighting • Objectives in ICR are not taken from PAD/Financing Agreement • ICR does not discuss how any poorly articulated objectives have been interpreted • “By” and “through” parts of objectives are treated as the intended outcomes • Relative weights given to different objectives in arriving at ICR ratings are not made explicit and justified

  5. Why do ICR and ICR Review Ratings Differ (Cont’d) Reason #2: Insufficient evidence on achievement of objectives/efficacy • Too much focus on outputs and inputs to the neglect of outcomes • Reported Key Performance Indicators do not fully capture achievement of objectives • Results chain neglected – attribution/plausible association between outputs and outcomes not adequately explained

  6. Why do ICR and ICR Review Ratings Differ? (Cont’d) Reason #3: Differences in treatment of country circumstances • In rating Project Outcome, IEG makes no allowance for difficult country context; objectives and design are supposed to take that into account • However, difficult country contexts do factor into the Bank Performance rating • Project Outcome rating is based on results not effort

  7. Why do ICR and ICR Review Ratings Differ (Cont’d) Reason #4: Lack of familiarity with the ratings • ICR rates project outcome based only on achievement of objectives, neglecting relevance and efficiency • Relevance in ICR may ignore current relevance • Harmonized criteria for deriving Bank Performance from its two constituent elements not observed (Ditto for Borrower Performance)

  8. How can the disconnect be reduced? • Ensure clarity about the objectives being used as the benchmark for evaluation • Ensure that the ICR contains evidence on the entire results chain, from outputs to outcomes, including for institutional objectives • Explicitly discuss attribution/plausible association, providing contextual information on overlapping government or other donor activities in the sector

  9. How can the disconnect be reduced? (Cont’d) • Discuss the relevance of project outcomes to the current CAS • Acknowledge gaps in achieving the intended outcomes, drawing insightful lessons • Acknowledge gaps in M&E, indicating proxy information used to underpin the ICR’s judgments

  10. Even an unsatisfactory project with poor M&E can win an IEG Good Practice ICR Quality Award! Thank you

More Related