Evaluation methods training and capacity building programs
1 / 21

Evaluation Methods Training and Capacity Building Programs - PowerPoint PPT Presentation

  • Uploaded on

Evaluation Methods Training and Capacity Building Programs. Nidhi Khattri Independent Evaluation Group November 17, 2008. IEG ’ s Mandate. The World Bank’s independent evaluation function established about 35 years ago The goals: learning from experience

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Evaluation Methods Training and Capacity Building Programs' - abra

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Evaluation methods training and capacity building programs

Evaluation MethodsTraining and Capacity Building Programs

Nidhi Khattri

Independent Evaluation Group

November 17, 2008

Ieg s mandate
IEG’s Mandate

  • The World Bank’s independent evaluation function established about 35 years ago

  • The goals:

    • learning from experience

    • accountability for the achievement of objectives

Ieg s independence direct report to the board
IEG’s Independence: Direct Report to the Board

  • Direct reporting to the Board of Executive Directors

  • Headed by a Director-General, Evaluation (DGE)

    • Appointed by the Board

    • No World Bank Group position after current position

  • Evaluations to the Committee on Development Effectiveness (CODE)

  • Evaluation content not negotiated with CODE/Board

Ieg s links to bank management
IEG’s Links to Bank Management

  • Bank management has opportunity to comment

  • Draft Bank Management Response accompanies evaluation

  • IEG responds to Bank management comments at Board meetings

  • IEG’ Management Action Record (MAR) reports on management’s progress on actions noted in management response

Ieg s evaluation products
IEG’s Evaluation Products

  • Project Evaluations

    • Project assessments (ICR Reviews, PPARs)

    • Impact studies (e.g., Bangladesh Health, Ghana Education)

  • Sector and Thematic Evaluations

    • Often linked to policy revision (e.g., forestry – altered sector policy)

  • Country Evaluations

    • Country Assistance Evaluations

    • Country Impact Reviews (IEG-IFC)

    • Reviews of CAS Completion Reports

  • Global and Regional Program Evaluations

  • Corporate Evaluations

    • Annual Review of Development Effectiveness (which now includes the Annual Report on Operations Evaluation)

Evaluation approaches
Evaluation Approaches

  • Based on evaluation products

  • Primarily Objectives Based for Projects and Programs

    • Outcome

    • Risk to Development Outcome

    • Bank Performance

    • Borrower Performance

    • Monitoring and Evaluation (M&E) Quality


  • The extent to which the operation’s major relevant objectives were achieved, or are expected to be achieved, efficiently

  • Outcome = Relevance + Efficacy + Efficiency

Bank performance
Bank Performance

  • The extent to which services provided by the Bank ensured quality at entry of the operation and supported effective implementation through appropriate supervision

  • Bank Performance = Quality at Entry + Quality of Supervision

Monitoring and evaluation m e quality
Monitoring and Evaluation (M&E) Quality

  • M&E design—the extent to which the project was designed to collect appropriate (input, output, outcome, and impact) data given project objectives and given already available data

  • M&E implementation—the extent to which appropriate data was actually collected using appropriate collection methods (to ensure data quality)

  • M&E utilization—the extent to which appropriate data was used to inform decision-making and resource allocation

Corporate evaluation annual review of development effectiveness arde
Corporate Evaluation: Annual Review of Development Effectiveness (ARDE)

  • Annual meta-evaluations that provide a comprehensive assessment of the Bank’s development effectiveness

  • Draw on IEG’s recent project, sector, thematic, country, and global evaluations

  • Synthesize lessons that can be used to increase the development effectiveness of World Bank assistance

  • Highlight the findings of recent IEG evaluations around a common theme

Recent ieg evaluations around training and capacity building
Recent IEG Evaluations around Training and Capacity Building Effectiveness (ARDE)

  • Using Training to Build Capacity (2007)

    • Avg amount of client training estimated at $720 mio per year (90% through projects, rest through WBI)

    • Key component in 60% of investment projects, particularly in social, rural, public sectors

    • 37 (incl. 8 WBI) training programs (Bangladesh, Burkina Faso, Mexico, Tunisia), 6 country surveys of 550 trainees, comparison with other DTIs, etc.

  • Capacity Building in Africa (2005)

    • 25% of Bank lending to Africa; $9 bill between 1995-2004

  • Public Sector Reform (2008)

    • 1/6 of Bank lending and advisory support – and increasing

Methodology in evaluation of training
Methodology in Evaluation of Training Effectiveness (ARDE)

  • Key Questions:

    • To what extent did Bank-financed training have an impact on the capacity of target organizations?

    • What factors contribute to successful training?

    • To what extent are such factors present in Bank-financed training?

  • Main Methods – using objectives-based methodology

    • Survey of training participants

    • Survey of training institutions

    • In-depth field reviews

Training results in capacity building only when certain conditions are met
Training Results in Capacity Building Only When Certain Conditions are Met




OutcomeWorkplace behavior change

ImpactEnhanced institutional or organizational capacity

Effectiveness of training what is the evidence
Effectiveness of Training: What is the Evidence? Conditions are Met

  • Most training resulted in demonstrable learning

  • But: Individual learning gains poor predictor of impact

  • Project-based trained lacked basic results measures




  • About half of trainees surveyed reported substantial

  • positive changes in work performance

Impact on


  • 10 project-based programs had significant impact (e.g.,

  • Procurement reform, Community Groups, Exporters, SME)

  • Best combined project funds + outside expertise

  • WBI programs not rated due to lack of data



  • Type of training provider not correlated with success

  • Good training outcomes in both higher- and low-

  • capacity environments

Training design what works
Training Design: What Works? Conditions are Met

  • Adequate diagnosis of capacity gaps associated with

  • strong client commitment / involvement

  • Training needs assessment the norm in highly-rated

  • programs, but often subject to funding constraints (WBI)

  • Good participant selection requires engagement and

  • supervision (e.g., IMF Institute, MASHAV)

  • Poor targeting most important cause for lack of impact

  • Generally high marks for design & teaching standards

  • But course length and topic coverage needs to be better matched with capacity building goals (ITCILO,JICA)

  • Allow time for practical learning techniques / action plans (InWent, JICA, MASHAV)

  • Provide systematic follow-on support (Motorola, InWent)


of Training





When training works what matters most
When Training Works: What Matters Most? Conditions are Met

  • Support by managers and peers is key driver for

  • successful workplace implementation (~90% feedback)

  • But: 1/3 of trainees didn’t have adequate resources,

  • incentives or org. support to apply what they learned

  • Focus needs assessment on organize. bottlenecks and

  • whether training is indeed the right tool.

  • Lack of institutional incentives recurring problem in civil

  • service and highly decentralized training programs;

  • Stronger incentives at work in training programs for

  • community and farmer groups, private sector firms

  • Hi-level support for training key for workplace transfer

  • Stand-alone training with limited ability for diagnosis,

  • dialogue, influence and follow-up






Key internal inhibitors for effective training
Key Internal Inhibitors for Effective Training? Conditions are Met

  • At design, most training programs fail to specify training

  • objectives and expected performance outcomes,

  • reflecting lack of broader capacity needs assessments

  • At completion, performance evaluation rarely done

  • No feedback / accountability loop >> no improvements

  • Lack of established standards for training design and

  • implementation undercuts quality assurance;

  • Team leaders for project-based training lack adequate

  • in-house support and voiced demand for more

  • WBI earns high marks for Country Team consultations,

  • but collaboration at task level remains rare. Risk of

  • diffusion of program.




Access to



Persistent capacity gaps in africa despite substantial inputs ieg 2005
Persistent Capacity Gaps in Africa Despite Substantial Inputs (IEG, 2005)

  • 40% of sampled lending operations achieved CB goals, with

  • better outcomes in roads than in health / education.

  • CB lacked clear results framework (only 1/3 of projects clear about relationships among individual, organizational. and institutional aspects of capacity)

  • Weak diagnostics of pol. econ and available country capacity

  • High fragmentation of efforts; supply driven TA

  • Training not embedded in broader HR strategies;

  • Strengthen K-base, operational framework, M&E

  • Develop sector-specific guidance

  • Promote country-led approach

  • Re-assess role / modalities of training

  • Capacity Building has moved to center stage in AFR CASs

  • Focus shifting beyond individual skills to institutional support

  • leadership, donor harmonization and better coherence

Key Finding






Mixed outcomes on public sector reform ieg 2008
Mixed Outcomes on Public Sector Reform (IEG, 2008) Inputs (IEG, 2005)

  • Public Financial Management: good diagnostics,

  • indicators, joint undertaking with govts (PEFA)

  • Tax Administration: strong MoF motivation, good TA

  • Transparency: widened access to information

  • Civil Service and Administrative Reforms: lacks good

  • models, indicators, buttoo important to ignore

  • Government-wide anti-corruption: key are political

  • commitment, strong judiciary

  • Recognize complex political + sequencing issues;

  • focus on basics first

  • Prioritize anti-corruption effort on most harmful aspects

  • Underpin civil service reforms through better diagnosis







Opportunities going forward
Opportunities Going Forward? Inputs (IEG, 2005)

  • Review & adopt good practice standards for training

  • programs based on a realistic results framework

  • Pool evaluation findings to expand practices/lessons

  • Encourage systematic use in decision-making

  • Pool expertise around core training and capacity building management practices

  • Identify case studies of influential training programs

  • Pilot new forms of collaboration

  • Bridge the gap between training and other modes of

  • capacity building

  • New modalities / increased demand for client-led

  • training

  • How training can contribute to development objectives



& Quality


Scope & Scale


Thank you
THANK YOU! Inputs (IEG, 2005)

IEG Website : http://www.worldbank.org/ieg