Evaluation methods training and capacity building programs
This presentation is the property of its rightful owner.
Sponsored Links
1 / 21

Evaluation Methods Training and Capacity Building Programs PowerPoint PPT Presentation


  • 121 Views
  • Uploaded on
  • Presentation posted in: General

Evaluation Methods Training and Capacity Building Programs. Nidhi Khattri Independent Evaluation Group November 17, 2008. IEG ’ s Mandate. The World Bank’s independent evaluation function established about 35 years ago The goals: learning from experience

Download Presentation

Evaluation Methods Training and Capacity Building Programs

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Evaluation methods training and capacity building programs

Evaluation MethodsTraining and Capacity Building Programs

Nidhi Khattri

Independent Evaluation Group

November 17, 2008


Ieg s mandate

IEG’s Mandate

  • The World Bank’s independent evaluation function established about 35 years ago

  • The goals:

    • learning from experience

    • accountability for the achievement of objectives


Ieg s independence direct report to the board

IEG’s Independence: Direct Report to the Board

  • Direct reporting to the Board of Executive Directors

  • Headed by a Director-General, Evaluation (DGE)

    • Appointed by the Board

    • No World Bank Group position after current position

  • Evaluations to the Committee on Development Effectiveness (CODE)

  • Evaluation content not negotiated with CODE/Board


Ieg s links to bank management

IEG’s Links to Bank Management

  • Bank management has opportunity to comment

  • Draft Bank Management Response accompanies evaluation

  • IEG responds to Bank management comments at Board meetings

  • IEG’ Management Action Record (MAR) reports on management’s progress on actions noted in management response


Ieg s evaluation products

IEG’s Evaluation Products

  • Project Evaluations

    • Project assessments (ICR Reviews, PPARs)

    • Impact studies (e.g., Bangladesh Health, Ghana Education)

  • Sector and Thematic Evaluations

    • Often linked to policy revision (e.g., forestry – altered sector policy)

  • Country Evaluations

    • Country Assistance Evaluations

    • Country Impact Reviews (IEG-IFC)

    • Reviews of CAS Completion Reports

  • Global and Regional Program Evaluations

  • Corporate Evaluations

    • Annual Review of Development Effectiveness (which now includes the Annual Report on Operations Evaluation)


Evaluation approaches

Evaluation Approaches

  • Based on evaluation products

  • Primarily Objectives Based for Projects and Programs

    • Outcome

    • Risk to Development Outcome

    • Bank Performance

    • Borrower Performance

    • Monitoring and Evaluation (M&E) Quality


Outcome

Outcome

  • The extent to which the operation’s major relevant objectives were achieved, or are expected to be achieved, efficiently

  • Outcome = Relevance + Efficacy + Efficiency


Bank performance

Bank Performance

  • The extent to which services provided by the Bank ensured quality at entry of the operation and supported effective implementation through appropriate supervision

  • Bank Performance = Quality at Entry + Quality of Supervision


Monitoring and evaluation m e quality

Monitoring and Evaluation (M&E) Quality

  • M&E design—the extent to which the project was designed to collect appropriate (input, output, outcome, and impact) data given project objectives and given already available data

  • M&E implementation—the extent to which appropriate data was actually collected using appropriate collection methods (to ensure data quality)

  • M&E utilization—the extent to which appropriate data was used to inform decision-making and resource allocation


Corporate evaluation annual review of development effectiveness arde

Corporate Evaluation: Annual Review of Development Effectiveness (ARDE)

  • Annual meta-evaluations that provide a comprehensive assessment of the Bank’s development effectiveness

  • Draw on IEG’s recent project, sector, thematic, country, and global evaluations

  • Synthesize lessons that can be used to increase the development effectiveness of World Bank assistance

  • Highlight the findings of recent IEG evaluations around a common theme


Recent ieg evaluations around training and capacity building

Recent IEG Evaluations around Training and Capacity Building

  • Using Training to Build Capacity (2007)

    • Avg amount of client training estimated at $720 mio per year (90% through projects, rest through WBI)

    • Key component in 60% of investment projects, particularly in social, rural, public sectors

    • 37 (incl. 8 WBI) training programs (Bangladesh, Burkina Faso, Mexico, Tunisia), 6 country surveys of 550 trainees, comparison with other DTIs, etc.

  • Capacity Building in Africa (2005)

    • 25% of Bank lending to Africa; $9 bill between 1995-2004

  • Public Sector Reform (2008)

    • 1/6 of Bank lending and advisory support – and increasing


Methodology in evaluation of training

Methodology in Evaluation of Training

  • Key Questions:

    • To what extent did Bank-financed training have an impact on the capacity of target organizations?

    • What factors contribute to successful training?

    • To what extent are such factors present in Bank-financed training?

  • Main Methods – using objectives-based methodology

    • Survey of training participants

    • Survey of training institutions

    • In-depth field reviews


Training results in capacity building only when certain conditions are met

Training Results in Capacity Building Only When Certain Conditions are Met

Input

Training

OutputLearning

OutcomeWorkplace behavior change

ImpactEnhanced institutional or organizational capacity


Effectiveness of training what is the evidence

Effectiveness of Training: What is the Evidence?

  • Most training resulted in demonstrable learning

  • But: Individual learning gains poor predictor of impact

  • Project-based trained lacked basic results measures

Outputs

Performance

Outcomes

  • About half of trainees surveyed reported substantial

  • positive changes in work performance

Impact on

Capacity

  • 10 project-based programs had significant impact (e.g.,

  • Procurement reform, Community Groups, Exporters, SME)

  • Best combined project funds + outside expertise

  • WBI programs not rated due to lack of data

Other

Findings

  • Type of training provider not correlated with success

  • Good training outcomes in both higher- and low-

  • capacity environments


Training design what works

Training Design: What Works?

  • Adequate diagnosis of capacity gaps associated with

  • strong client commitment / involvement

  • Training needs assessment the norm in highly-rated

  • programs, but often subject to funding constraints (WBI)

  • Good participant selection requires engagement and

  • supervision (e.g., IMF Institute, MASHAV)

  • Poor targeting most important cause for lack of impact

  • Generally high marks for design & teaching standards

  • But course length and topic coverage needs to be better matched with capacity building goals (ITCILO,JICA)

  • Allow time for practical learning techniques / action plans (InWent, JICA, MASHAV)

  • Provide systematic follow-on support (Motorola, InWent)

Targeting

of Training

Content

Pedagogy

Workplace

Transfer


When training works what matters most

When Training Works: What Matters Most?

  • Support by managers and peers is key driver for

  • successful workplace implementation (~90% feedback)

  • But: 1/3 of trainees didn’t have adequate resources,

  • incentives or org. support to apply what they learned

  • Focus needs assessment on organize. bottlenecks and

  • whether training is indeed the right tool.

  • Lack of institutional incentives recurring problem in civil

  • service and highly decentralized training programs;

  • Stronger incentives at work in training programs for

  • community and farmer groups, private sector firms

  • Hi-level support for training key for workplace transfer

  • Stand-alone training with limited ability for diagnosis,

  • dialogue, influence and follow-up

Workplace

Context

Incentives

Client

Commitment


Key internal inhibitors for effective training

Key Internal Inhibitors for Effective Training?

  • At design, most training programs fail to specify training

  • objectives and expected performance outcomes,

  • reflecting lack of broader capacity needs assessments

  • At completion, performance evaluation rarely done

  • No feedback / accountability loop >> no improvements

  • Lack of established standards for training design and

  • implementation undercuts quality assurance;

  • Team leaders for project-based training lack adequate

  • in-house support and voiced demand for more

  • WBI earns high marks for Country Team consultations,

  • but collaboration at task level remains rare. Risk of

  • diffusion of program.

Results

Orientation

Standards

Access to

Expertise

Alignment


Persistent capacity gaps in africa despite substantial inputs ieg 2005

Persistent Capacity Gaps in Africa Despite Substantial Inputs (IEG, 2005)

  • 40% of sampled lending operations achieved CB goals, with

  • better outcomes in roads than in health / education.

  • CB lacked clear results framework (only 1/3 of projects clear about relationships among individual, organizational. and institutional aspects of capacity)

  • Weak diagnostics of pol. econ and available country capacity

  • High fragmentation of efforts; supply driven TA

  • Training not embedded in broader HR strategies;

  • Strengthen K-base, operational framework, M&E

  • Develop sector-specific guidance

  • Promote country-led approach

  • Re-assess role / modalities of training

  • Capacity Building has moved to center stage in AFR CASs

  • Focus shifting beyond individual skills to institutional support

  • leadership, donor harmonization and better coherence

Key Finding

Obstacles

Recommended

Actions

Recent

Changes


Mixed outcomes on public sector reform ieg 2008

Mixed Outcomes on Public Sector Reform (IEG, 2008)

  • Public Financial Management: good diagnostics,

  • indicators, joint undertaking with govts (PEFA)

  • Tax Administration: strong MoF motivation, good TA

  • Transparency: widened access to information

  • Civil Service and Administrative Reforms: lacks good

  • models, indicators, buttoo important to ignore

  • Government-wide anti-corruption: key are political

  • commitment, strong judiciary

  • Recognize complex political + sequencing issues;

  • focus on basics first

  • Prioritize anti-corruption effort on most harmful aspects

  • Underpin civil service reforms through better diagnosis

Successful

Themes

Reform

Challenges

Recommended

Actions


Opportunities going forward

Opportunities Going Forward?

  • Review & adopt good practice standards for training

  • programs based on a realistic results framework

  • Pool evaluation findings to expand practices/lessons

  • Encourage systematic use in decision-making

  • Pool expertise around core training and capacity building management practices

  • Identify case studies of influential training programs

  • Pilot new forms of collaboration

  • Bridge the gap between training and other modes of

  • capacity building

  • New modalities / increased demand for client-led

  • training

  • How training can contribute to development objectives

Guidelines

Learning

& Quality

Awareness

Scope & Scale

Frontier


Thank you

THANK YOU!

IEG Website : http://www.worldbank.org/ieg


  • Login