Challenges to institutionalizing impact evaluation
Download
1 / 39

Challenges to Institutionalizing Impact Evaluation - PowerPoint PPT Presentation


  • 71 Views
  • Uploaded on

Challenges to Institutionalizing Impact Evaluation. Discussion Points * presented at the International Conference on Impact Evaluation Cairo, Egypt March 28- April2, 2009. * By Getahun Tafesse, CIDA –ECCO M&E Advisor. Strengths and Challenges to Institutionalizing Impact Evaluation.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Challenges to Institutionalizing Impact Evaluation' - raine


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Challenges to institutionalizing impact evaluation

Challenges to Institutionalizing Impact Evaluation

Discussion Points* presented at the International Conference on Impact Evaluation

Cairo, Egypt

March 28- April2, 2009

*By Getahun Tafesse, CIDA –ECCO M&E Advisor


Strengths and challenges to institutionalizing impact evaluation
Strengths and Challenges to Institutionalizing Impact Evaluation

Opportunities

  • Enormous development challenges in Ethiopia - widespread poverty, chronic food insecurity, HIV/AIDs, illiteracy, environmental degradation etc.

  • This provides fertile ground for evaluation as intervention resources are limited and different stakeholders are pressurized to demonstrate results

  • The Government of Ethiopia (GoE) policy statements express commitment to poverty reduction and sustainable development.


Strengths and challenges to institutionalizing impact evaluation1
Strengths and Challenges to Institutionalizing Impact Evaluation

Opportunities

  • Ethiopia has also formally adapted the MDGs as an overarching development guiding framework

  • The existence of well established sectoral annual review mechanisms

  • The pool of M&E professionals in the country is gradually increasing as more and more organizations are currently recruiting experts specifically assigned as such


Strengths and challenges to institutionalizing impact evaluation2
Strengths and Challenges to Institutionalizing Impact Evaluation

Opportunities

  • Recent trends show growing recognition and institutionalization of monitoring and evaluation practices:

    • A growing appreciation of evaluation across different stakeholders and particularly encouragement of participatory evaluation at different levels

    • Institutionalization of M&E in non-government circles

    • Increasing conduct of M&E training by organizations

    • Incorporation of M&E courses in some training programs


Actors involved in m e

Actors involved in Evaluation

Actors involved in M &E

  • The Government of Ethiopia (GoE)

    • National – MOFED, PMO, NBE

    • Sector ministries

    • Regional and Woreda administrations

  • Donors (through projects/ programs)

    • Bilateral

    • Multilateral

  • INGOs/ NGOs (through projects/ programs)

    • Relief and Development

    • Advocacy

    • Professional associations

  • Academic and Research Institutions

  • The Private Sector/ business community

  • Communities


Ministry of Finance & Economic Development

Central Statistical Authority

Welfare Monitoring Unit

Sector Ministries

Donors

INGOs/

NGOs

Branch Offices

Regional Bureaus

Programs

Branch Offices

Surveys/

Censuses

Participatory Poverty Assessments

Woreda Desks

Facilities

Projects

Projects

Grass Root Communities


Welfare Monitoring System in Ethiopia

Ministry of Finance and Economic Development

Chair

National Welfare Monitoring Steering Committee

Ministry of Agriculture and Rural Dev’t

Member

Ministry of Health

Member

RCBP interest could be reflected here

Ministry of Labour and Social Affairs

Member

Ministry of Women Affairs

Member

Ministry of Education

Member

Ethiopian Road Authority

Member

National Welfare Monitoring Technical Advisory Committee

Same as Above

INGOs/ NGOs

Welfare Monitoring Unit

Central Statistical Authority

Data Users

HHICES

Sector Reports

Welfare Monitoring Survey

PPAs


The status quo in impact evaluation
The Status Quo in Impact Evaluation

  • Impact Evaluation is not a common practice

  • Some sorts of Evaluation are practiced mostly to fulfil donor requirements

    Can Impact Evaluation meet the sense of urgency that characterize the need for development intervention in low income countries?

  • Frequently development needs require urgent/ immediate assistance

  • Programming in developing countries is geared towards fulfilment of basic needs

  • Choice of area of Intervention is hardly questionable

    * Difficult to make the main thrust of development programming as Experimentation


The status quo in impact evaluation1
The Status Quo in Impact Evaluation

Is there national ownership in Impact Evaluation?

  • Impact Evaluation implies there is a preconceived desired State

  • Whose definition of ‘ development ‘ matters

  • If desired state is imposed, ownership to IE is lost


The status quo in impact evaluation2
The Status Quo in Impact Evaluation

Is the Cure for development problems known?

  • Good governance – participation, accountability, transparency…

  • Investment on education, health, agriculture, road…

  • Equity, peace, security…

  • Ownership, partnership, harmonization…

    *But the degree of effectiveness of a specific programming in these areas is not easily known


The status quo in impact evaluation3
The Status Quo in Impact Evaluation

The Impact evaluation Dilemma

  • There is sense of urgency for development programming

  • Appropriate cure depends on Impact evaluation/ experimentation

    So, the focus of Impact Evaluation in LDCs should be

    • Not on identification of Areas of investment

    • Rather on methods of delivery

      The Cure is known (the vaccine is identified). The main question is how to deliver best.


The status quo in impact evaluation4
The Status Quo in Impact Evaluation

Methodological Challenges

How to make the Case of IE strong?

  • Demonstrating practical benefits of Impact Evaluation

  • Resources are scarce – how much to spend on something that is intuitively known is good

  • Flexible and easy IE techniques/ methods that are less costly in terms of time and resources

  • IE aimed at guiding/ improving decision making

    • Show possible alternative use of resources


The status quo in impact evaluation5
The Status Quo in Impact Evaluation

Recommendations

  • Make Impact Evaluation a parallel endeavour/ not a major thrust of development programming – experimentation on a small scale

  • Impact evaluation on key programs without disturbing regular programming, i.e., without creating discontinuity in program

  • Impact Evaluation as a Second Stage Experimentation with the focus being on Approach to delivery, exploring practical alternatives

  • Impact Evaluation in developing countries should have a strong component of comparison element among known strategies

  • From the supply side, effort should be made to expand the availability of easy and flexible tools of IE that are less costly in terms of both time and resources


The status quo in impact evaluation6
The Status Quo in Impact Evaluation

Recommendations

  • Impact Evaluation in developing countries should have a strong component of comparison element among known strategies – not simple and full focus on the specific program that is the subject of evaluation

  • IE should be designed to aide decision making on ‘so what’

  • From the supply side, effort should be made to expand the availability of easy and flexible tools of IE that are less costly in terms of both time and resources


Strengths and challenges to institutionalizing impact evaluation3
Strengths and Challenges to Institutionalizing Impact Evaluation

Strengths

  • GoE national Reports have greatly been improved in quality (depth of analyses) and coverage (sectors, sub-sectors, themes) due to

    • Growing demand for and use of such reports

    • DAG financial and technical support extended to MoFED & use of professionals (consultants) for data analyses and report production

    • Inclusion of Governance, foreign Aid, MDGs, Environment, Gender, etc. themes in the reports although not to sufficient degree

    • Implementation of regular household surveys (HHICE, WMS, DHS)

    • Improved capacity and performance on the part of CSA

    • Improved capacity and performance on the part of selected sectoral line ministries

  • Great improvements in sectoral reports, especially Health and Education aided by

    • Sector Management Information Systems

    • Annual Review Mechanisms

    • High political commitment (Health, for example)

    • Improved standardization, rationalization an harmonization of indicators, data collection and reporting procedures


  • Strengths and challenges to institutionalizing impact evaluation4
    Strengths and Challenges to Institutionalizing Impact Evaluation

    Progresses

    • Strong and growing capacity in statistical data collection

      • CSA’s impressive data collection program

        • Annual and periodical regular surveys

        • Long experience and institutional capacity

  • Civil service reform including business process reengineering (BPR)

    • Streamlined tasks and responsibilities

    • Result oriented work planning

  • Good practice of contracting out data collection and analyses

    • Supported by growing # and capacity of private consultants


  • Strengths and challenges to institutionalizing impact evaluation5
    Strengths and Challenges to Institutionalizing Impact Evaluation

    Opportunities

    • Strong culture of collaboration by beneficiaries in responding to studies and gradual development in their level of active participation and articulate responses


    Strengths and challenges to institutionalizing impact evaluation6
    Strengths and Challenges to Institutionalizing Impact Evaluation

    Opportunities

    • Availability of administrative data by sector ministries (eg. education, health etc), and survey data by Central Statistical Agency

    • Cooperation among stakeholders (government and non-government alike) to share data available

    • Structured societal organizations established down to small community level.

    • Some level of established practice in using evaluation for planning purposes…


    Strengths and challenges to institutionalizing impact evaluation7
    Strengths and Challenges to Institutionalizing Impact Evaluation

    Challenges

    • Lack of informed debate on local development perspectives and relevant evaluation conceptual frameworks and approaches; which leads to:

      • Lack of consensus on development concept and measurement criteria

      • Lack of consensus on evaluation concept and criteria

      • Uniform application of evaluation techniques and lack of adoption to specific cultural and behavioral contexts


    Limitations challenges
    Limitations/ Challenges

    • Widespread traditional management practice that gives focus to counting activities and outputs rather than focus to assessing higher level results, i.e., poor result based management practice. This is particularly reflected:

      • In absence of baseline data

      • Poor feedback mechanism

      • Lack of informed decision making or poor linkage between assessment and decision making


    Strengths and challenges to institutionalizing impact evaluation8
    Strengths and Challenges to Institutionalizing Impact Evaluation

    Challenges

    • Data collection, analyses and reporting aimed at demonstrating achievements and less focus given to analyses of constraints and challenges

    • Significant discrepancies between administrative and survey data


    Strengths and challenges to institutionalizing impact evaluation9
    Strengths and Challenges to Institutionalizing Impact Evaluation

    Limitations

    • Lack of linkage/ integration across different sectoral M&E systems

      • Different timing

      • Different level of reporting

      • duplication

  • sectors at varying stages of capacity and performance in evaluation practice

  • poor practice of verification methods/ triangulation of data from different sources


  • Limitations
    Limitations

    • Generating Compelling and evidence based results attributable to programming

    • Large number and varying quality of indicators used in GoE reports and challenges to discern overall progress

    • Missing reference comparisons or lack of standard reference points

    • Missing indicators w.r.t. gender, disadvantaged groups (disabled, destitute, etc.)


    Limitations challenges1
    Limitations/ Challenges

    • Routine data collection at lower level cumbersome and unsystematic

      • Simplifying data collection & ensuring timely use of data

    • The progressive data aggregation at each higher level not necessarily conducive to data analyses

    • No systematic integration of national data collection activities

      • The need to support national strategy for the development of statistics

      • Integration of data collection systems (within & outside sectors)


    Limitations challenges2
    Limitations/ Challenges

    • Poor feed back and linkage with planning and decision making

    • weak data producer and users common forums and linkages


    Limitations challenges3
    Limitations/ Challenges

    • Lack of standardization of survey methods, definition of indicators and measurement tools.

    • Poor recognition given to evaluation importance as reflected by:

      • Poor integration of evaluation approach in programs/ projects design

      • Lack of earmarked budget for M&E

      • Lack of earmarked human resource for M&E in established structures

      • M&E task usually undertaken as an add-on task


    Limitations1
    Limitations

    • Generation of data disaggregated at woreda level

      • Different levels of reporting across different sectors

    • No regular complementary qualitative information (PPA, citizen card, etc.)/ public opinion


    Limitations challenges4
    Limitations/ Challenges

    • Poor Maximizing of benefits from data analyses

      • Gender disaggregated data

      • Thorough/ in depth analyses of data sets

      • Timely analyses of data

    • Inadequate practice and capacity to review and enforce good ethical standards in the undertaking of evaluation

    • Teaching in evaluation not well recognized and integrated in curricula of different disciplines


    Limitations challenges5
    Limitations/ Challenges

    • Poor institutional capacity especially at lower levels

      • Fragile and over loaded

      • No earmarked human and financial resource – M&E is usually an add-on task

      • High staff turn-over


    Limitations challenges6
    Limitations/ Challenges

    • Lack of agency/ home, networks for

      • developing and disseminating knowledge on evaluation approaches, tools, and best practices

      • adoption and popularization of internationally set goals and commitments (ex., MDGs, Conventions, Declarations )

      • adoption and popularization of established methodologies

      • sharing of experiences and exchanging ideas


    Food security monitoring evaluation
    Food SecurityMonitoring & Evaluation


    Goe food security m e system
    GoE Food Security M&E System

    • The FSCB has an overall responsibility for programme M&E

    • Food Security Programs M&E Plan

      • different stakeholders were involved in the process & many acquired opportunity to comment

      • Simple and Practical

      • 4 Principles were applied:

        • Simplicity –vs- Utility

        • Process –vs- Product/ outcome

        • Decentralization –vs- Accountability

        • Participation –vs- Rigour


    Goe food security m e system1
    GoE Food Security M&E System

    • Food Security Programs M&E Plan

      • Result Frameworks (recently revised based on two 2-day workshops involving GoE & Donors)

      • Monitoring Formats

        • Activity Reports (from community up to federal)

        • Quarterly financial and procurement reports developed by FSCB in accordance with GoE accounting procedures and PSNP Procurement Guidelines.

      • Focal persons responsible at different levels

        Training given to focal persons on Monitoring Formats


    Goe food security m e system2
    GoE Food Security M&E System

    • Food Security Programs M&E Plan

      • Programme Description

      • FSP Logical Framework

      • M&E System: Objectives and Approaches

      • Institutional Roles and Information Flow

      • Monitoring Guidelines and Methods

      • Evaluation Guidelines and Methods

      • Human Resource Needs and Training Plan

      • Reporting Formats


    Goe food security m e system3
    GoE Food Security M&E System

    • Monitoring and Evaluation Technical Task Force

      • Comprises members from GoE and Donors financing food security programs

      • Meets every two-weeks

      • Oversees the implementation of the M&E plan

      • Reviews study designs and mobilizes the necessary technical assistance and capacity building resources


    Goe food security m e system4
    GoE Food Security M&E System

    • JCC - meetings every fortnight to discuss and decide on various issues related to the implementation of the program (resource flows to the beneficiary, targeting issues, capacity building and other pertinent issues as they arise).

    • Rapid Response Mechanism field monitoring (usually with both FSCB and donor representatives) undertaken on monthly basis to examine ad hoc issues and constraints as they arise. Team reports identifying issues and recommendations are presented to the JCC for consideration.


    Goe food security m e system5
    GoE Food Security M&E System

    • An Information Centre has been set up and staffed in the FSCB to regularly follow up on problems at woreda and regional levels (particularly with respect to the flow of funds to woredas and payments to beneficiaries).

    • Joint Implementation Support Missions are conducted twice each year (May and October) to review progress with program implementation per se and with related capacity building actions (e.g., financial management, public works, woreda planning and quality control, etc.). The FSCB presents financial and technical progress reports during the missions, as well as reports prepared by the RRM Teams. Additional reports may be produced as required, including reviews of procurement.


    Goe food security m e system6
    GoE Food Security M&E System

    • Surveys and Studies

      • Baseline Survey

      • Annual Survey

      • Public Work Reviews

      • Food Aid Assessments

      • Other specific studies

        • Targeting

        • Institutional Assessment

        • Impact


    Goe food security m e system7
    GoE Food Security M&E System

    Other data sources:

    • Annual Agricultural Statistics

      • Crop forecast, Actual crop assessment

      • Woreda level Ethiopian Agricultural Sample Enumeration (EASE)

    • Agricultural Census

    • DPPA Early Warning System

    • Vulnerability Profiles


    ad