challenges to institutionalizing impact evaluation
Download
Skip this Video
Download Presentation
Challenges to Institutionalizing Impact Evaluation

Loading in 2 Seconds...

play fullscreen
1 / 39

Challenges to Institutionalizing Impact Evaluation - PowerPoint PPT Presentation


  • 71 Views
  • Uploaded on

Challenges to Institutionalizing Impact Evaluation. Discussion Points * presented at the International Conference on Impact Evaluation Cairo, Egypt March 28- April2, 2009. * By Getahun Tafesse, CIDA –ECCO M&E Advisor. Strengths and Challenges to Institutionalizing Impact Evaluation.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Challenges to Institutionalizing Impact Evaluation' - raine


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
challenges to institutionalizing impact evaluation

Challenges to Institutionalizing Impact Evaluation

Discussion Points* presented at the International Conference on Impact Evaluation

Cairo, Egypt

March 28- April2, 2009

*By Getahun Tafesse, CIDA –ECCO M&E Advisor

strengths and challenges to institutionalizing impact evaluation
Strengths and Challenges to Institutionalizing Impact Evaluation

Opportunities

  • Enormous development challenges in Ethiopia - widespread poverty, chronic food insecurity, HIV/AIDs, illiteracy, environmental degradation etc.
  • This provides fertile ground for evaluation as intervention resources are limited and different stakeholders are pressurized to demonstrate results
  • The Government of Ethiopia (GoE) policy statements express commitment to poverty reduction and sustainable development.
strengths and challenges to institutionalizing impact evaluation1
Strengths and Challenges to Institutionalizing Impact Evaluation

Opportunities

  • Ethiopia has also formally adapted the MDGs as an overarching development guiding framework
  • The existence of well established sectoral annual review mechanisms
  • The pool of M&E professionals in the country is gradually increasing as more and more organizations are currently recruiting experts specifically assigned as such
strengths and challenges to institutionalizing impact evaluation2
Strengths and Challenges to Institutionalizing Impact Evaluation

Opportunities

  • Recent trends show growing recognition and institutionalization of monitoring and evaluation practices:
      • A growing appreciation of evaluation across different stakeholders and particularly encouragement of participatory evaluation at different levels
      • Institutionalization of M&E in non-government circles
      • Increasing conduct of M&E training by organizations
      • Incorporation of M&E courses in some training programs
actors involved in m e

Actors involved in Evaluation

Actors involved in M &E
  • The Government of Ethiopia (GoE)
    • National – MOFED, PMO, NBE
    • Sector ministries
    • Regional and Woreda administrations
  • Donors (through projects/ programs)
    • Bilateral
    • Multilateral
  • INGOs/ NGOs (through projects/ programs)
    • Relief and Development
    • Advocacy
    • Professional associations
  • Academic and Research Institutions
  • The Private Sector/ business community
  • Communities
slide6

Ministry of Finance & Economic Development

Central Statistical Authority

Welfare Monitoring Unit

Sector Ministries

Donors

INGOs/

NGOs

Branch Offices

Regional Bureaus

Programs

Branch Offices

Surveys/

Censuses

Participatory Poverty Assessments

Woreda Desks

Facilities

Projects

Projects

Grass Root Communities

slide7

Welfare Monitoring System in Ethiopia

Ministry of Finance and Economic Development

Chair

National Welfare Monitoring Steering Committee

Ministry of Agriculture and Rural Dev’t

Member

Ministry of Health

Member

RCBP interest could be reflected here

Ministry of Labour and Social Affairs

Member

Ministry of Women Affairs

Member

Ministry of Education

Member

Ethiopian Road Authority

Member

National Welfare Monitoring Technical Advisory Committee

Same as Above

INGOs/ NGOs

Welfare Monitoring Unit

Central Statistical Authority

Data Users

HHICES

Sector Reports

Welfare Monitoring Survey

PPAs

the status quo in impact evaluation
The Status Quo in Impact Evaluation
  • Impact Evaluation is not a common practice
  • Some sorts of Evaluation are practiced mostly to fulfil donor requirements

Can Impact Evaluation meet the sense of urgency that characterize the need for development intervention in low income countries?

  • Frequently development needs require urgent/ immediate assistance
  • Programming in developing countries is geared towards fulfilment of basic needs
  • Choice of area of Intervention is hardly questionable

* Difficult to make the main thrust of development programming as Experimentation

the status quo in impact evaluation1
The Status Quo in Impact Evaluation

Is there national ownership in Impact Evaluation?

  • Impact Evaluation implies there is a preconceived desired State
  • Whose definition of ‘ development ‘ matters
  • If desired state is imposed, ownership to IE is lost
the status quo in impact evaluation2
The Status Quo in Impact Evaluation

Is the Cure for development problems known?

  • Good governance – participation, accountability, transparency…
  • Investment on education, health, agriculture, road…
  • Equity, peace, security…
  • Ownership, partnership, harmonization…

*But the degree of effectiveness of a specific programming in these areas is not easily known

the status quo in impact evaluation3
The Status Quo in Impact Evaluation

The Impact evaluation Dilemma

  • There is sense of urgency for development programming
  • Appropriate cure depends on Impact evaluation/ experimentation

So, the focus of Impact Evaluation in LDCs should be

    • Not on identification of Areas of investment
    • Rather on methods of delivery

The Cure is known (the vaccine is identified). The main question is how to deliver best.

the status quo in impact evaluation4
The Status Quo in Impact Evaluation

Methodological Challenges

How to make the Case of IE strong?

  • Demonstrating practical benefits of Impact Evaluation
  • Resources are scarce – how much to spend on something that is intuitively known is good
  • Flexible and easy IE techniques/ methods that are less costly in terms of time and resources
  • IE aimed at guiding/ improving decision making
    • Show possible alternative use of resources
the status quo in impact evaluation5
The Status Quo in Impact Evaluation

Recommendations

  • Make Impact Evaluation a parallel endeavour/ not a major thrust of development programming – experimentation on a small scale
  • Impact evaluation on key programs without disturbing regular programming, i.e., without creating discontinuity in program
  • Impact Evaluation as a Second Stage Experimentation with the focus being on Approach to delivery, exploring practical alternatives
  • Impact Evaluation in developing countries should have a strong component of comparison element among known strategies
  • From the supply side, effort should be made to expand the availability of easy and flexible tools of IE that are less costly in terms of both time and resources
the status quo in impact evaluation6
The Status Quo in Impact Evaluation

Recommendations

  • Impact Evaluation in developing countries should have a strong component of comparison element among known strategies – not simple and full focus on the specific program that is the subject of evaluation
  • IE should be designed to aide decision making on ‘so what’
  • From the supply side, effort should be made to expand the availability of easy and flexible tools of IE that are less costly in terms of both time and resources
strengths and challenges to institutionalizing impact evaluation3
Strengths and Challenges to Institutionalizing Impact Evaluation

Strengths

  • GoE national Reports have greatly been improved in quality (depth of analyses) and coverage (sectors, sub-sectors, themes) due to
      • Growing demand for and use of such reports
      • DAG financial and technical support extended to MoFED & use of professionals (consultants) for data analyses and report production
      • Inclusion of Governance, foreign Aid, MDGs, Environment, Gender, etc. themes in the reports although not to sufficient degree
      • Implementation of regular household surveys (HHICE, WMS, DHS)
      • Improved capacity and performance on the part of CSA
      • Improved capacity and performance on the part of selected sectoral line ministries
  • Great improvements in sectoral reports, especially Health and Education aided by
      • Sector Management Information Systems
      • Annual Review Mechanisms
      • High political commitment (Health, for example)
      • Improved standardization, rationalization an harmonization of indicators, data collection and reporting procedures
strengths and challenges to institutionalizing impact evaluation4
Strengths and Challenges to Institutionalizing Impact Evaluation

Progresses

  • Strong and growing capacity in statistical data collection
      • CSA’s impressive data collection program
        • Annual and periodical regular surveys
        • Long experience and institutional capacity
  • Civil service reform including business process reengineering (BPR)
      • Streamlined tasks and responsibilities
      • Result oriented work planning
  • Good practice of contracting out data collection and analyses
      • Supported by growing # and capacity of private consultants
strengths and challenges to institutionalizing impact evaluation5
Strengths and Challenges to Institutionalizing Impact Evaluation

Opportunities

  • Strong culture of collaboration by beneficiaries in responding to studies and gradual development in their level of active participation and articulate responses
strengths and challenges to institutionalizing impact evaluation6
Strengths and Challenges to Institutionalizing Impact Evaluation

Opportunities

  • Availability of administrative data by sector ministries (eg. education, health etc), and survey data by Central Statistical Agency
  • Cooperation among stakeholders (government and non-government alike) to share data available
  • Structured societal organizations established down to small community level.
  • Some level of established practice in using evaluation for planning purposes…
strengths and challenges to institutionalizing impact evaluation7
Strengths and Challenges to Institutionalizing Impact Evaluation

Challenges

  • Lack of informed debate on local development perspectives and relevant evaluation conceptual frameworks and approaches; which leads to:
      • Lack of consensus on development concept and measurement criteria
      • Lack of consensus on evaluation concept and criteria
      • Uniform application of evaluation techniques and lack of adoption to specific cultural and behavioral contexts
limitations challenges
Limitations/ Challenges
  • Widespread traditional management practice that gives focus to counting activities and outputs rather than focus to assessing higher level results, i.e., poor result based management practice. This is particularly reflected:
      • In absence of baseline data
      • Poor feedback mechanism
      • Lack of informed decision making or poor linkage between assessment and decision making
strengths and challenges to institutionalizing impact evaluation8
Strengths and Challenges to Institutionalizing Impact Evaluation

Challenges

  • Data collection, analyses and reporting aimed at demonstrating achievements and less focus given to analyses of constraints and challenges
  • Significant discrepancies between administrative and survey data
strengths and challenges to institutionalizing impact evaluation9
Strengths and Challenges to Institutionalizing Impact Evaluation

Limitations

  • Lack of linkage/ integration across different sectoral M&E systems
      • Different timing
      • Different level of reporting
      • duplication
  • sectors at varying stages of capacity and performance in evaluation practice
  • poor practice of verification methods/ triangulation of data from different sources
limitations
Limitations
  • Generating Compelling and evidence based results attributable to programming
  • Large number and varying quality of indicators used in GoE reports and challenges to discern overall progress
  • Missing reference comparisons or lack of standard reference points
  • Missing indicators w.r.t. gender, disadvantaged groups (disabled, destitute, etc.)
limitations challenges1
Limitations/ Challenges
  • Routine data collection at lower level cumbersome and unsystematic
    • Simplifying data collection & ensuring timely use of data
  • The progressive data aggregation at each higher level not necessarily conducive to data analyses
  • No systematic integration of national data collection activities
    • The need to support national strategy for the development of statistics
    • Integration of data collection systems (within & outside sectors)
limitations challenges2
Limitations/ Challenges
  • Poor feed back and linkage with planning and decision making
  • weak data producer and users common forums and linkages
limitations challenges3
Limitations/ Challenges
  • Lack of standardization of survey methods, definition of indicators and measurement tools.
  • Poor recognition given to evaluation importance as reflected by:
      • Poor integration of evaluation approach in programs/ projects design
      • Lack of earmarked budget for M&E
      • Lack of earmarked human resource for M&E in established structures
      • M&E task usually undertaken as an add-on task
limitations1
Limitations
  • Generation of data disaggregated at woreda level
    • Different levels of reporting across different sectors
  • No regular complementary qualitative information (PPA, citizen card, etc.)/ public opinion
limitations challenges4
Limitations/ Challenges
  • Poor Maximizing of benefits from data analyses
    • Gender disaggregated data
    • Thorough/ in depth analyses of data sets
    • Timely analyses of data
  • Inadequate practice and capacity to review and enforce good ethical standards in the undertaking of evaluation
  • Teaching in evaluation not well recognized and integrated in curricula of different disciplines
limitations challenges5
Limitations/ Challenges
  • Poor institutional capacity especially at lower levels
    • Fragile and over loaded
    • No earmarked human and financial resource – M&E is usually an add-on task
    • High staff turn-over
limitations challenges6
Limitations/ Challenges
  • Lack of agency/ home, networks for
      • developing and disseminating knowledge on evaluation approaches, tools, and best practices
      • adoption and popularization of internationally set goals and commitments (ex., MDGs, Conventions, Declarations )
      • adoption and popularization of established methodologies
      • sharing of experiences and exchanging ideas
goe food security m e system
GoE Food Security M&E System
  • The FSCB has an overall responsibility for programme M&E
  • Food Security Programs M&E Plan
      • different stakeholders were involved in the process & many acquired opportunity to comment
      • Simple and Practical
      • 4 Principles were applied:
        • Simplicity –vs- Utility
        • Process –vs- Product/ outcome
        • Decentralization –vs- Accountability
        • Participation –vs- Rigour
goe food security m e system1
GoE Food Security M&E System
  • Food Security Programs M&E Plan
    • Result Frameworks (recently revised based on two 2-day workshops involving GoE & Donors)
    • Monitoring Formats
      • Activity Reports (from community up to federal)
      • Quarterly financial and procurement reports developed by FSCB in accordance with GoE accounting procedures and PSNP Procurement Guidelines.
    • Focal persons responsible at different levels

Training given to focal persons on Monitoring Formats

goe food security m e system2
GoE Food Security M&E System
  • Food Security Programs M&E Plan
    • Programme Description
    • FSP Logical Framework
    • M&E System: Objectives and Approaches
    • Institutional Roles and Information Flow
    • Monitoring Guidelines and Methods
    • Evaluation Guidelines and Methods
    • Human Resource Needs and Training Plan
    • Reporting Formats
goe food security m e system3
GoE Food Security M&E System
  • Monitoring and Evaluation Technical Task Force
    • Comprises members from GoE and Donors financing food security programs
    • Meets every two-weeks
    • Oversees the implementation of the M&E plan
    • Reviews study designs and mobilizes the necessary technical assistance and capacity building resources
goe food security m e system4
GoE Food Security M&E System
  • JCC - meetings every fortnight to discuss and decide on various issues related to the implementation of the program (resource flows to the beneficiary, targeting issues, capacity building and other pertinent issues as they arise).
  • Rapid Response Mechanism field monitoring (usually with both FSCB and donor representatives) undertaken on monthly basis to examine ad hoc issues and constraints as they arise. Team reports identifying issues and recommendations are presented to the JCC for consideration.
goe food security m e system5
GoE Food Security M&E System
  • An Information Centre has been set up and staffed in the FSCB to regularly follow up on problems at woreda and regional levels (particularly with respect to the flow of funds to woredas and payments to beneficiaries).
  • Joint Implementation Support Missions are conducted twice each year (May and October) to review progress with program implementation per se and with related capacity building actions (e.g., financial management, public works, woreda planning and quality control, etc.). The FSCB presents financial and technical progress reports during the missions, as well as reports prepared by the RRM Teams. Additional reports may be produced as required, including reviews of procurement.
goe food security m e system6
GoE Food Security M&E System
  • Surveys and Studies
    • Baseline Survey
    • Annual Survey
    • Public Work Reviews
    • Food Aid Assessments
    • Other specific studies
      • Targeting
      • Institutional Assessment
      • Impact
goe food security m e system7
GoE Food Security M&E System

Other data sources:

  • Annual Agricultural Statistics
    • Crop forecast, Actual crop assessment
    • Woreda level Ethiopian Agricultural Sample Enumeration (EASE)
  • Agricultural Census
  • DPPA Early Warning System
  • Vulnerability Profiles
ad