monitoring evaluation in nie n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Monitoring & Evaluation in NIE PowerPoint Presentation
Download Presentation
Monitoring & Evaluation in NIE

Loading in 2 Seconds...

play fullscreen
1 / 70

Monitoring & Evaluation in NIE - PowerPoint PPT Presentation


  • 107 Views
  • Uploaded on

Module 20. Monitoring & Evaluation in NIE . Learning objectives. Be familiar with the basic concepts and main characteristics of monitoring and evaluation Understand the differences between various kinds of evaluations Explain the different kinds of indicators

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Monitoring & Evaluation in NIE' - teo


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
learning objectives
Learning objectives
  • Be familiar with the basic concepts and main characteristics of monitoring and evaluation
  • Understand the differences between various kinds of evaluations
  • Explain the different kinds of indicators
  • Describe the very basics of a ‘log frame’
  • Optional: Be familiar with the monitoring and evaluation of CMAM interventions
the project cycle

Disaster

The project cycle

ASSESSMENT

EVALUATION

Monitoring

PROGRAMME DESIGN

IMPLEMENTATION

slide6
M&E

performance

efficiency

outputs

A WASP NEST………?

effectiveness

appropriateness

outcomes

Quantitative indicators

Qualitative indicators

target

Logframes

impact

assessment

DO NO HARM

coverage

INPUTS

connectedness

accountability

timeliness

*

definition
Definition

Monitoring

‘The systematic and continuous assessment of the

progress of a piece of work over time….’

‘To continuously measure progress against programme

objectives and check on relevance of the programme’

It involves collecting and analysing data/information

It is NOT only about PROCESS

*

purpose of monitoring
Purpose of monitoring
  • to document progress and results of project
  • to provide the necessary information to Management for timely decision taking and corrective action (if necessary)
  • to promoteaccountability* to all stakeholders of a project (to beneficiaries, donors, etc)

*

information collected for monitoring
Information collected for monitoring

must be:

  • Useful and relevant
  • Accurate
  • Regular
  • Acted upon
  • Shared
  • Timely

*

slide10
Monitoring is an implicit part of an evaluation.

It is often done badly:

  • Routine data collection not done routinely!
  • Data collection done poorly
  • Information not processed/used in a timely manner
  • Focus only on process indicators and neglecting (lack of) preliminary impact

*

can you give examples of monitoring in your current work
Can you give examples of Monitoring in your current work?

For example

- From a CMAM programme?

  • From a Micronutrient programme?
  • From a General Food Distribution?
  • From a Health programme?
  • From a Livelihoods programme?

*

monitoring
Monitoring

Monitoring compares intentions with results

It guides project revisions, verifies targeting criteria and whether assistance is reaching the people intended.

It checks the relevance of the project to the needs.

It integrates and responds to community feedback

It enhances transparency and accountability

difference between
Difference between

Monitoring of

  • Process/activities
  • Impact/results

*

the project cycle1

Disaster

The project cycle

ASSESSMENT

EVALUATION

Monitoring

PROGRAMME DESIGN

IMPLEMENTATION

*

definitions
Definitions

Evaluation

The aim is to determine relevance and fulfilment of

objectives, as well as efficiency, effectiveness, impact

and sustainability of a project.

It involves the objective assessment of an

ongoing or completed project/programme, its

design, implementation and results.

*

slide17
There has been an increased

focus on evaluation of

humanitarian action as part

of efforts to improve quality and standards

*

evaluation
Evaluation

It aims to

  • Improve policy and practice
  • Enhance accountability

*

evaluations are done when because
Evaluations are done when / because:
  • Monitoring highlightsunexpected results
  • More information is needed for decision making
  • Implementation problemsor unmet needsare identified
  • Issues of sustainability, cost effectiveness or relevance arise
  • Recommendations for actions to improve performance are needed
  • Lessons learningare necessary for future activities
evaluations
Evaluations
  • Evaluation involves the same skills as assessment and analysis
  • Evaluation should be done impartially and ideally by externalstaff
  • Evaluation can also occur during (e.g. mid-term) and after implementation of the project

Why?

One of the most important sources

of information for evaluations is

data used for monitoring

*

the oecd dac criteria organisation for economic co operation and development
The OECD-DAC criteriaOrganisation for Economic Co-operation and Development
  • The Development Assistance Committee (DAC) evaluation criteria are currently at the heart of the evaluation of humanitarian action.
  • The DAC criteria are designed to improve evaluation of humanitarian action.

*

slide22

Evaluation looks at

  • Relevance/Appropriateness: Doing the right thing in the right way at the right time.
  • Connectedness (and coordination): Was there any replication or gaps left in programming due to a lack of coordination?
  • Coherence: Did the intervention make sense in the context of the emergency and the mandate of the implementing agency? Are their detrimental effects of the intervention on long run?
  • Coverage: Who has been reached by the intervention, and where: linked to effectiveness?
  • Efficiency:Were the results delivered in the least costly manner possible?
  • Effectiveness:To what extent has the intervention achieved its objectives?
  • Impact: Doing the right thing, changing the situation more profoundly and in the longer-term.

*

slide23

Evaluation looks at

  • Relevance/Appropriateness: Doing the right thing in the right way at the right time.
  • Connectedness (and coordination): Was there any replication or gaps left in programming due to a lack of coordination?
  • Coherence: Did the intervention make sense in the context of the emergency and the mandate of the implementing agency? Are their detrimental effects of the intervention on long run?
  • Coverage: Who has been reached by the intervention, and where: linked to effectiveness?
  • Efficiency:The extent to which results have been delivered in the least costly manner possible.
  • Effectiveness:The extent to which an intervention has achieved its objectives –
  • Impact: Doing the right thing, changing the situation more profoundly and in the longer-term.

*

slide24
Example on General Food Distribution
  • Relevance/Appropriateness: Doing the right thing in the right way at the right time.

Was food aid the right thing to do, not cash?

  • Connectedness: Are their detrimental effects of the intervention on long run?

Did food aid lower food prices? Did local farmers suffer from that?

slide25
Coverage: Who has been reached by the intervention, and where: linked to effectiveness?

Were those that needed food aid indeed reached?

  • Efficiency: Were the results delivered in the least costly manner possible?

Was it right to import the food or should it have been purchased locally? Could the results have been achieved with less (financial) resources? Food aid was provided, would cash have been more cost-effective?

slide26
Effectiveness:To what extent has the intervention achieved its objectives?

Did food aid avoid undernutrition? (assuming it was an objective)

  • Impact: Doing the right thing, changing the situation more profoundly and in the longer-term.

Did the food aid avoid people becoming displaced? Did the people become dependent on food aid?

slide27
Impact:
  • Very much related to the general goal of the project
  • Measures both positive and negative long-termeffects, as well as intended and unintended effects.

GFD: did it lower general food prices with long-term economic consequences for certain groups ? Were people that received food aid attacked because of the ration? (therefore more death…?)

  • Need for baseline information!!!!

(to measure results against….)

m e in emergencies
M&E in emergencies?

YES

Any project without Monitoring and/or Evaluation is a BAD project

*

evaluations in humanitarian context
Evaluations in Humanitarian Context
  • Single-agency evaluation (during/after project)
  • There is an increasing move towards:
    • Inter-agency evaluations: the objective is to evaluate responses as a whole and the links between interventions
    • Real-time evaluations: carried out 8 to 12 weeks after the onset of an emergency and are processed within one month of data collection
real time evaluations 1
Real-time evaluations (1)
  • WHY?

Arose from concern that evaluations came too late to affect the operations they were assessing

  • Various groups of organizations aim to undertake real-time evaluations
  • Same purpose as any other evaluation
  • Common characteristics:
    • Takes place during the course of implementation
    • In a short time frame

*

real time evaluations 2
Real-time evaluations (2)
  • It is an improvement-oriented review; it can be regarded more as an internal function than an external process.
  • It helps to bring about changes in the programme, rather than just reflecting on its quality after the event.
  • A real-time “evaluator” is a “facilitator”, working with staff to find creative solutions to any difficulties they encounter.
  • It helps to get closer to the people affected by crisis, and this enables to improve accountability to ‘beneficiaries’.

*

monitoring evaluation systems
Monitoring & Evaluation systems
  • Main components of M&E systems:
    • M&E work plan for data collection and analysis, covering baseline, on-going M&E
    • Logical framework, including indicators and means/source of verification
    • Reporting flows and formats
    • Feedback and review plan
    • Capacity building design
    • Implementation schedule
    • Human resources and budget
indicators
Indicators
  • An indicator is a measure that is used to show change in a situation, or the progress in/results of an activity, project, or programme.
  • Indicators:
    • enable us to be “watchdogs”;
    • are essential instruments for monitoring and evaluation.
    • are objectivelyverifiable measurements
what are the qualities of a good indicator
What are the Qualities of a Good Indicator?
  • Specific
  • Measurable
  • Achievable
  • Relevant
  • Time-bound

The Sphere Project provides the most

accepted indicators for nutrition and

food security interventions in emergencies: see Module 21.

And there is also the SMART initiative….

Standardised Monitoring and Assessment in Relief and Transition

Initiative - interagency initiative to improve the M&E of humanitarian

assistance

types of indicators
Types of indicators

Indicators exist in many different forms:

Examples?

  • Direct indicators correspond precisely to results at any performance level.
  • Indirect or "proxy" indicators demonstrate the change or results if direct measures are not feasible.

Direct

Indirect / proxy

  • Indicators are usually quantitative measures, expressed as percentage or share, as a rate, etc.
  • Indicators may also be qualitative observations.

Qualitative

Quantitative

Global / standardised

  • Standardised global indicators are comparable in all settings.
  • Other indicators tend to be context specific and must be developed locally.

Locally developed

slide41

Impact

Outcome

Output

Input

slide42

Impact

Related to Goal

Outcome

Related to Objectives (or Purposes)

Output

Related to Outputs

Input

Related to Activities/Resources

slide43

Impact

Malnutrition rates amongst young

children reduced

Related to Goal

% of young children

getting appropriate

complementary food

Outcome

Related to Objectives (or Purposes)

X number of mothers

know about good

complementary food and

how to prepare that

Output

Related to Outputs

Nutritional education to mothers on complementary food

Input

Related to Activities/Resources

*

what is a log frame
What is a Log Frame?

The logical framework or logframe is an analytical tool

used to plan, monitor, and evaluateprojects.

?

?

?

?

Victim of a log frame?

log frames
Log Frames

IMPACT

OUTCOME

INPUTS

slide47

?

?

?

?

Impact

?

Outcome

Outcome

Outcome

?

Output

Output

Output

Output

Output

Output

?

?

?

?

?

?

Impact

?

Outcome

Output

?

Output

Output

?

INPUTS

?

?

slide48

Other terms that can be found in a logframe:

The means of verificationof progress towards achieving the indicators highlights the sources from where data is collected. The process of identifying the means of verification at this stage is useful as discussions on where to find information or how to collect it often lead to reformulation of the indicator.

Assumptions are external factors or conditions that have the potential to influence the success of a programme. They may be factors outside the control of the programme. The achievement of a programme’s aims depends on whether or not assumptions hold true or anticipated risks do not materialise.

6-Jun-14

48

logical framework for m e
logical framework for M&E

If the OBJECTIVES are produced, then this should contribute to the overall GOAL

If OUTPUTS/RESULTS are produced, then the OBJECTIVES are accomplished

If adequate ACTIVITIES are conducted, then OUTPUT/RESULTS can be produced

If adequate RESOURCES/INPUTS are provided; then activities can be conducted

activities versus results
Activities versus Results

Completed activities are not results.

  • e.g. a hospital was built, does not mean that injured and sick people can be treated in the hospital, maybe the hospital has no water and the beds have not been delivered.

Results are the actual benefits or effects of

completed activities:

  • e.g. Injured and sick people have access to a fully functional health facility.

*

key messages
Key messages
  • The monitoring of nutrition interventions in emergencies is an integral part of saving lives and maintaining nutrition status of the affected population.
  • Successful monitoring systems allow for improvements in interventions in ‘real time’.
  • Evaluations are important tools for learning, assessing interventions, comparing the costs of the interventions and their impact. Essential evaluation parameters are: effectiveness; efficiency; relevance/appropriateness; impact and coverage
  • Involving communities in M&E places the affected population at the heart of the response, providing the opportunity for their views and perceptions to be incorporated into programme decisions and increases accountability towards them.
  • A common mistake of designing M&E systems is creating a framework which is overly complex. Always make an M&E system practical and doable.
  • The logical framework or logframe is an analytical toolused to plan, monitor, and evaluateprojects.
monitoring for cmam interventions
Monitoring for CMAM interventions
  • Types of monitoring, e.g.
    • Individual case monitoring,
    • Programme / activities monitoring
individual monitoring for cmam
Individual monitoring for CMAM
  • It is the basic follow up of cases in SFP / OTP / SC services:
    • Anthropometric / clinical assessment
  • Tools for individual case follow up include:
    • Medical / nutrition and action protocols
    • Individual follow up card
    • Referral forms
objectives of monitoring cmam activities
Objectives of monitoring CMAM activities
  • Assess service performance / outcomes
  • Identify further needs
    • Support decision-taking for quality improvement (staffing, training, resources, site location,…)
  • Contribute to the analysis of the general situation
    • Assessing the nutrition trends in the area
methods and tools for monitoring cmam interventions
Methods and tools for monitoring CMAM interventions
  • Monthly / weekly reporting:
      • Reporting needs to be done per site (service unit) and compiled per area (district…) up to the national level
  • Routine supervision
  • External evaluations
    • Coverage surveys are one of the most important tools for evaluation of CMAM interventions
routine data collection for monitoring cmam interventions
Routine data collection for monitoring CMAM interventions
  • Routine data is collected for specified time-periods:
    • Nb. of new admissions ,
    • Nb. of discharges (total and by category: cured, died, defaulted, non-recovered
    • Nb. of cases in treatment (nb. of beneficiaries registered at the end of the reporting time-period)

Data on admissions should be disaggregated by gender

monitoring of cmam interventions key indicators for sam sphere
Monitoring of CMAM interventions: key indicators for SAM (Sphere)
  • The proportion of discharges from therapeutic care should be:
    • Recovered > 75 %
    • Deaths < 10 %
    • Defaulter < 15 %

They are primarily applicable to the 6–59 month age group, although others may be part of the programme.

  • Distance: > 90 % of the target population is within less than one day’s return walk (including time for treatment) of the service / site.
  • Coverage is > 50 % in rural areas, > 70 % in urban areas and >90 % in camp situations
monitoring of cmam interventions key indicators for mam sphere
Monitoring of CMAM interventions: key indicators for MAM (Sphere)
  • The proportion of discharges from targeted SFP should be:
    • Recovered > 75 %
    • Deaths < 3 %
    • Defaulter < 15 %

They are primarily applicable to the 6–59 month age group,

although others may be part of the programme.

  • Distance: > 90 % of the target population is within less than one day’s return walk (including time for treatment) of the programme site for dry ration SFP and no more than one hour’s walk for on-site wet SFP
  • Coverage is > 50 % in rural areas, > 70 % in urban areas and > 90 % in a camp situation
additional data for monitoring cmam interventions
Additional data for monitoring CMAM interventions

Derived from routine monitoring and other sources:

Sources of data:

Registration books

Individual follow up charts

Interviews and Focus group discussions

Observation, home-visits

  • Average length of stay
  • Average weight gain
  • Relapse rate
  • Distribution of admissions per type, per age, per origin…
  • Causes of death
  • Reasons for defaulting
  • Investigation of non-recovery cases
m e for cmam interventions supervision
M&E for CMAM interventions: Supervision

6-Jun-14

Supportive supervision visits to sites are designed to ensure / improve the quality of care offered by:

  • Identifying weaknesses in the performance of activities, taking immediate action and applying shared corrective solutions
  • Strengthening the technical capacity of health workers and motivating staff through encouragement of good practices

Supervisors and managers ensure that the performance of activities and organization of the services meet quality standards.

66

evaluation of sam management interventions
Evaluation of SAM management interventions

Effectiveness: programme performance with a strong focus on coverage

Appropriateness: e.g. distribution and time of opening of treatment sites

Connectedness: relates to the links with health system and shows levels of possible integration

Cost-effectiveness has also been measured with various methods and showing high differences between contexts and different approaches

m e of cmam interventions population level assessments
M&E of CMAM interventions: population level assessments
  • Community level assessment can be done through:
    • Repeated anthropometric surveys
    • Programme coverage
evaluation of coverage for cmam
Evaluation of coverage for CMAM
  • Coverage is one of the most important elements behind the success of the CMAM approach.
    • It is measured through studies using two main approaches:
      • The centric systematic area sampling (CSAS)
      • The Semi-Quantitative Evaluation of Access and Coverage (SQUEAC)
  • Coverage should reach at least 90% of severe cases in camps situation, 70% in urban setting, 50% in rural setting (SPHERE standards)
evaluation of management of mam interventions
Evaluation of management of MAM interventions

Same criteria as for all other interventions (relevance, efficiency, etc.)

SFP evaluations are rarely shared, but evidence showed that defaulting and non-response are very common

Needs for evaluating use of Ready-to-Use-Supplementary Food products in terms of efficiency: gain of weight, effect of defaulting, effect on easiness for beneficiaries, etc.