Introduction to M&E

1. Introduction to M&E 28 February, 2011 University of Pretoria

2. Session Objectives • Describe Health Information/M&E Systems • Define Monitoring • Define Evaluation • Purposes of M&E • Describe the relationship between monitoring and evaluation • Define Program Components • Identify the different types of M&E

3. Learning Objectives By the end of this session you will be able to: • Define key M&E terms • Identify components of programs to evaluate • Describe different purposes for M&E • State why Monitoring & Evaluation are important in programming • Describe characteristics of good M&E system

4. Session Overview • Key M&E terms • “M” vs. “E” • The purpose of M&E • Types of evaluation

5. Defining Programs, Projects & Interventions • Program: organized effort to respond to a broad social problem (typically organized at national level) • Project: specific set of activities with linked objectives, that contribute to the overall objective of a program (typically organized at sub-national level) • Intervention: often used in same was as “project”, sometimes a specific sub-set of program or project activities

6. Program Planning Involves Setting… • Goals & Objectives based on intended Impact • Intended Outcomes • Intended Outputs or Deliverables • Planned Activities (Processes) • Inputs or Resources

7. Inputs What you need to implement the program e.g. the financial , human and material resources used to implement an HIV counseling and testing program • Trained personnel • C&T protocols and guidelines • Training materials • HIV test kits and other supplies • Money

8. Activities What the program provides to accomplish its objectives (micro level) • Training workshops on C&T for personnel and site managers • Providing pre-test and post-test counseling to clients • Curriculum development • Recording & reporting • Dissemination of IEC materials • Supervision

9. Outputs What the program delivers • Condoms distributed • Clients receiving pre-test counseling, HIV tests, post-test counseling • Materials distributed • People reached • Intervention sessions completed

10. Outcomes • Changes in knowledge, behavior, attitudes and skills • Quality of C&T improved • Clients develop and adhere to personalized risk-reduction and treatment strategy The results of the program or changes that occur both immediately or some time after activities are completed

11. Impact The wider effect of the program on long-term results • HIV transmission rates decrease • HIV incidence decreases • Changes in HIV mortality and morbidity • Decrease in deaths due to HIV-related TB

12. Example: Components of an HIV program Population level Programme level Input Process Output Outcome Impact Capacity building BCC interventions Community mobilization Policy development Improved skills of providers Increased knowledge Mobilized communities Sound policies available Financial resources Staff Supplies Reduced risk behaviors Declining HIV/AIDS morbidity and mortality Economic growth

13. So how does this relate to M&E? Back to our definitions…

14. What Do We Mean by M&E? • Set of procedures & analytical tools to examine • how programs are conducted (inputs & activities) • their level of performance (outputs) • whether they achieved what they were intended to achieve (outcomes & impact) • Types of evaluation • monitoring (including process evaluation) • evaluation (outcome and impact evaluation) [+ surveillance]

15. Monitoring: What are we doing? Tracking inputs and outputs to assess whether programs are performing according to plans (e.g., people trained, condoms distributed) Evaluation: What have we achieved? Assessment of impact of the programme on behaviour or health outcome (e.g., condom use at last risky sex, HIV prevalence) Monitoring vs Evaluation

16. What is Monitoring? • Is a a continuous systematic process of collecting, analyzing and using information to track the efficiency of achieving program goals and objectives • Provides regular feedback that measures change over time in any of the program components such as costs, personnel and program implementation • An unexpected change in monitoring data may trigger the need for a more formal evaluation of activities

17. Illustration of Program Monitoring Program indicator Program end Program start TIME

18. Example of programme MonitoringMale STD Cases at Thai Government Clinics: 1987-1993 Start of HIV-Control Program Source: Hannenberg et al. 1994

19. Key M&E Questions • Is the program being implemented as planned? • Are things moving in the right direction? • Did the program achieve its objectives? • Can results be attributed to program efforts? • Which program activities were more (or less) important/effective? • Did the target population benefit from the program? • At what cost?

20. What is Evaluation? • Is a systematic process limited in time of collecting, analyzing and using information to assess the effectiveness, relevance and impact of achieving your program’s goals. • Requires study design; sometimes a control or comparison group; often measurement over time. • Often involves measuring changes in knowledge, attitudes, behaviors, skills, community norms, utilization of health services, & health status at population level • Provides regular feedback that helps programs analyze the consequences, outcomes and results of its actions

21. What is Evaluation? (con’t) Rigorous research design need for: • Establishing a causal link between program effort and desired outcomes • Isolating program effect from other, non-program influences on the outcome of interest • Not undertaken routinely; usually reserved for specific situations, such as determining the success of a project for scale-up or replication

22. Outcome Evaluation With program Attribution “Impact” Program Indicator: CPR Without program Program Start Program End Time

23. Impact Evaluation With program Attribution “Impact” Program Indicator: Fertility or HIV prevalence Without program Program Start Program End Time

24. Illustration of Program Evaluation/ Impact With program Change in program outcome Without program Program impact Program start Program end TIME

25. Types of Monitoring and Evaluation

26. Comparison of Potentially Confusing Terms Using the PEPFAR Example

27. Key M&E Questions • Did the program achieve its objectives? • Did the target population benefit from the program? • At what cost? • Can improved health outcomes be attributed to program efforts? • Which program activities were more (or less) important/effective? • What would have happened in the absence of the program? • How can we know or measure this (the counterfactual)?

28. Why Monitor & Evaluate? • To make decisions about project management and service delivery • To ensure effective and efficient use of resources and provide accountability to donors • To assess whether the project has achieved its objectives - has the desired effects • To learn from our activities, and provide information to design future projects Program Improvement Share with Partners Reporting/ Accountability

29. Purposes of Monitoring and Evaluation • Determine whether a plan or program is on schedule with planned activities • Assess whether a policy, plan or program has produced desired impacts • Generate knowledge: • Identify factors (individual, community, programmatic) that influence health outcomes • Help inform policy, planning or program decisions: new services, resource allocation, corrections, etc.

30. Reasons to Monitor & Evaluate - Different Needs for Different Stakeholders: Funding Agencies & Policy Makers • What M&E Measures • Evidence of achievement of program objective • Program outcome and impact • Program cost-efficiency • Data about the target population • What M&E Results Identify • Priorities for strategic planning • Programs that qualify for donor assistance • Best practices • Impact of donor assistance • What Decisions are Guided by M&E Results • How much funding should be allocated to a program • What types of programs should be funded • Which programs approaches should be presented as models • New strategic objectives, activities or results packages • Replication and scaling up of successful programs

31. Reasons to Monitor & Evaluate - Different Needs for Different Stakeholders: Communities & Youth • What Decisions are Guided by M&E Results • Degree to which community members and youth should participate in and support the program • How to better coordinate community actions to address ARH • How many and what type of local resources should be allocated to ARH • What M&E Measures • Youth behavior related to reproductive health • Young peoples’ needs • How program funs are being spent • The process and impact of community participation • What M&E Results Identify • Actual and potential benefits of youth programs • Need for new and better youth services • Community resources that can be used to support ARH programs • Need for local support for ARH issues and actions

32. Reasons to Monitor & Evaluate - Different Needs for Different Stakeholders: Program Managers & Staff • What M&E Measures • Quality of activities and/or services • Why some sites are more successful • Program coverage • What M&E Results Identify • Priorities for strategic planning • Training and supervision needs • How to improve reporting to funding agency • Feedback from clients • Why program is not accomplishing what it set out to do • What Decisions are Guided by M&E Results • Resource allocation • Replication and scaling up of intervention • Fund-raising • Motivating staff • Policy advocacy

33. Fundamental Steps to Carry out M&E • Agreement on the scope and objectives of M&E plan with stakeholders • Selection of Indicators • Systematic and consistent collection of information on the selected indicators • Analyze the information gathered • Compare results with program initial goals and objectives • Share results with stakeholders

34. Defining the Scope of M&E Effort • Scope refers to the extent of the activity you will undertake in a M&E effort. Scope is determined by several factors (questions) • What should be monitored and evaluated? • When should health programs be monitored and evaluated? • How much will M&E cost? • Who should be involved in M&E? • Who should carry out the evaluation? • Where should M&E take place?

35. What Should be Monitored & Evaluated? • M&E can be measure each stage of your program development: design, system development and functioning, and implementation. • After goals, objectives and activities are developed, decision on about M&E at each stage is needed. • M&E effort can measure each stage to determine how the program is working and its impact on the target population.

36. What Should be Monitored & Evaluated? • Program Design is measured by process evaluation: • Assessing how well the program has designed • Documenting the success/challenges with program design • System development and functioning is measured by monitoring and process evaluation: • Document the development of support systems and determine if they are actually operating once program implementation begins • Assess the performance of support systems • Measure how effective the preparatory activities are in readying program personal for program implementation • Implementation is measured by monitoring, process evaluation, and outcome/impact evaluation: • Reveal how program implementation is occurring • Determine whether program is achieving its objectives by measuring the changes in outcomes in your target population

37. When Should Health Programs be Monitored & Evaluated? • Monitoring and process evaluation should occur throughout the life of a program • Outcome and impact evaluations are usually done near the end of a program (baseline gathered to measure change) • Starting M&E at the beginning of a program is ideal • Some activities can still be measured if M&E is started in the middle of a program • Even Fewer activities can be measured if M&E is started towards the end of the program

38. When Should Health Programs be Monitored & Evaluated? Stage of Monitoring Process Outcome/Impact Program

39. Where are the levers in a working health systems • 6 building blocks of health systems strengthening • Service delivery • Health workforce • INFORMATION • Medical products ,Vaccines and technologies • Financing • Leadership and governance • WHO: Nellie Bristol www. globalhealthmagazine.com

40. Input-Output Exercise

41. Program components as they relate to Types of M&E • Assessment and Planning • Input/Output Monitoring • Outcome Monitoring • Impact Monitoring • Process Evaluation • Outcome Evaluation • Impact Evaluation

42. Program Components as They Relate to M&E Outcomes Impact Planning Inputs Processes/Activities Outputs Outcome Monitoring Outcome Evaluation Impact Monitoring Impact Evaluation Input/Output Monitoring Assessments Process Evaluation

43. Monitoring & Evaluation Pipeline Inputs Outputs All Most Outcomes Impact Some Few Number of Projects Measuring each Component Long-term effects Short-term and intermediate effects • Behavior change • Attitude change • Changes in STI trends • Increase in social support • Condom availability • Trained staff • Quality of services (e.g. STI, VCT, care) • Knowledge of HIV transmission • Resources • Staff • Funds • Materials • Facilities • Supplies • Training Levels of Monitoring & Evaluation Efforts

44. M&E Terminology Quiz Work in pairs/group and determine which type of M&E each example is describing.

45. Additional M&E Websites • MEASURE Evaluation: www.cpc.unc.edu/measure • Health Metrics Network: www.who.int/healthmetrics • John Snow Inc.: www.jsi.com • HIV Global Partners: www.globalhivmeinform.org

46. MEASURE Evaluation is funded by the U.S. Agency for International Development (USAID) through Cooperative Agreement GHA-A-00-08-00003-00 and is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill, in partnership With Futures Group International, John Snow, Inc., Macro International Inc., Management Sciences for Health, and Tulane University. The views expressed in this presentation do not necessarily reflect the views of USAID or the United States government.