IDEV 624 – Monitoring and Evaluation

1. IDEV 624 – Monitoring and Evaluation Assessing Program Theory Payson Center for International Development and Technology Transfer Tulane University

2. Monitoring and Evaluation (M&E): Definitions and Concepts

3. What is Monitoring and Evaluation (M&E)? • Monitoring and evaluation of program activities provides the means for: • Learning from past experience • Improving service delivery • Planning and allocating resources • Demonstrating results as part of accountability to stakeholders (World Bank)

4. Monitoring: A continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds Evaluation: The systematic and objective assessment of an on-going or completed project, program or policy, its design, implementation and results The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact and sustainability (OECD, Glossary of Key Terms in Evaluation and Results Based Management, 2002)

5. A Public Health Questions Approach to HIV/AIDS M&E Are collective efforts being implemented on a large enough scale to impact the epidemic? (coverage; impact)?Surveys & Surveillance Are we doing them on a large enough scale? Determining Collective Effectiveness OUTCOMES & IMPACTS Are interventions working/making a difference? Outcome Evaluation Studies OUTCOMES Monitoring & Evaluating National Programs Are we doing them right? Are we implementing the program as planned? Outputs Monitoring OUTPUTS What are we doing? Are we doing it right? Process Monitoring & Evaluation, Quality Assessments ACTIVITIES Are we doing the right things? What interventions and resources are needed? Needs, Resource, Response Analysis & Input Monitoring INPUTS Understanding Potential Responses What interventions can work (efficacy & effectiveness)? Efficacy & Effectiveness Studies, Formative & Summative Evaluation, Research Synthesis What are the contributing factors? Determinants Research Problem Identification What is the problem? Situation Analysis & Surveillance (UNAIDS 2008)

6. Strategic Planning for M&E: Setting Realistic Expectations All Most Some Few* Number of Projects Input/ Output Monitoring Process Evaluation Outcome Monitoring / Evaluation Impact Monitoring / Evaluation Levels of Monitoring & Evaluation Effort *Disease impact monitoring is synonymous with disease surveillance and should be part of all national-level efforts, but cannot be easily linked to specific projects 6

7. (World Bank 2009)

8. Assessment of Program Theory

9. Assessment of Program Theory • Program theory: • Conception of what must be done to bring about the intended social benefits • Assessment of program theory: • Task of assessing the quality of program conceptualization • Good: If it represents the “know-how” necessary to attain the desired results • Poor: If it would not produce the intended effects even if the project is implemented well  Evaluators find that program theory often is not very convincing (Rossi/Lipsey/Freeman: 2004)

10. Program Impact vs. Program Process Theory • Program process theory: • Represents assumptions about the capability of the program to provide services that are accessible to the target population and compatible with their needs • Program impact theory: • Involves the sequence of casual links between program services and outcomes that improve the targeted social conditions (Rossi/Lipsey/Freeman: 2004)

11. Evaluability Assessment • Evaluability assessment: Does the program/project meet the minimal conditions for evaluation? • Description of the program model (the program’s design and logic)with particular attention to defining goals and objectives • Assessment how well-defined and evaluable that model is • Identification of stakeholder interest in evaluation and the likely use of findings  A common outcome is that managers and sponsors recognize the need to modify the program (Rossi/Lipsey/Freeman: 2004)

12. Assessment of Program Theory (cont.) • Program theory can be “implicit” or (better) “articulated” • If implicit, the evaluator must extract and describe program theory before a program can be evaluated • Objective: To depict the program “as intended”, based on the actual expectations held by decision-makers about what the program is supposed to do and the results expected to follow • Program theory has to be assessed in relationship to the needs of the target population that the program is intended to serve ( Needs assessment) (Rossi/Lipsey/Freeman: 2004)

13. Theory of Change: Theory of how an initiative leads to desired results (World Bank 2009)

14. (World Bank 2009)

15. Standard Flow Chart (World Bank 2009)

16. Example: Theory of Change (World Bank 2009)

17. Logic Model - Example (for a teen mother parenting program) (Rossi, P. H. et al., p. 95)

18. The Management Plan

19. Key Steps • Conduct a needs assessment • Develop and refine a conceptual framework • Create a management plan: • Clarify your goals • Define your objectives • Identify your key activities

20. Factor Factor Activity Target Condition Factor Activity Factor Factor Initial Conceptual Model

21. The Management Plan: Goals Objectives Activities

22. Program Management Plan

23. Management Plan Structure Goal • Objective 1 • Activity 1 • Activity 2 • Activity 3 • Objective 2 • Activity 1 • Activity 2 • Activity 3 • Objective 3 • Activity 1 • Activity 2 • Activity 3

24. Goals • The goal is a "general summary of the desired state that a project is working to achieve" (Margoluis/Salafsky 1998: 60).  • An ideal goal is • visionary • relatively general • brief • measurable • Typically, a project has only one goal.

25. Develop Goals • Reshape target condition (conceptual model) into a goal: • Future oriented • Positive statement • Brief and clearly articulated • Measurable

26. Refer to the social problem that you are going to address Refer to the target population and location Use clear terminology Develop Goals (cont.)

27. Project goal: Increase knowledge about sexual and reproductive health in Mexico What is wrong? Poorly Written Goal

28. Project goal: Increase knowledge about sexual and reproductive health in Mexico What is wrong? Goal does not refer to the major health or social problem to be addressed Why is it important to increase knowledge? What do we think will happen if knowledge is increased? Goal does not provide information about the focus population; it is to general Poorly Written Goal

29. Project goal: Improve the sexual and reproductive health of young people ages 15 to 24 in City X, through the reduction of unwanted pregnancy and sexually transmitted infections, including HIV Revised Goal

30. Well-written goals: To reduce incidence of HIV infection among young people ages 15 to 24 in Country X. Increase the capacity of youth-serving organizations to implement more effective peer education programs for preventing HIV infection among sexually active young people (ages 15-24). More Project Goal Examples

31. Objectives • The objectives are "specific statements detailing the desired accomplishments or outcomes of a project. If a project is well conceptualized and designed, realization of a project's objectives should lead to the fulfillment of the project's goal" (Margoluis/Salafsky 1998: 64). 

32. Goals are usually aspiring and far-reaching – not limited in terms of time or other measurement Objectives should be measurable, quantifiable, time-limited and action-oriented  The test of a goal is to make certain that it fits within your organization’s mission; the test for objectives is that they lead to the accomplishment of a goal Goals Versus Objectives

33. Develop Objectives • Select a factor (threat) from your conceptual model • Develop 1-3 specific objectives related to this factor • What do we need to do to accomplish our desired outcome? • What information do we need to demonstrate we actually accomplished what we planned to do?

34. Factor Factor Activity Target Condition Factor Activity Factor Factor Initial Conceptual Model

35. It is important that objectives be realistic, not just impressive, as lofty, unfounded objectives undercut the credibility of the entire project Well-written objectives identify: WHO will be reached WHAT change will be achieved IN WHAT TIME PERIOD the change will be achieved WHERE (in what location) Develop Objectives (cont.)

36. Objectives should be “SMART” Specific To avoid differing interpretations Measurable To monitor & evaluate progress (preferably quantitatively) Appropriate To the problems, goal and your organization’s mission Realistic Achievable, yet challenging and meaningful Time-bound With a specific time for achieving them SMART Objectives

37. When writing objectives, choose action verbs that indicate a change and the direction that change will take Examples: Decrease Increase Strengthen Improve Enhance Objectives and Action

38. Project objective: Train 60 peer educators to promote the ability to practice safer sex What is wrong? Poorly Written Objective

39. Project objective: Train 60 peer educators to promote the ability to practice safer sex What is wrong? This objective refers to a strategy or activity Why does the organization want to train peer educators – what change is sought? Objective is not time-bound Focus population is not specified For this project, is the change to be achieved among the peer educators or among the people the educators will reach? Unclear terminology What does the ability to practice safe sex mean? How will the organization recognize it when it occurs? Poorly Written Objective

40. Project objective: At the end of the three-year project, increase condom use among sexually active young people (ages 15 to 24) in Community X Revised Objective

41. By the end of the first project year, increase knowledge and skills of staff in five clinics in City X in how to counsel young clients (ages 15 to 24) about HIV prevention, with emphasis on condom use and negotiation. By the end of the two-year project, increase the availability of information about the effectiveness of community-based versus center-based peer education programs in increasing condom use among sexually active youth (ages 15 to 24). More Sample Objectives

42. Activities • The activities are "specific actions or tasks undertaken by project staff designed to reach each of the project's objectives" (Margoluis/Salafsky 1998: 73). • The project activities should be • linked • focused • feasible • appropriate 

43. Feasibility • Available resources • Constraints • Cost-effectiveness • Any better alternatives?

44. Analysis of Alternatives • There is no fixed format for the analysis of alternatives. • The purpose of this tool is to systematically compare various possible intervention strategies focusing on the project's key criteria. • The analysis of alternatives helps to choose the best approach for any intervention. This may take a form similar to the table below.

45. Develop Activities • Why do this activity? - Justification • How will activity be carried out? • Who is responsible? • When will activity take place? • Where will the activity take place? • Underlying assumptions • Prerequisites

46. Management Plan Structure Goal • Objective 1 • Activity 1 • Activity 2 • Activity 3 • Objective 2 • Activity 1 • Activity 2 • Activity 3 • Objective 3 • Activity 1 • Activity 2 • Activity 3

47. Management Plan Timeline

48. Timeline • A bar graph which lists the major activities and tasks involved in the project. • Shows how long the activities are supposed to last and the relationship between different activities.

49. Management Plan Timeline

50. Gantt Chart • Visualizes tasks and milestones of a project over time. • Bars typically represent tasks and diamonds represent milestones. • Reflects the length of tasks. • May indicate the exact dates, the persons in charge, etc. for each project component. • Widely used by project managers in the project planning and implementation.