Integrating Planning, Monitoring and Evaluation to Increase Effectiveness of your Interventions

1. Integrating Planning, Monitoring and Evaluation to Increase Effectiveness of your Interventions Training organized on 12 December 2011 for HIDA (Ethiopiabased NGO) by Laetitia Lienart , IAS Planning, Monitoring & Evaluation Expert

2. PROGRAMME

3. INTRODUCTION

4. M&E DEFINITION & ROLE M&E is a systematic process of collecting and analyzing information to accompany the implementation of an action, project or programme and assess its process, outcomes and impacts. LEARNING / CONTINUOUS IMPROVEMENT ACCOUNTABILITY Institutional Memory & Evidence* M&E is not an audit nor does it seek to place blame.

5. WHAT MAKES M&E SUCCESSFUL?

6. Another Success Key for M&E: Good Planning

7. PROJECT PLANNING

8. DEFINITION • Planning refers to the phase when you design your project. In other words, when you define its goal, objective, expected results, activities you will have to implement to deliver expected results and inputs/resources needed to implement such activities. • The way you design your project should follow a certain logic that shows in a clear way the links between all these elements. Such logic model is essential for results-based management and for a successful M&E which needs to have a clear understanding of what the project is supposed to achieve and how. Failing to plan is Planning to fail!

9. LOGIC MODEL • Also called ‘’theory of change’’ or ‘’results chain” • Various types of logic model exist and all have pros & cons (just choose the one which is the best adapted to your needs and available resources): • Logframe • Outcome mapping • Outcome chain • Social framework • Etc.

10. THE RESULTS CHAIN The logframe (project framework) is an expression of the “Results Chain” – the results you expect the project to achieve. The box below provides an example of a Results Chain and how it aligns with the LF.

11. WHATIS A LOGFRAME? • A table giving a clear and synthetic picture of the project impact, outcome, outputs and inputs, their respective indicatorsand sources of verificationas well as assumptions. • The LF process helps guide the planning of a journey from where we are now, HERE, to where we want to go, THERE. • A - Where do we want to be? • What are our expected impact and outcome? • B - How will we get there? • What arethe outputs do we have to deliver? • C - What may stop us getting there? • What are the risksand how can we manage them? • What assumptionsare we making? • D - How will we know if we’ve got there? • What are our indicators, baseline, milestones and targets? • What source of information(evidence) do we need? • E – What do we need to get there? • What detailed resources and related budget are needed? HERE THERE

12. STEP 1 - DEFINE THE IMPACT (GOAL) This is thehigher order objective, the longer term impact, that the project will contribute to. It defines the overall “big picture” need or problem being addressed; it expresses the justification of what is planned. TIPS Have only oneImpact/Goal statement. Example: Decreased AIDS mortality rate in the Nbiya region.

13. This describes the specific and immediate results (outcomes)of the project. It should not be entirely deliverable, i.e. fully within the project manager’s control. If it is deliverable, then it should be an Output*. TIPS Have only one Outcome. If you think you have more, then you may need more than one logframe; or your multiple outcomes are in fact indicators of a single outcome or lower outputs. Example: Improved access to HIV/AIDS treatment in the Nbiya region. STEP 2 - DEFINE THE OUTCOME (PURPOSE/OBJECTIVE)

14. The Outputs describe what the project will deliver in order to achieve the Outcome/Purpose. They are the results that the project must deliver in the control of the project manager. Examples: a) Medical infrastructures rehabilitated/reinforced in the Nbiya region; b) HIV/AIDS awareness of Nbiya region’s inhabitants raised; c) Health care workers of the Nbiya region adequatly trained on HIV/AIDS; d) Partnerships between Nbiya’s government and drug supplier(s) initiated; etc. Typically there are between 2 – 6 Outputs; any more than that and the logframe will become over-complicated. STEP 3 - DEFINE THE ‘RESULTS/PRODUCTS’ OR OUTPUTS

15. The Activities describewhatactionswill be undertaken to achieve each output. Examples: needs assessment, recruitment of experts/consultants, design of materials, development of training programme, selection of participants/trainers/suppliers, organization of coordination meetings, distribution of awareness materials, implementation of works, procurement of equipment & supplies, etc. Unlike the previous LF template, activities do not have to be captured in the LF. You can report them in a separate table and depending on the donor requirements, specify for each of them the milestones, risks and staff responsible (see example next slide). STEP 4 – DEFINE THE ACTIVITIES

16. ACTIVITIES LOG TEMPLATE

17. STEP 5 – TEST THE LOGIC FROM THE BOTTOM TO THE TOP Use the IF/THEN testto check cause and effect, reading from the bottom up: If we do these activities, then this output will be delivered. If we deliver these outputs, then this outcome will be achieved. If the outcome is achieved, then this will contribute to the impact.

18. Taking all the activities needed for Output 1, ask the question: ‘if we complete these Activities successfully, thenwhat can stop us deliveringOutput1?’ . Repeat for all the other Outputs taking each Output and its associated activities in turn. Taking all the Outputs together, ask the question: ‘if we deliver all these Outputs successfully, then what can stop us achieving our Outcome?’ Now ask the question: ‘if we are achieve our Outcome successfully, then what can stop us contributing to the Impact?’ STEP 6 – UNDERTAKE A RISK ANALYSIS IDENTIFY THE RISKS

19. What is its likely impact? high, medium or low. What is its likely probability? high, medium or low. You may at this point decide to hereafter disregard insignificant risks. Discuss & agree possible mitigation measures; transfer them into the Activities log template (i.e. extra activities). Example: risk: decrease in the # of health care workers affected to the Nbiya region mitigation measure/new activity: advocacy/lobbying to the relevant authorities & stakeholders involved in health care workers management STEP 6 – UNDERTAKE A RISK ANALYSIS (cont.) ANALYSE AND RATE EACH RISK

20. Assumptions are what remains afterthe mitigation measureshave been put in place (even if mitigation measures are successful, it is unlikely you can remove the risk completely). Assumptions are external factorswhich could affect the success of the project but over which the project manager has no direct control. Example:‘’Strong commitment and continuous support of the Nbiya’s authorities towards combatting HIV/AIDS‘’ Logic test: once Activities have been carried out, and if the Assumptions at this level hold true, Outputs will be delivered. Test the same logic for the upper levels. STEP 7 – FORMULATE THE ASSUMPTIONS

21. STEP 8 – FORMULATE INDICATORS Once yourlogic model isestablished, itis important to think about which information youwillneed to collect to measureyourproject performance. Indicators are performance measures, which tell us what we are going to measurenot what is to be achieved. Indicators should be specific, usable and easy to measure. The basic principle is that “if you can measure it, you can manage it’’. TIPS Do not start from scratch and use indicators that have been used & tested in your field, such as: http://www.unaids.org/en/resources/presscentre/featurestories/2011/november/20111118reportingguidelines/

22. Indicators at impact level • Indicators should be “impact” measures. They should only state what will be measured – i.e. they should not include elements of the baseline or target. • Under-five AIDS mortality rate falls by at least 5% points by 2012 (to be avoided) • Under-five AIDS mortality rate (recommended)

23. Indicators at outcome level • Outcomeindicatorsdemonstratethe changes whichtake place as a result of the organization’swork, and show progresstowards meeting the outcome. • As with the Impact, indicators should not include elements of the baseline or target. The rule is that each Indicator you choose to measure your objectives must be verifiable by some means. If not, you must find another indicator. • 50% increase in ARV uptake rate in targeted areas by 2012 • ARV uptake rate in targeted areas

24. Indicators at output level • Output indicatorsdemonstrate the work the organizationdoes and show progresstowards meeting objectives. • Best Practice suggests a maximum of three Indicators per Output • Output indicators can be set for: • Quantity: Number of HIV testing services • Take-up: Number of people using HIV testing services • Accessibility: Type of people using HIV testing services • Quality: Level of user satisfaction with HIV testing services • Cost

25. STEP 9 – DEVELOP BASELINE All projects SHOULD have baseline data at Impact, Outcome & Output level before being sent for approval • To make sure the information– in particular baseline data - is robust, you might need to commission new analysis. However, this should be avoided wherever possible: you are encouraged to draw data and analyses from existing sources (for example, from UNAIDS, a partner government statistical office, etc.). • What is important is that the data and analysis are current, consistent and as accurate as can be reasonably achieved, and disaggregated by sex where appropriate.

26. STEP 10 – MILESTONES & TARGETS Indicators are a means by which change will be measured while milestones & targets are respectively mid-term and end “goals”. Milestonesare intended to help you track progress, and therefore you should consider the specific trajectory of your project, taking into account all relevant factors, including the sequencing of activities and the release of data from the source of monitoring information*. Targets must be Specific, Measurable, Achievable, Relevant and Time bound (i.e. SMART), thereby indicating the desired result at the end of the project. In line with the guidance for Indicators and baselines, targets must be included wherever baseline data is available and should be disaggregated by sex where appropriate.

27. STEP 11 – IDENTIFY THE SOURCE Whenyouchoose an indicator, itis important to decide how youwillmeasure. It iswhatweusuallyrefer to as « Evidence » or « Verification ». This is a vital stage of the initial planning that is often overlooked. It should be considered when you formulate your indicators.

28. Sometypical sources of verification National and international statistics Project records, reviews and reports Minutes of meetings and attendance lists Stakeholder feedback & focus groups Surveys and reports Newspapers, radio and TV recordings, photographs, satellite imagery External evaluation reports, training evaluation questionnaires, etc.

29. STEP 12 – COMPLETE THE ROW LINKING THE INDICATOR, BASELINE, MILESTONES, TARGET & SOURCE

30. STEP 13 – DEFINE INPUTS This level indicates the inputs and related costs needed to fund the project activities,as well as the breakdown of costs by donors (in case the project is funded by several donors). It also shows the staff who will be involved in the project implementation, and if possible the proportion of their time allocated to the project

31. LOGFRAME TEMPLATE

32. LOGFRAME TEMPLATE (cont.)

33. MAKING CHANGES TO THE LF • Logframes are dynamic, subject to change throughout the active life of the project to which they refer. • Changes to a logframe are normally made during a formal review, or in response to circumstances, and should be done in close consultation with all key stakeholders.

34. RECOMMENDED WEBSITES TO LEARN MORE ON THE LF http://www.dfid.gov.uk/Documents/publications1/how-to-guid-rev-log-fmwk.pdf http://www.dfid.gov.uk/Documents/funding/gpaf/Logframe-how-to-note2011.pdf http://forexsoftwarebusiness.com/tags/guidance-on-using-the-revised-logical-framework.html

35. EXERCISE Subject: the Nbiya region is severely affected by HIV/AIDS, with children and women among the most HIV infected populations. You wish to apply for a tender launched by a major donor in order to address HIV/AIDS related health problems in that region. Methodology: each working group is expected to follow all the above-presented steps to eventually design and present a complete LF with 2 outputs. Use the fact sheet as main background.

36. M&E SCOPING AND FRAMEWORK

37. M&E SCOPING Have a focus for your M&E because you cannot measure everything • Possible focus: • Particular project or activity • Particular target group • Inputs, process, outputs, outcomes (changes) and/or impact • Particular evaluation questions • Criteria to define your M&E scope: • Info needed for further funding • You know exactly how the info will be used and for what • Level of efforts required to collect the information (always consider the info which is already being collected)

38. M&E FRAMEWORK Indicatorsand their sources of evidenceconstitute the basis of an M&E framework • Other elements that should be included: • Staff responsibilities • Resources needed and Budget • Frequency of data collection

39. INFORMATION COLLECTION TOOLS • Sometimes, sources of evidence are not immediately available and you will have to create a tool to collect the required information (e.g. survey, interview, etc.). TIPS • Turn the indicators into questions • Indicator: type of food eaten • Question: how many times do you eat meat per week? • Select the best tool to collect this information. Criteria for selection tools: • Reliability • Validity • Ease of collection & analysis

40. DATA MANAGEMENT, ANALYSIS AND USE

41. Management: • How to store information? • Paper • Simple database (e.g. Excel) or sophisticated database (e.g. Access) • Special database: standard or tailored to your needs (financial and training requirements) • Analysis: • Data cleaning and harmonization • Coding if necessary • Interpreting* • Recommending • Use: • Sharing results with staff, users and other key stakeholders • Making decisions on what must we do differently to increase our success

42. PROPOSAL WRITING Some tips that should increase your chances to get money … $ $ $ $ $ $

43. Processoverview: 1. Draft proposal: project team 3. Finalize proposal: project team leader 5. Develop work plan: project team 1 2 3 4 5 2. Review draft proposal: PME expert 4. Disseminate proposal: project team leader

44. Key requirements • Involve all staff concerned and other stakeholders as needed • START designing the project framework & test its validity/logic with main stakeholders • Write the narrative, making sure it matches the project framework (LF) • Ensure consistency throughout the proposal* • Ask colleagues/partners to read it & adjust it if needed • Make sure all teams are properly informed of what they are supposed to deliver as part of the new project, including reporting duties .

45. PROPOSAL STRUCTURE • Background and rationale • Project framework (Logframe) • Activities • Implementation/Coordination (mechanisms) • Risks • M&E • Resources & Budget • Reporting • Appendices (detailed budget, ToRs, etc.)

46. Background and rationale • This section should describe clearly: • The problem that the project aims to address (in other words, why you propose this project) and if applicable, the extent to which the project will contribute to the national targets and MDGs. • Which key stakeholders were involved in the design of this project. • If the project is a follow-up action of a previous project, intervention, workshop or other event. In that case, provide an overview of this previous project, intervention, workshop or other event (you may use Appendixes to provide further information). • The role of the organization in this project (leadership, partnership, representation or affiliation function) and why it is strategically positioned to lead or take part to this project (in other words, highlight your comparative advantages/added values). • The extent to which the project fits into the organization’s strategy. • In which way(s) your key constituencies will be engaged in this project.

47. Project framework • Insert the logframe • Because activities and inputs are described in a separate section of the proposal, the logframe may only includes the 3 highest levels (i.e. Impact, Outcome and Outputs).

48. Activities • This section should describe the main activities that will be implemented to deliver the expected results. • TIPS • Use bullet points (one per activity) or the Activities Log Table. • Match activities with expected outputs as stated in the LF. • Don’t provide too many details (they will be part of the detailed project work plan you will define at a later stage). For example, instead of detailing all sub-activities you will implement to organize a training (such as assess needs of participants, develop curricula, identify speakers, book training venue and accommodation, etc.), just write as a main activity: Organize a training on [topic] for [target group]. • Once the proposal is approved, develop a project work plan or Gantt chart (Excel sheet) with detailed activities and sub-activities.

49. Implementation/Coordination • This section should describe clearly: • Who will be the project leader and what will be his/her role/responsibilities. • Who will be member of the project team and what will be the role/responsibilities of each member. • Role/responsibilities of your Communication team (if not part of the project team). • Role/responsibilities of the staff who are not in the project team and do not belong to the Communication team (only if relevant to the project). • Mandate and composition of the project steering committee, advisory group, working group or any other kind of consultative body set up for this project. • Coordination mechanisms put in place to create synergies and avoid overlapping with other similar initiatives (only if relevant to the project).

50. Risks • This section should describe any external factors (i.e. risks) • that may have a negative impact on the project and cause its partial or total failure. • It should also describe how you will monitor and mitigate them. • M&E • This section should describe clearly: • Role/responsibilities of the project team. • Role/responsibilities of the M&E officer/team. • What the evaluation will focus on (i.e. process, outcomes and/or impact). You may already specify your key evaluation questions: • What worked well? • What did not work so well? • What could be improved in the design phase? • What could be improved in the implementation phase? • What follow-up actions are needed to make project results sustainable?