Research Methods in Monitoring & Evaluation

1. Research Methods in Monitoring & Evaluation Deciding How to Collect and Communicate Information

2. Types of Information Sought • Research can be classified as qualitative or quantitative • Criteria for classification depends on: • The purpose of the study; • How variables are measured; and • How the information is analysed

3. Types of Information Sought

4. Sampling • Sometimes it is not possible to study the entire population - In these cases a representative sample of the study population has to be drawn - The representativeness ensures that conclusions are reflective characteristics of the whole population - If a sample is biased or too small, the results will be less reliable or even invalid.

5. Factors to keep in mind when sampling • The Sampling frame: • is a description of the set of all possible individuals you could choose for a sample. • Put simply, it is a set of identified, specific units to be studied within a population • need to clearly delineate the sector of the population that needs to be part of the sample.

6. Factors to keep in mind when sampling Cont… • An Appropriate sample size: • A sample size determines the validity of the findings. • Besides availability of resources, a sample size is determined by the variation within the population, the number of subgroups to be analysed, and the maximum allowable error. • The latter refers to the likelihood of the sample not representing the population.

7. Factors to keep in mind when sampling Cont… • Sampling method: • one can choose between random sampling and non-random sampling. The choice depends on the type of information sought. • Random sampling is associated with quantitative data collection and analysis, and every individual stands an equal chance of being selected • Non-random sampling is usually associated with qualitative data collection and analysis, and involves a deliberate sampling within a population.

8. Random Sampling • Sampling is done without prior knowledge or consideration of particular characteristics. • Starts by identifying, naming or numbering all the units in a population (i.e.. making a sampling frame) so that every individual within the population has equal chance of being sampled. • Actual units to be selected are using the random sampling methods (simple random, systematic and stratified) to be discussed below. • Sampling error or risk of bias is known and can be minimised

9. Non-random sampling • is associated with qualitative research methods. • Involves a more focused or deliberate sampling within a population - sample is based on the researcher’s judgment about exactly whom to include a sample. • You might want a specific perspective and seek certain people or groups. • Risk of bias is high and is difficult to assess

10. Sampling Methods

11. Core M&E Methods • Stakeholder Analysis: • Is useful at different stages during the project: • It can help identify stakeholders to involve in (re)designing a project and its M&E system, and assess their interests and how these relate to the project and to M&E. • It may be used to assess stakeholder relations, including cooperation and conflicts; and considering factors affecting stakeholders and their activities. • Can be used to develop an appropriate sample of stakeholders most central to the issue at hand • May provide a foundation and strategy for participation throughout the project, thereby making it easy for stakeholders to learn from one another.

12. Stakeholder Analysis cont… • Methods in Stakeholder Analysis: 1. Clarify the purpose of stakeholder analysis and agree on criteria for assessing the stakeholders: • For M&E the purpose could be “to make sure that we include all key stakeholders in developing the project’s M&E systems” OR • For specific M&E tasks such as participating in the annual project review process, the purpose could be “to ensure that we include the key opinions in our annual project review” 2. List the criteria to be used to prioritise stakeholders to be involved 3. List all the people and organisations you think of that might fit your criteria. This list has to be revisited several times as you design the M&E systems to ensure that all groups are included and updated.

13. Stakeholder Analysis cont… 4. Classify the stakeholders on the basis of criteria which involves developing a stakeholder matrix with the criteria along one axes and the stakeholders along the other. Prioritise which stakeholders to involve in developing the M&E systems. 5. Finally reach an agreement on how best to involve people. This is done by asking different people how they think they can be optimally involved.

14. Core M&E Methods Cont… • Documentation Review: • Is useful to understand the historical evolution and performance of a project or organisation through its documentation be it written, electronic, photographic or video form. • For M&E this method can provide baseline information on a project area or a particular indicator. • Can also identify key issues needing to be addressed in a further M&E analysis. • It also can provide a good background to activities today to explain whether changes are occurring and why or why not.

15. Documentation Review Cont.. • Methods in Documentation Review: • Be clear about the questions you want to answer and the type of information required for this. • List all possible sources of existing information. • Prioritise those that will provide useful information in a cost- and time-effective manner. 4. Collect the identified documentation and check its reliability. Note contradictory evidence and analyse it in terms of questions to be answered. 5. Identify information gaps or whether contradictory evidence needs to be clarified. Identify another data collection method such as interviews or questionnaires to fill the gaps.

16. Core M&E Methods Cont… • Biophysical Measurements: • Is useful to measure physical changes over time related to any indicator (e.g. health, agriculture, etc.) using any accepted measurement unit and procedure. • For M&E this can provide reliable, statistically verifiable data from an important basis fro measuring change and impact. • Methods in Biophysical Measurements: • Be clear about what indicator or piece of data is to be monitored. • Agree on what the required degree of accuracy is. If a high level of scientific accuracy is needed, then expertise and an appropriate method must be sought. • Adjust the selected method and how it is used to be suitable to local conditions, skills and resources.

17. Methods in Biophysical Measurements Cont… 4. The data need to be recorded in graphs and tables with numbers or words. These can then be used as a framework to follow in order to make comparisons over time. 5. Direct measurements can be an important part of methods

18. Core M&E Methods Cont… • Direct Observation: • Is useful to obtain timely information by observing what people do, to help make decisions on improving a project’s performance or for generating insights and findings that can serve as hypotheses for more focused studies. • For M&E this method can be used to understand the context in which information is collected and can help explain results. • Methods in Direct Observation: • Have a clear conceptual framework as well as guidelines for what needs to be observed and the information required • Choose appropriate observers who must be properly trained in observational skills.

19. Methods in Direct Observation • Factor to consider: Outsiders might need more time to know what is significant. On the other hand an outsider might observe significant issues that local people no longer notice or take for granted. 3. Collect and record data as observed. The findings must be discussed with other staff, implementing partners and perhaps also with primary stakeholders.

20. Core M&E Methods Cont… • Cost-benefit Analysis (CBA) • Is used to provide an accounting framework to enumerate the range of benefits and costs surrounding a decision to avoid costly activities that yield few benefits. • For M&E this method is used to evaluate a project by comparing actual final measurements of the costs and benefits against those proposed in the design of the project. • Comparison is sometimes made with other projects that deliver similar services and products. • It is also used to assess costs and benefits of elements of a project such as specific activities or indicators. • A CBA can be carried out at the design stage of a project to inform decisions on what activities to include.

21. Cost-benefit Analysis (CBA)Cont… • Methods in CBA: • Enlist the help of a trained economist or expert in CBA since this method uses various formulas for calculating costs and benefits, discounting, marginal-return analysis and aggregation of the figures • List all the project activities • Calculate all project costs over the project period. CBA only includes costs and benefits you define. Decide on how comprehensive you want the CBA to be – you might include socio-environmental costs and benefits as well. • For each project activity estimate the benefits that may continue to occur beyond the project period. This step requires research and help form specific statistical formulas.

22. Cost-benefit Analysis (CBA)Cont… • Including aggregation in CBA: • Aggregate project costs and benefits through the discounting formula, according to the point of interest. • Calculate annual net benefits by subtracting costs from benefits for each year • Calculate the IRR* occurring at regular periods (might be annually) of a series of annual net benefits. *IRR - interest received on an investment consisting of costs (- values) and benefits (+ values) • Do a sensitivity analysis by increasing costs and/or benefits by a certain percentage and check the impact on the IRR. If the IRR is more than the market rate of return even when costs are increased and benefits decreased, the project is usually considered to be economically robust.

23. Cost-benefit Analysis (CBA)Cont… • Not including aggregation in CBA: • Decide whether to estimate costs and benefits for individual project activities, in order to choose between alternatives. • Calculate the potential marginal rate of return from each activity option, by estimating the potential costs and benefits of each alternative activity for the same activity. • Add the calculations from the above step to the social, institutional and technical features of the activity options, to permit a more informed choice. • Present the findings to key stakeholders for analysis and discussion, to assist in decision making.

24. Core M&E Methods Cont… • Questionnaires and Surveys: • Are used to gather information from a large number of people using a set of specific questions, often to allow for statistical analysis. • These methods are widely used in M&E as they allow for focused data collection about specific performance questions or indicators.

25. Questionnaires and Surveys Cont… • Methods in Questionnaires and Surveys: • Agree on the purpose and information needs of the question • Decide whether the information requires a questionnaire or survey format • Ensure that the questioning is focused and well formulated or it will not be useful • Agree on the participants and how many. Also decide on whether to have face-to-face, or self-administered questionnaires • Pre-test the interview questions to make sure that they are appropriate, accurate and give the information needed. • Collect and analyse information.

26. Core M&E Methods Cont… • Semi-structured interviews: • Is used to gather face-to-face information from an individual or a group, using a series of broad questions aimed at guiding the conversation. • This method allows for in-depth understanding of qualitative issues in particular. • Group meetings require the use of simple language to ensure that all participants understand.

27. Semi-structured interviews Cont… • Methods in semi-structured interviews: • Define the purpose and information needs of the enquiry. • Agree on who the participants are, and how big the sample should be. • Gather and train a team of people to ensure that they understand the purpose and develop a proper skills. • Pre-test the interview questions to ensure that they are appropriate and accurate enough, and that the answers permit useful analysis. • If you conduct group interviews with more than one interviewer, it might be useful to have a review of the interview, assess the validity of the answers and decide if the interview need to be adapted. • Analyse the information coming from the interviews.

28. Core M&E Methods Cont… • Case Studies: • Is used to document a life story or a sequence of events over time related to a person, location, household or organisation. • In M&E, case studies allow for an in-depth understanding of the context and human factors behind general data that might be collected through other means. • This method is useful in complex situations where many variables interrelate and where outcomes and impact are likely to vary across different populations.

29. Case Studies Cont… • Methods in case studies: • Define the purpose and precise information sought. • Decide on sampling • Develop the information checklist that will guide the information collection. Case studies involve a written account of both observations and answers. • Repeat the discussions with enough frequency depending on the rate of change.

30. Core M&E Methods Cont… • Other Methods: • Brainstorming, • Focus groups, • Mapping (sketching), • Diaries, • Historical trends and timelines, • Mind-maps, • Impact-flow diagrams