Effective Monitoring & Evaluation for Malaria Control in Pregnancy

1. Basic Monitoring and Evaluation (M&E) for Control of Malaria in Pregnancy Module 8 Version 2

2. Learning objectives By the end of this module, learners will be able to: • Define “community-based information system” (CBIS) and state why such a system is important • Describe the importance of recordkeeping • Define basic M&E concepts and state the differences between monitoring and evaluation* • State why M&E is important in programming *Health care workers only

3. Learning objectives, cont. • Differentiate between data and information • Describe the qualities of good data • Identify sources of data (facility and community) • Demonstrate how to fill out the community health worker (CHW) register and the referral form

4. What is a CBIS? • A process of collecting information for services that are provided at the community level (i.e., outside of health facilities [HFs]) • Requires planning, collecting, managing, reporting, and using information • The process is carried out with some level of community participation

5. Information communities can report to the CBIS • Births and deaths (vital events registration) • Notification of cases of infectious diseases and outbreaks (disease surveillance), for example: • Ebola • Lassa fever • Plague • Identification of pregnant women • Identification of orphans

6. Information the CBIS can collect for community-based malaria programs • Records of: • Home management of malaria for children and adults • Rapid diagnostic tests (RDTs) • Services concerned with malaria in pregnancy, for example: • Antenatal care (ANC) • Community-directed intermittent preventive treatment in pregnancy (c-IPTp) • Home-based prevention of malaria, such as: • Indoor residual spraying • Use of mosquito repellents • Use of insecticide-treated bed nets or long-lasting insecticidal nets (ITNs/LLINs)

7. Information the CBIS can collect for community-based malaria programs, cont. • Referrals for: • Case management of complicated malaria and other illnesses • ANC services • Adverse reactions to drugs

8. What is recordkeeping? Recordkeeping is a process of documenting services provided over a period of time using specific formats designed for the purpose Remember, “If it is not in writing, it does not exist!”

9. Reasons why recordkeeping is important • The need to know how many people are provided specific services over a period of time, for example: • How many pregnant women received doses of intermittent preventive treatment in pregnancy (IPTp)? • How many pregnant women attended ANC? • How many pregnant women were referred for complicated malaria? • The need to know how the program is performing in HFs, for example: • What proportion of HFs provide ITNs/LLINs at ANC? • What proportion of HFs provide IPTp with sulfadoxine-pyrimethamine?

10. Reasons why recordkeeping is important, cont. • For planning purposes, for example: • The quantities of different drugs and other supplies that are required to meet clients’ needs over a specific period of time • Whether the right people are receiving the services • To encourage service providers to work more effectively and be accountable to the community and HFs.

11. What is reporting? A process of providing a summary of activities carried out over a period of time, for example number of times different services are provided in a given time period.

12. Why is it important to report services provided? • Summarizes accomplishments • Shows progress at a glance • Informs planning • Gives supervisors an easy overview of each CHW’s activities over a period of time • Provides feedback to funders, fund recipients, and program beneficiaries

13. M&E

14. What is M&E? • Monitoring is a process of routine tracking of program activities on a regular basis. • Evaluation is a process of measuring the extent to which program outcomes and output are achieved.

15. What is monitoring? • Often called process evaluation • A continuous, systematic process of collecting, analyzing, and using information to track the efficiency with which a program is achieving its goals and objectives What does the chart on the right tell you about the program?

16. What is monitoring? (cont.) • Provides regular feedback that measures change over time in any of the program components such as: • Costs • Personnel • Implementation • An unexpected change observed in monitoring data may trigger the need for a more formal evaluation of activities

17. What are the things to monitor? • Progress of project activities: • Services • Training • Outreach • Meetings • Information, education, and communication activities • Use of resources: • Budget/actual expenditure • Commodities • Equipment • Supplies • Staff • Results: • Achievement of project objectives • Coverage • Outputs • Impact • Effect on project beneficiaries • Institutionalization and sustainability: • Government acceptance of ownership • Government financial inputs • Scale-up plans • Staffing • Changes in policies and procedures • Organizational systems

18. What are the things to monitor at community level? • Stock levels of all the malaria commodities and medicines: • Artemisinin-based combination therapies (ACTs) • Sulfadoxine-pyrimethamine (SP) • RDTs • ITNs/LLINs • Types of ACTs provided

19. What are the things to monitor at community level? (cont.) • Storage facility and condition of commodities, for example: • ACTs • SP • RDTs • Training in all aspects of malaria control • Stock levels and use of data capturing tools

20. What is evaluation? • A systematic process, limited in time, of collecting, analyzing, and using information to assess your program’s effectiveness, relevance, and impact in achieving its goals • Often involves measuring population-level changes in knowledge, attitudes, behaviors, skills, community norms, utilization of health services, and health status • Provides regular feedback that helps a program analyze the consequences, outcomes, and results of its actions Source: World Health Organization 2004

21. What are the things to evaluate? • Effectiveness—Is the project achieving satisfactory progress toward stated objectives? • Efficiency—Are the effects being achieved at an acceptable cost, compared with alternative approaches? • Relevance—Are project objectives still relevant as time passes? • Impact—What difference has the project made?

22. What are the things to evaluate at community level? • Reduction in fever/malaria cases seen in routine reports • Number of referrals • Number of structures adequately treated with indoor residual spraying • Availability and accessibility of malaria commodities and medicines • Use of ITNs/LLINs by the community • Appropriate and effective use of ACTs • Appropriate storage conditions (temperatures) of ACTs and RDTs

23. Brainstorming session! What are the differences between monitoring and evaluation?

24. Good-quality data

25. What are data? • Facts and statistics collected together for reference or analysis • A set of values of qualitative or quantitative variables • Pieces of data are individual pieces of information • Data are collected, measured, analyzed, and reported • Data can be visualized using graphs or images • Data can be collected at both community and HF levels Adapted from Padilla 2016

26. Collecting and visualizing HF data • The data use poster (also known as a monitoring wall chart) is used to: • Chart monthly HF data • Visualize performance trends • Facilitate discussion among providers and during supervision on key indicators and project areas, for example: • IPTp third dose (IPTp3) • At least one ANC contact

27. What makes good data? • Good data are those that are: • Reliable • Valid • Complete • Timely • Precise • Truthful (have integrity) • Should be easy and cost-effective to collect and sufficiently up-to-date

28. What can reduce data quality? • Unclear data definitions • Poor data collection instrument or tool • Untrained data collectors • People selected for data collector training were poorly chosen • Poorly trained data officers • Unreliable data entry system • Insufficient data checks • Insecure data storage system • Frequent turnover of data officers • Illegible handwriting • Typing or transcription errors

29. What are examples of data sources in your community? • There are routine sources (continuous data collection) and nonroutine sources • The HF: • Routine service statistics • Patient’s/client’s record booklets • Charts • Registers (ANC, delivery, inpatient, outpatient) • The community: • Survey findings • Census • CHW reports • Immunization reports • Vital statistics in registration centers

30. Data are helpful for... • The individual client • The HFs • Communities • The district, subnational, and national health systems

31. Data are necessary to... • Adequately monitor health conditions • Provide continuity of care: • Give the correct number of doses of IPTp at the correct time in pregnancy • Communicate effectively among health care providers or among health care sites (if a patient is referred) • Provide information for health planning and budgeting at HF, district, subnational, and national levels

32. Who is responsible for data? • The client first provides accurate information • The volunteers and service providers record this information on the client’s card and in the register • The HF in-charge • M&E officersat district, subnational, national malaria elimination or control program, and ministry of health levels

33. Demonstration: How to fill the data collection tools for c-IPTp • Tools for capturing community-based data: • CHW register (records daily cases, including c-IPTp) • CHW monthly tally sheet • Referral form • Exercises on completing community-based data capturing tools

34. Sample data collection toolsCHW register

35. CHW monthly tally sheet

36. Summary • The planning, collection, management, reporting, and use of information are carried out with community participation • Data are needed for the community, HF, district, subnational areas, national malaria elimination or control program, and ministry of health to: • Identify and prioritize health problems • Allocate resources adequately

37. Summary, cont. • Referral is crucial for: • Additional ANC services not provided at the community or HF levels, for example: • Hematinics • Tetanus toxoid immunization • Case management of complicated malaria and other illnesses (all cases with fever illness should be referred to HF)

38. Summary, cont. • To collect high-quality, reliable data, we need effective supportive supervision • Establish a reporting, feedback, and data use mechanism, for example: • Monthly graphing of service statistics • CHWs monthly meetings • Malaria focal persons monthly meetings • HF staff weekly meetings

39. References • Padilla E. 2016. Substation Automation Systems: Design and Implementation. West Sussex, UK: John Wiley & Sons. • World Health Organization (WHO). 2004. Compendium of Indicators for Monitoring and Evaluating National Tuberculosis Programs. Geneva, Switzerland: WHO. http://apps.who.int/iris/bitstream/handle/10665/68768/WHO_HTM_TB_2004.344.pdf. Accessed October 2, 2018.

40. Thank you! Any questions or comments?