Program Evaluation and Improvement Using Small Tests Of Change Kristen A. Stafford, MPH Pat Bass, RN, MA Track 1.0 Meeti

1. Program Evaluation and Improvement Using Small Tests Of Change Kristen A. Stafford, MPH Pat Bass, RN, MA Track 1.0 Meeting September 25, 2007

2. “Any road will do if you don’t know where you are going” - Lewis G. Carroll Alice in Wonderland

5. Objective So they’ve collected the data… …what now? A look at how our local partners are using data to inform and improve care and treatment

6. S  C T O Components of our Process Patient Outcomes Data Quality Care Delivery System

7. See What is your data telling you What is your goal Continue Do it on a larger scale or try something else. Try/Track What will you try? Who will do it? When How will you track?  Observe What happened? Did it work? What were the balancing results? What is the STOC Process

8. What is it • Intended to speed up system improvements • Evidenced based management • Site driven • Incorporated into daily routine • Consistent and repeated reviews of information already captured

9. Why do it • To build the monitoring and evaluation capacity of in-country teams and treatment sites to provide a sustainable system of quality care and treatment • To work ourselves out of a job

10. Steps to STOC • Leadership buy in • Engage all levels of care delivery and clinic teams • Incorporate STOC activities into day to day duties – not extra work • Try something small • If it works, keep doing it…if it doesn’t try something else • Keep track of what you try and what happened • Share, share, share now

11. How to tell if its an improvement Started adherence support group on clinic days Goal/target % with undetectable Viral Load Months

12. Patient Outcomes • On treatment cross-sectional review • Randomized population of focus • Started on tx 9 – 15 months before review • Chart abstraction • Patient Survey • Viral Load • Aggregated and by treatment site analysis • Group and site by site feedback • Selection of indicators most related to viral suppression and failure • STOC development with sites • Bi-annual to annual

13. Patient Outcome Tools

14. Example of Findings Statistically Significant p<.05

15. Small Test of Change Example

16. Building Local Capacity: ARV Pickup Tracking ARV pickup in Kenya • Goal: To improve patient adherence to picking up ARVs • Strategy: Automated reports were created in IQTools: • 1) Identify patients that are supposed to come in during a certain time period (i.e. this week) to pick up ARVs • 2) Identify patients that have missed picking up their ARVs after x days • Reports are run and analyzed by LPTF • LPTFs are required to report (monthly) the number of patients that did not pick up their ARVs within x days and feedback is shared

17. Pharmacy Visits

18. How can LPTF use the data? • Based on the data, the local Clinical, SI, and Management Team work with LPTFs to develop a small test of change process to improve patient adherence to picking up ARVs • Since this process was implemented, LPTFs have had positive outcomes in the number of patients picking up their ARVs within x days • Best practices shared across project

19. STOC Plan from LPTFs • The monthly analysis of missed ARV appointments can point to problems in various areas: • Data entry backlog/data entry errors • Poor communication • True defaulters

20. Data entry backlog/data entry errors Problem Defined: Computerizing patient data is long and data entry errors leading to patients erroneously appearing as defaulters. One STOC Plan (Mombasa): • 76 patients late in May but only 9 by June; patients in May were not true defaulters. The problem was due to an accumulation of backlog • Put a plan in place to use additional temporary data clerks to clear their backlog. [Temp staff must be trained in PMM system] • Eliminated backlog and true default rate determined • Implemented as solution when/if backlog arises

21. Data Quality • Scalable, robust, flexible and sustainable platforms for data collection and retrieval • Routine data cleaning and monitoring • Treatment site driven data use and analysis for adaptive management

22. Patient Monitoring and Management Systems: IQ Solutions Strategy: • To offer a library of tools and solutions built around adaptive management, quality, and sustainability • Requirements developed through practical field experience and lessons learned • Collaborative approach using local experts throughout the development process • Current areas of focus PMM, ART Registers, Data Quality Tools

23. Summary • Quality evaluation and improvement activities are a vital function of program management • The ability to scale ARV treatment programs ultimately will be dependent on efficient and sustainable care. • Sustainable care is intimately tied to achieving consistent high levels of medical care. • Data is for more than data reporting • Changing the culture of data capture and use will ensure the most effective and sustainable use of this funding

24. Thank You