Acknowledgements

1. Development and Use of Ambulatory Adverse Event Trigger ToolsAmy K. Rosen, PhDAHRQ Conference Sept. 29, 2010Boston University School of Public Health, Boston, MA, VA Center for Organization, Leadership and Management Research (COLMR), Boston MAakrosen@bu.edu

2. Acknowledgements Sponsored by AHRQ Contract No. HHSA290200600012, Task Order Officer: Amy Helwig, MD • PI Amy Rosen, PhD • Co-PI Jonathan Nebeker, MD, MS • Co-Investigators: • Stephan Gaehde, MD • Haytham Kaafarani, MD, MPH • Brenna Long, MA • Hillary Mull, MPP • Brian Nordberg, BS • Steve Pickard, MS • Peter Rivard, PhD • Lucy Savitz, PhD, MBA • Chris Shanahan, MD, MPH • Stephanie Shimada, PhD

3. Project Goal and Settings • Goal: Develop adverse event (AE) triggers for the outpatient setting • Outpatient surgery AEs • Outpatient diagnostic testing and loss to follow-up AEs • Outpatient adverse drug events (ADEs) • Three sites for patient data: • Boston Medical Center (BMC) • Intermountain Healthcare • Western region of the Veterans Health Administration (VA)

4. Background • Trigger tool: A screen applied to healthcare data that triggers review of a patient record for a potential adverse event. • Trigger: An algorithm that uses electronic medical record data to identify patterns consistent with a possible adverse event. Example: combination of a lab value threshold and an active prescription. • AE specific trigger: A trigger intended to identify a specific adverse event.

5. Methods List of triggers Inter-mountain VA BMC Project database: Inpatient, outpatient, vitals, demographics, lab, pharmacy, notes Program trigger algorithms Trigger-flagged cases Mock EMR Chart Review

6. Methods: Outpatient Surgery Triggers • Sample: • N=17,492 ambulatory surgeries from three institutions in CY05 • Sampling for chart review: 17 trigger-flagged cases from each site to the extent the data were available. • Definition of AE: • NSQIP AEs: Surgeries with an event that meets the National Surgical Quality Improvement Program criteria for an AE (e.g., postoperative urinary tract infection). • Other AE: Surgery with an AE determined by nurse clinical judgment (e.g., postoperative iatrogenic injury). • Analysis: • PPV = number of patients with AE/number of positive triggers

7. Methods: Loss to Follow-up Trigger • Definition of AE: • Failure to follow-up on abnormal fecal occult blood test (FOBT) results, which could potentially lead to a failure to detect colorectal cancer or pre-cancerous growths in a timely manner. • In the event a patient has a positive FOBT, the standard of care is a colonoscopy within 2 months. • We designed the trigger to exclude bleeding that might potentially be due to a recently diagnosed ulcer or GI surgery. • Analysis: • PPV was calculated for cases without a colonoscopy and cases missing an appropriate colonoscopy.

8. Methods: Focus Groups • We conducted 90-minute focus group and multiple 30-45 minute interviews at each of the sites between October and December 2009. • Participants were frontline staff, middle managers and executives including: • Physicians (medicine and surgery) • Nurses • Informatics and information systems experts • Pharmacists • Quality and patient safety experts

9. Results: Outpatient Surgery Triggers

10. Results: Outpatient Surgery Triggers

11. Results: Loss to Follow-up Trigger • Sample: • N=995 cases with positive FOBT tests in CY05 from one institution

12. Results: Focus Group/Interviews Triggers ranked from highest to lowest likelihood of adoption

13. Results: Focus Group/Interviews Triggers ranked from highest to lowest perceived ease of implementation

14. Summary • The set of surgical triggers (excluding ‘Hematocrit’) required review of 454 cases and detected at least one AE in 204 ambulatory surgeries (PPV for any AE=45%). • Many surgeries had more than one AE detected. • Triggers best suited for adoption: • Admission: flag rate=22%; PPV for detection of any AE=41%; popular in focus groups • PE/DVT: flag rate=1%; PPV for detection of any AE=65%; very popular in focus groups • FOBT: flag rate=8%; PPV for detection of any cases without colonoscopy after 6 months=66%; very popular in focus groups

15. Implications • Triggers may have potential to screen for outpatient AEs using a focused sample of cases. • Triggers could complement existing screening programs used to detect AEs. • Potential for real-time AE detection, particularly with electronic medical records.

16. Dissemination to Date • Rosen AK & Nebeker JR, Use of Trigger Tools to Identify Risks and Hazards to Patient Safety, Sept. 2010. Speakers, AHRQ 2010 Annual Conference, Bethesda, MD. • Long BL, Pickard S, Mull HJ, Hoffman JM, Rosen AK & Nebeker JN, Reliability of AHRQ Harm Scale Used with Explicit Criteria in Retrospective ADE Classification, Sept. 2010. Poster, AHRQ 2010 Annual Conference, Bethesda, MD. • Shimada S, Mull HJ, Kaafarani H, Rosen AK, Nebeker JR, Nordberg B, Pickard S & Singh H. Development and Assessment of a Trigger Tool to Detect Patients with Positive Fecal Occult Blood Test (FOBT) Results Who Are Lost to Follow-Up. Accepted poster, AMIA 2010 National Symposium, Nov 2010, Washington, DC. • Kaafarani H, Rosen AK, Nebeker JR, Shimada SL, Mull HJ, Rivard PE, Savitz L, Helwig A, Shin MH & Itani KMF. Development of Trigger Tools for Surveillance of Adverse Events in Ambulatory Surgery, 2010. Quality and Safety in Health Care. [E-pub ahead of print] http://www.ncbi.nlm.nih.gov/pubmed/20513790 • Mull H & Nebeker JR, Informatics Tools for the Development of Action-Oriented Adverse Drug Event Triggers, 2008. AMIA Annual Symposium Proceedings. Nov 6:505-9. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2655939/ • Mull HJ, Nordberg B, Nebeker JR, Using Electronic Medical Records Data for Health Services Research, Case Study: Development and Use of Ambulatory Adverse Event Trigger Tools. Speaker, AcademyHealth Annual Research Meeting, June 2010, Boston, MA. 

17. Dissemination to Date (cont’d) • Shimada S, Rivard P, Nebeker JR, Savitz L, Shanahan C, Gaehde S & Rosen AK. Priorities & Preferences of Potential Ambulatory Trigger Tool Users. Speaker, AcademyHealth Annual Research Meeting, June 2009, Chicago, IL. • Kaafarani H, Rosen AK, Nebeker JR, Shimada SL, Rivard PE, Mull HJ, Long B, Shin MH, Savitz L & Itani KMF. Developing Trigger Tools for Surveillance of Adverse Events in Same-Day Surgery: A Literature-Based, End-User Inspired and Expert-Evaluated Methodology, Sept 2009.Poster,AHRQ 2009 Annual Meeting, Bethesda, MD. • Mull HJ, Shimada S, Nebeker JR & Rosen AK, A Review of the Trigger Literature:  Adverse Events Targeted and Gaps in Detection,June 2008. Proceedings of the Trigger and TIDS Expert Meeting, Agency for Healthcare Research and Quality, Bethesda, MD. http://www.ahrq.gov/qual/triggers/triggers1.htm • Nebeker JR, Stoddard GJ & Rosen AK, Considering Sensitivity and Positive Predictive Value in Comparing the Performance of Triggers Systems for Iatrogenic Adverse Events, Proceedings of the Trigger and TIDS Expert Meeting, Agency for Healthcare Research and Quality, June 2008. Bethesda, MD. http://www.ahrq.gov/QUAL/triggers/triggers2.htm • Shimada S, Rivard PE, Mull HJ, Nebeker JR & Rosen AK, Triggers and Targeted Injury Detection Systems: Aiming for the Right Target With the Appropriate Tool, June 2008. Proceedings of the Trigger and TIDS Expert Meeting, Agency for Healthcare Research and Quality, Bethesda, MD. http://www.ahrq.gov/QUAL/triggers/triggers3.htm