POSITIVE PREDICTIVE VALUE OF AHRQ PATIENT SAFETY INDICATORS IN A NATIONAL SAMPLE OF HOSPITALS

1. POSITIVE PREDICTIVE VALUE OF AHRQ PATIENT SAFETY INDICATORS IN A NATIONAL SAMPLE OF HOSPITALS AcademyHealth Annual Research Meeting June 9, 2008 Team presenter: Pat Zrelak PhD, CNRN, CNAA-BC1 Team: Patrick Romano1; Garth Utter1; Richard White1; Dan Tancredi1; Ruth Baron1; Banafsheh Sadeghi; Sheryl Davies; Jeff Geppert2. 1 University of California Davis, Sacramento, CA. 2 Battelle Memorial Institute, Elk Grove, CA.

2. Background of PSIs • Set of quality indicators designed to identify potentially preventable problems that patients may experience as a result of contact with the health care system • Based on severity adjusted inpatient hospital discharge data • Initially developed through a contract with UC-Stanford Evidence-based Practice Center • Little is known about the criterion validity of the PSIs across multiple hospitals

3. AHRQ PSI Validation Pilot Goals • Gather evidence on the criterion validity of the PSIs based on medical record review • Improve guidance about how to interpret & use the data • Evaluate potential refinements to the specifications • Develop medical record abstraction tools • Develop mechanisms for conducting validation studies on a routine basis

4. Positive Predictive Value • The positive predictive value or post-test probability is the proportion of flagged cases who actually had the event. • The Positive Predictive Value (PPV) can be further defined as:

5. Methods • Retrospective cross-sectional study design • Volunteer sample of collaborative partners • Facilitating organizations (e.g., Arizona) • Hospital systems • Individual hospitals • Sampling based on administrative data • Sampling probabilities assigned using AHRQQI software to generate desired sample size nationally

6. Data collection methods • Each hospital identified chart abstractors • Training occurred via webinars • Medical record abstraction tools & guidelines • Pretested in the Sacramento area • Targeted the ascertainment of the event, risk factors, evaluation & treatment, and related outcomes

7. Timeline • 10 indicators- divided into 2 phases of 5 each • Phase I review- • Training early 2007 • Chart review 4 month process • 4th Qtr 2005, 2006, & 1st Qtr 2007 • Phase II review – • Waiting OMB approval • Phase III –sensitivity determination

8. Patient Safety Indicators

9. Medical record sample

10. Accidental Puncture or Laceration • N=249 • 90% of events occurred during the hospitalization • 10% were false positives • 8% identified by abstractor as miscoded • 2% present on admission

11. Iatrogenic Pneumothorax • N=205 • 89% of events occurred during the hospitalization • 11% were false positives • 7% present or suspected at admission • 4% no documentation of event (miscoded)

12. Postoperative DVT or PE • N=155 • 68% occurred during the index hospitalization • 32% were false positives • 16% had no surgical procedure performed in the OR • 16% did not have a new post-op PE or DVT

13. N=194 61% occurred during the index hospitalization 39% were false positives 17% were present on admission 22% had no documentation of an infection Selected Infection due to Medical Care

14. Postoperative Sepsis • N=164 • 40% of the events occurred during the hospitalization • 60% were false positive • 17% had no documentation of bacteremia, septicemia, sepsis or SIRS • 17% had infection (=14%) or sepsis (=3%) POA • 25% did not have elective surgery

15. Summary of PPVsPreliminary estimates

16. Recognizing limitations • Data elements available via chart review • Time constraints (burden on collaborators) • Inter-hospital variation • Volunteer sample

17. AHRQQI Validation PilotNext steps for analysis • Further analysis of: • Potential preventability • Management/treatment & patient outcomes • Inter-hospital variation • Evaluation of alternative ICD-9-CM specifications • Can we improve PPV through numerator or denominator changes?

18. Policy implications • “Present at admission” data would substantially improve the PPV • Current PSI software has POA option • October 2008 release will require POA • Coding changes are needed to enhance specificity and PPV in some areas • AHRQ proposed codes for upper extremity and thoracic venous thromboses, and to distinguish acute from sub-acute thromboses • New codes for catheter-associated bloodstream infection • Several PSIs have been endorsed by NQF • Accidental Puncture & Laceration (phase I) • Iatrogenic pneumothorax (phase I) • Foreign body (phase II) • Wound dehiscence (phase II)

19. Acknowledgments • AHRQ project team • MamathaPancholi &Marybeth Farquhar • Battelle training and support team • Laura Puzniak & Lynne Jones • All of the validation pilot partners!