State of AHRQ Part II Improvements in Patient Safety: The Future is Now

1. State of AHRQ � Part IIImprovements in Patient Safety: The Future is Now AHQA Annual Meeting & Technical Conference March 12, 2004 I will take a few minutes to talk about how we can benefit from the large amount existing administrative data. I will take a few minutes to talk about how we can benefit from the large amount existing administrative data.

2. Session Objectives Describe the major components of AHRQ�s patient safety portfolio and their applicability to QIOs Describe AHRQ�s Patient Safety Indicators (PSIs) and their utility for identifying risk and hazards to patient safety Describe the elements of �safety culture� and explain how to use the safety culture assessment tool developed by the Quality Interagency Coordination (QuIC Task Force)

3. Congressional Mandate for AHRQ identify the causes of preventable health care errors and pat injury in health care delivery; develop, demonstrate, and evaluate strategies for reducing error and improving patient safety; and disseminate such effective strategies throughout the health care industry.

4. Congressional Funding for Patient Safety FY 01 $50 Million FY 02 $55 Million FY 03 $55 Million FY 04 $79.5 Million FY 05 $84 Million (proposed)

5. Funding Program Areas Identifying risk and hazards Reporting System Demonstrations (16 projects) Working Conditions (22 projects) Building Capacity Centers of Excellence in Patient Safety Research (3 centers) Developing Centers of Excellence in Patient Safety Research (18 projects) Patient Safety Improvement Corps

6. Funding Program Areascontinued Raising awareness Dissemination and education (6 grants) User Liaison Program with states Conferences and workshops Identify proven patient safety practices System best practices (6 projects) Computer application (11 projects) EPC report on evidence based patient safety practices

7. Funding Program Areascontinued Challenge patient safety improvements Risk assessment (6 projects) Implementing safe practices (7 projects) Transforming through information technology FY 04 initiative with 3 RFAs focusing on: implementing technologies, demonstrating value of technologies, and planning for technology implementation Receipt date: 04/22/04 Applications at: www.ahrq.gov, funding opportunities

8. Porter Question SO WHAT? How can AHRQ�s research improve quality and patient safety?

9. Tools and Products Software Web-based Tools Survey Instruments Educational Interventions

10. Software Decision support ADHD HIV management Diagnostic errors in emergency cardiac care Electronic event detection methods Risk factor identification Falls and delirium Adverse event chart review MedDecide: Decision support software suite for PDAs running Palm-OS (Berner) WARFDOCS: Decision support software for PDAs running Palm-OS to predict accurate dosing of Warfarin and clinical response (White) BARTRACE: Decision support software to aid with outpatient anticoagulation care (White) DS-ADHD: A point-of-care decision-support system for use in primary care settings (Lozano) Software that incorporates genotype information into an electronic medical record system (Novak) MedDecide: Decision support software suite for PDAs running Palm-OS (Berner) WARFDOCS: Decision support software for PDAs running Palm-OS to predict accurate dosing of Warfarin and clinical response (White) BARTRACE: Decision support software to aid with outpatient anticoagulation care (White) DS-ADHD: A point-of-care decision-support system for use in primary care settings (Lozano) Software that incorporates genotype information into an electronic medical record system (Novak)

11. Web-based Tools Website on infusion devices Endoscopic sinus surgery simulator Voluntary event reporting MisssouriPro patient safety information Communicating with patient and families Diagnostic errors Safety Toolkit for clinicians and patients Information on infusion devices, including the design of infusion devices (Cook) Information on Endoscopic Sinus Surgery Simulator (Fried) Identifying and reducing errors with surgical simulation: Database available on the web (Fried) EQuIP for Quality: Web based resource for continuous quality improvement, endorsed by the American Association of Homes and Services for the Aging. The resource synthesizes knowledge derived from up-to-date standards of care, protocols and best practices and visually represents the knowledge in useful formats, including individualized resident risk profiles, quality indicator reviews and error identification (Teigland) Information on infusion devices, including the design of infusion devices (Cook) Information on Endoscopic Sinus Surgery Simulator (Fried) Identifying and reducing errors with surgical simulation: Database available on the web (Fried) EQuIP for Quality: Web based resource for continuous quality improvement, endorsed by the American Association of Homes and Services for the Aging. The resource synthesizes knowledge derived from up-to-date standards of care, protocols and best practices and visually represents the knowledge in useful formats, including individualized resident risk profiles, quality indicator reviews and error identification (Teigland)

12. Survey Instruments Handheld computer barrier survey Evaluating medication safety practices in the ambulatory care setting Regional patient safety: staff and management Communicating about errors: physicians, pediatric, clinical staff Leadership self-assessments Handheld Computer Barrier Survey: An instrument to measure attitudes toward the use of handheld computers in ambulatory settings (Berner) Handheld technology adoption assessment tool: Instrument to assess likelihood of adopting handheld technology for daily clinical use (Galt) Handheld Computer Barrier Survey: An instrument to measure attitudes toward the use of handheld computers in ambulatory settings (Berner) Handheld technology adoption assessment tool: Instrument to assess likelihood of adopting handheld technology for daily clinical use (Galt)

13. Educational Interventions Anticipating the human factors of next generation infusion devices Inter-professional curriculum on patient safety for health professionals Statewide hospital summits Integrating risk management with patient safety Falls management program for nursing facilities Anticipating the Human Factors of next Generation Infusion Devices: A colloquium held at HFES meeting on the impact of emerging technology on the design of infusion devices (Cook) Dissemination and promotion of the new software system to pharmacists, physicians, and other facility staff (Lapane) Anticipating the Human Factors of next Generation Infusion Devices: A colloquium held at HFES meeting on the impact of emerging technology on the design of infusion devices (Cook) Dissemination and promotion of the new software system to pharmacists, physicians, and other facility staff (Lapane)

14. Patient Safety Research Using Administrative Data Chunliu Zhan Agency for Healthcare Research Quality March , 2004 I will take a few minutes to talk about how we can benefit from the large amount existing administrative data. I will take a few minutes to talk about how we can benefit from the large amount existing administrative data.

15. By-product of administering/reimbursing health services (also called claims data) Common data elements: admission date, discharge date and status, primary and varying numbers of secondary ICD-9-CM diagnosis and procedure codes,DRG, payment, demographic Available from Government payers (Medicare, Medicaid, Veterans Affairs) and private insurance companies Administrative Data Briefly, administrative data, also called claims data, are regularly collected and maintained primarily for reimbursement purpose. Common data� Available, The data usually compiled for research use. HCUP is a collection of all hospital claims from about 30 states.Briefly, administrative data, also called claims data, are regularly collected and maintained primarily for reimbursement purpose. Common data� Available, The data usually compiled for research use. HCUP is a collection of all hospital claims from about 30 states.

16. 1970s: Wennberg�s pioneer work on small-area variation in practice patterns 1980s: Outcomes & burden of illness Early 1990s: Iezzoni and colleagues� Complication Screening Program (CSP) Mid 1990s: AHRQ Quality Indicators (QI) 2002: AHRQ Patient Safety Indicators (PSI) Administrative Data & Patient Safety Research Starting from early 1970s, claims data have been used to study practice patterns, In the 1980s, many used claims to study outcomes and burden of illness Iezzoni and colleagues work on CSP in early 90s is the first systematic exploration of claims data in quality of care research. CSP uses ICD-9-CM codes to identify 27 potentially preventable in-hospital complications I mentioned earlier that AHRQ compiled all claims from up to 30 states into a research database. In mid 1980s, many states wondered whether we can use this database to study quality of care. In response, AHRQ developed a set of QI for claims data, which included 33 indicators. Jack Needleman and colleagues in Harvard used some QIs to study the effect of nurse staffing on quality, published in NEJM. After the IOM report in late 1999, we at AHRQ made some efforts to develop tools specifically for studying safety problems using claims data. We developed an initial list, and then had UCSF-Stanford EPC to further expand, validate, and build expert consensus. The product is the AHRQ PSI.Starting from early 1970s, claims data have been used to study practice patterns, In the 1980s, many used claims to study outcomes and burden of illness Iezzoni and colleagues work on CSP in early 90s is the first systematic exploration of claims data in quality of care research. CSP uses ICD-9-CM codes to identify 27 potentially preventable in-hospital complications I mentioned earlier that AHRQ compiled all claims from up to 30 states into a research database. In mid 1980s, many states wondered whether we can use this database to study quality of care. In response, AHRQ developed a set of QI for claims data, which included 33 indicators. Jack Needleman and colleagues in Harvard used some QIs to study the effect of nurse staffing on quality, published in NEJM. After the IOM report in late 1999, we at AHRQ made some efforts to develop tools specifically for studying safety problems using claims data. We developed an initial list, and then had UCSF-Stanford EPC to further expand, validate, and build expert consensus. The product is the AHRQ PSI.

17. One of the Quality Indicator Sets: Inpatient Quality Indicator (IQI) Prevention Quality Indicator (PQI) AHRQ PSIs: conservatively identify �never� events are indicators not definitive measures use with administrative data Reporting at institutional-level QI, not public level AHRQ Patient Safety Indicators (PSIs) Starting from early 1970s, claims data have been used to study practice patterns, In the 1980s, many used claims to study outcomes and burden of illness Iezzoni and colleagues work on CSP in early 90s is the first systematic exploration of claims data in quality of care research. CSP uses ICD-9-CM codes to identify 27 potentially preventable in-hospital complications I mentioned earlier that AHRQ compiled all claims from up to 30 states into a research database. In mid 1980s, many states wondered whether we can use this database to study quality of care. In response, AHRQ developed a set of QI for claims data, which included 33 indicators. Jack Needleman and colleagues in Harvard used some QIs to study the effect of nurse staffing on quality, published in NEJM. After the IOM report in late 1999, we at AHRQ made some efforts to develop tools specifically for studying safety problems using claims data. We developed an initial list, and then had UCSF-Stanford EPC to further expand, validate, and build expert consensus. The product is the AHRQ PSI.Starting from early 1970s, claims data have been used to study practice patterns, In the 1980s, many used claims to study outcomes and burden of illness Iezzoni and colleagues work on CSP in early 90s is the first systematic exploration of claims data in quality of care research. CSP uses ICD-9-CM codes to identify 27 potentially preventable in-hospital complications I mentioned earlier that AHRQ compiled all claims from up to 30 states into a research database. In mid 1980s, many states wondered whether we can use this database to study quality of care. In response, AHRQ developed a set of QI for claims data, which included 33 indicators. Jack Needleman and colleagues in Harvard used some QIs to study the effect of nurse staffing on quality, published in NEJM. After the IOM report in late 1999, we at AHRQ made some efforts to develop tools specifically for studying safety problems using claims data. We developed an initial list, and then had UCSF-Stanford EPC to further expand, validate, and build expert consensus. The product is the AHRQ PSI.

18. AHRQ PSI (20 indicators) AHRQ has 20 indicators, here is the list. You should have a copy of this table in your handout. This table shows the number of cases identified in all claims from 924 short term general hospitals from 30 states in year 2000, approximating a 20% US hospitals. Each PSI has its unique risk pool.AHRQ has 20 indicators, here is the list. You should have a copy of this table in your handout. This table shows the number of cases identified in all claims from 924 short term general hospitals from 30 states in year 2000, approximating a 20% US hospitals. Each PSI has its unique risk pool.

19. Large number of records Continuous National coverage Cheap Power for studying rare events Administrative Data: Merits The merits: using HCUP as example Large number of records: we can access 7.5 millions claims Continuous: HCUP started from 1983, every year National coverage: now 30 states Cheap compared to other sources of data Only with this kind of data it is possible to identify 536 foreign body left The merits: using HCUP as example Large number of records: we can access 7.5 millions claims Continuous: HCUP started from 1983, every year National coverage: now 30 states Cheap compared to other sources of data Only with this kind of data it is possible to identify 536 foreign body left

20. ICD-9-CM coding: Incomplete, coding errors, coding variation across hospitals, �DRG creep� PSI: high false negative, low false positive Limited clinical details for risk analysis and risk adjustment Small but statistically significant findings Many potential sources of bias Administrative Data: Flaws The problems: ICD-9-CM codes Validity and reliability in identify medical errors: we can only identify errors if corresponding ICD-9-CM codes exist. High false negative, low false positive and good specificity Lack of clinical details: a. can�t identify risk factors, b. inadequate risk adj Because large data size, we could find small, clinically meaningless but statistically significant results. If you look at the CI for risk pool, SE are very small. If you compare LOS between cases with obstetric trauma with instrumentation and risk pool cases, a difference of 0.014 day is statistically significant Many potential bias. One not well-understood is bias associated with the small stat sig difference, such as to explain what contribute to the 0.014 days differences, The problems: ICD-9-CM codes Validity and reliability in identify medical errors: we can only identify errors if corresponding ICD-9-CM codes exist. High false negative, low false positive and good specificity Lack of clinical details: a. can�t identify risk factors, b. inadequate risk adj Because large data size, we could find small, clinically meaningless but statistically significant results. If you look at the CI for risk pool, SE are very small. If you compare LOS between cases with obstetric trauma with instrumentation and risk pool cases, a difference of 0.014 day is statistically significant Many potential bias. One not well-understood is bias associated with the small stat sig difference, such as to explain what contribute to the 0.014 days differences,

21. Screening Tool: Iatrogenic Pneumothorax (n=3931): ID cases to guide medical records abstraction and in-depth analysis Risk Factors: Guide medical record abstraction to study root cause Patients admitted for pleurisy and undergone thoracentesis have high risk for developing iatrogenic pneumothorax Potential Uses I & II: Screening Tool and Risk Factors How we can take advantage of administrative data in patient safety? I listed 5 potential uses 1. With tools like AHRQ PSIs, we can screen cases of potential safety concerns. For example:How we can take advantage of administrative data in patient safety? I listed 5 potential uses 1. With tools like AHRQ PSIs, we can screen cases of potential safety concerns. For example:

22. Foreign body Left (n=536): Extrapolate to 2,700 US cases; 0.002% in medical admissions and 0.024% in surgical admissions, in 2002. Nosocomial infection (n=11,449): Extrapolate to 5,700 US cases; 0.147% in medical admissions and 0.307% in surgical admissions, in 2002. Potential Use III: Incidence tracking 2. Incidence or prevalence study. For example, we were able to identify �2. Incidence or prevalence study. For example, we were able to identify �

23. Possible for tracking at state/national level; used in the annual National Healthcare Quality Report (NHQR) Challenges: Incomplete, erroneous ICD-9-CM codes, coding variation across hospitals, limited clinical details and risk adjusters Not recommended for provider comparison Potential Use IV: Public Reporting We desperately need a national medical error reporting system. Administrative data is not the answer. We may use to get some overall impression. We are working on the first annual NHQR mandated by congress. Safety is apparently a key quality issue but we have no national data. So we used PSIs applied to all available states. Not recommended for comparing hospitals.We desperately need a national medical error reporting system. Administrative data is not the answer. We may use to get some overall impression. We are working on the first annual NHQR mandated by congress. Safety is apparently a key quality issue but we have no national data. So we used PSIs applied to all available states. Not recommended for comparing hospitals.

24. Foreign Body Left: 2.08 extra days $13,000 extra charges 2.14% excess mortality Nosocomial infection: 9.58 extra days $39,000 extra charges 4.31% excess mortality Potential Use V: Impacts We can study adverse effects of errors on patients. Example� I obtained this result by matching patients with and without errors to the same hospitals, same DRGs, comorbidity, and patient factors. Following Dr Bates approach to adverse effects on adverse drug reactions�We can study adverse effects of errors on patients. Example� I obtained this result by matching patients with and without errors to the same hospitals, same DRGs, comorbidity, and patient factors. Following Dr Bates approach to adverse effects on adverse drug reactions�

25. Potential Use V: Impacts (continued) Total Impact: 18 of the 20 PSIs (excluding death in low mortality DRGs and failure to rescue) ? 2.4 million hospital days $9.2 billion 32,600 deaths attributable

26. Research Resources Chunliu Zhan, Marlene Miller �Excess Length of Stay, Costs, and Mortality Attributable to Medical Injuries during Hospitalization: An Administrative Data-Based Analysis� JAMA. October 8, 190(14), 1868-1874, 2003 Chunliu Zhan, Marlene Miller �Administrative Data-Based Patient Safety Research: A Critical Review� Quality and Safety of Health Care. 12(Suppl 2), 58-63, 2003

28. Hospital Survey on Patient Safety Jim Battles Agency for Healthcare Research Quality March , 2004

29. Culture and Safety

30. What is Safety Culture The safety culture is the product of individuals and group values, attitudes, perceptions, competencies, and patterns of behavior that determine the commitment to, and the style and proficiency of, an organization�s health and safety management. A positive safety culture is characterized by communications founded on mutual trust, by shared perceptions of the importance of safety, and by confidence in the efficacy of preventive measures.

31. Measuring Safety Culture The IOM and other organizations have stressed the need to measure the safety culture in healthcare There have been a number of attempts to develop such a measure including efforts within the federal government However, no valid and reliable instrument existed within the public domain that could be used with confidence

32. Survey Purpose and Instrument Development The survey is intended to help hospitals assess the extent to which their cultures emphasize the importance of patient safety, facilitate open discussion of error, encourage error reporting, and create an atmosphere of continuous learning and improvement The Quality Interagency Coordination Task Force (QuIC) Medical Errors Workgroup sponsored the development which required OMB clearance AHRQ funded the project under contract to Westat

33. Steps in Development A literature review pertaining to safety, error and accidents, and error reporting was conducted. Hospital employees and managers were interviewed to identify key patient safety and error reporting issues. Existing published and unpublished safety culture assessment tools were reviewed. Psychometric analysis of two federally funded instruments from the VA and the NHLBI funded MERS-TM project Testing was conducted to determine reliability and usability of the instrument. All dimensions of the instrument were shown to have acceptable levels of reliability (defined as Cronbach�s alpha equal to or greater than 0.60

34. Measures Overall Outcomes Overall perceptions of safety (.74) Overall patient safety grade (.84)

35. Dimensions of Patient Safety Culture Supervisor/manager expectations & actions promoting patient safety (.75) Organizational learning--Continuous improvement (.76) Teamwork within units (.83) Communication openness (.72) Feedback & communication about error (.78) Non-punitive response to error (.79) Staffing (.63) Hospital management support for patient safety (.83) Teamwork across hospital units (.80) Hospital handoffs & transitions (.80)

36. Dimensions of Patient Safety Culture Each of these dimensions serves as a different component of an organization�s safety culture Safety culture can be defined as the set of values, beliefs, and norms about what�s important, how to behave, and what attitudes are appropriate when it comes to patient safety in a work group.

37. The QuIC Instrument Comparison The QuIC instrument is a public domain instrument and is intended for free use by institutions Relatively short with 51 items - easily be completed by hospital personnel without undue burden Several instruments are longer with some shorter but none as reliable Outstanding psychometric properties that have been fully tested

38. The QuIC Instrument Comparisoncontinued Measures culture across ten domains and two outcome measures which are all extremely reliable (with Chronbach�s alpha >.06) Several other instruments do not contain separate domains that have demonstrated reliability Others must be analyzed only as single items rather than as underlying dimensions Other instruments have not been as comprehensive

40. Thank You Jbattles@ahrq.gov ehogan@ahrq.gov czhan@ahrq.gov Agency for Healthcare Research and Quality Center for Quality Improvement and Patient Safety 540 Gaiter Road, Rockville, MD 20850