Reliability of hospital abstracted data a comparison with cdac abstraction
Download
1 / 33

Reliability of Hospital-Abstracted Data: - PowerPoint PPT Presentation


  • 300 Views
  • Updated On :

Reliability of Hospital-Abstracted Data: A Comparison with CDAC Abstraction. Andrei Kuznetsov, MA MissouriPRO. 7SOW Measurement. Two parallel measurement processes: State-level Surveillance: CMS Task 1c monitoring through a random sample representative of Medicare discharges

Related searches for Reliability of Hospital-Abstracted Data:

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Reliability of Hospital-Abstracted Data:' - MartaAdara


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Reliability of hospital abstracted data a comparison with cdac abstraction l.jpg

Reliability of Hospital-Abstracted Data:A Comparison with CDAC Abstraction

Andrei Kuznetsov, MA

MissouriPRO


7sow measurement l.jpg
7SOW Measurement

  • Two parallel measurement processes:

    • State-level Surveillance: CMS Task 1c monitoring through a random sample representative of Medicare discharges

    • Hospital-level tracking: Task 2b, data contributed through ORYX Core Measures and/or CART to the QNet Exchange Clinical data repository


Measurement processes linked l.jpg

Surveillance - 1c

CMS includes record in a surveillance sample

If Yes, use the electronic record from the Repository

Repository - 2b

Hospital contributes an abstracted record

Hospital may save money by not having to copy paper chart

Measurement processes linked

A Match?


2b accuracy required l.jpg
2b: Accuracy Required

  • 7SOW RFP:

    • Hospitals that consistently perform below 80 percent reliability… will be required to provide hardcopy versions of charts when data are requested… Those hospitals performing at or above 80 percent reliability will be allowed to submit electronic versions of abstracted data when requested.

  • Operationalize accurate abstraction: agreement with ‘gold standard’=CDAC


A test case oryx pilot l.jpg
A test case: ORYX Pilot

  • MO was one of the 5 states in the Pilot of ORYX Core Measures

  • HF, AMI and Pneumonia discharges (CY 2001) were reviewed by hospitals

  • JCAHO used 6SOW inclusion/exclusion criteria for the Pilot


Oryx pilot in missouri l.jpg
ORYX Pilot in Missouri

  • 18 hospitals took part

  • AMI and Pne abstraction tools created by MissouriPRO in MedQuest

    • collected all information for the official 6SOW indicators on 2 sides of one sheet

  • HF abstraction tool designed by MPRO (Michigan QIO) as a modification the national tool (NHF)


Oryx pilot steps to ensure accuracy l.jpg
ORYX Pilot: Steps to ensure accuracy

  • Training in use of abstraction tools was conducted up front

  • A support hotline was operated

  • IRR testing was conducted as a condition of admitting an abstractor

  • Kappa=0.4 used as a threshold

  • New abstractors had to submit to IRR


Comparison with cdac data l.jpg
Comparison with CDAC data

  • For Pne (155 charts) and AMI (135 charts), comparisons can only be made at the level of a Numerator/Denominator for an indicator:

    • Example

    • AMI QI-1 Denominator:

    • Eligible for ASA at admission?

    • Hospital: Yes CDAC: Yes

    • => Agreement

      • AMI QI-1 Numerator:

      • Received ASA at admission?

      • Hospital: Yes CDAC: No

      • => Disagreement


Comparison to cdac cont d l.jpg
Comparison to CDAC, cont’d

  • For 49 variables in the HF module, comparisons could be made directly between CDAC data and hospital-generated data

  • Data available on 90 HF charts


Compare abstraction results ami l.jpg
Compare abstraction results: AMI

CDAC Provider

QI-1: ASA at admission 84% 89%

QI-2: ASA at discharge 91% 79%

QI-3: Beta Blocker at admission 61% 79%

QI-4: Beta Blocker at discharge 79% 79%

QI-5: ACEI at discharge 73% 76%

QI-6: Smoking cessation counseling 35% 37%

Data: 155 AMI charts reviewed by CDAC and providers


Compare abstraction results hf l.jpg
Compare abstraction results: HF

CDAC Provider

QI-1: Appropriate use of ACEI at discharge 87% 86%

QI-2: Appropriate use of ACEI or ARB at disch. 88% 87%

QI-3: EF Evaluated before or during admission for pts not admitted on ACEI/ARB 73% 63%

QI-4: Discharge on ACEI or documentated reason for no ACEI Rx for pts with LVSD not admitted on ACEI/ARB 60% 59%

Data: 90 HF charts reviewed by CDAC and Providers


Compare abstraction results pne l.jpg
Compare abstraction results: Pne

CDAC Providers

QI-1: Antibiotic within 8 hours 91% 96%

QI-2: Antibiotic consistent with rec’s 79% 88%

QI-3: Blood cultures before antibiotics 91% 75%

QI-5: Pneumococcal immunization screening 41% 47%

Data: 155 charts reviewed by CDAC and Providers


But agreement is low l.jpg
But agreement is low

  • Heart Failure - 90 charts, 49 variables

    • Kappa=0.40, exact agreement=84%

  • Pneumonia - 155 charts, 8 measures

    • Kappa=0.52, exact agreement=86%

  • AMI - 135 charts, 12 measures

    • Kappa=0.46, exact agreement=80%


Method of further analysis l.jpg
Method of further analysis

  • Separated agreement on denominator (was patient eligible?) from that on numerator (was treatment received?)

  • Used 2x2 tables:

    AMI QI-1 Denominator, ASA at admission

    CDAC: No CDAC:Yes

    Provider: No 58 1

    Provider: Yes 40 36


Tuna dolphin safe l.jpg
Tuna, dolphin-safe

CDAC: No CDAC:Yes

Provider: No 58 1

Provider: Yes 40 36

CDAC’s dolphin catch

Agreement: tuna

Provider’s dolphin catch


Disagreements over the denominator status l.jpg
Disagreements over the denominator status

Across multiple indicators...

  • AMI: CDAC and Provider disagreed on denominator status in 31% of cases

    • 28% of 31% were “Provider Dolphins”

  • HF: disagreement in 13% of cases

    • 12% of 13% were “Provider Dolphins”

  • Pne: disagreement in 11% of cases

    • 4% of 11% were “Provider Dolphins”


Disagreements over the numerator status l.jpg
Disagreements over the numerator status

Analyzed only cases where CDAC and Provider agreed that pt was eligible

  • Pne: 14% of cases in disagreement

  • HF: 6% of cases in disagreement

  • AMI: 10% of cases in disagreement

  • =>Disagreements not a huge problem with numerator decisions (plus, the N is smaller)


Working hypothesis l.jpg
Working hypothesis

  • Ho: More exclusion rules (screening criteria) => more opportunities for error and disagreement

  • Example: AMI QI-1, ASA at admission has 13 exclusion rules. Exclude case if:

    • transferred from another acute care hospital

    • transferred from another ER

    • UTD admission source

    • allergy to aspirin

    • bleeding on admission,

    • etc.


Exclusion rules denominator variables l.jpg
Exclusion rules - denominator variables

Number of % of “CDAC % of “Provider

Indicator variable exclusions Dolphins” Dolphins”

AMI-1 Den ASA at admission 13 1 30

AMI-2 Den ASA at discharge 18 2 47

AMI-3 Den BB at admission 15 4 22

AMI-4 Den BB at discharge 17 1 33

AMI-5 Den ACEI at discharge 18 3 32

AMI-6 Den Smoking cessation 2 6 1

HF-1 Den 5 3 8

HF-2 Den 4 0 8

HF-3 Den 6 2 13

HF-4 Den 12 0 19

Pne-1 Den 4 9 1

Pne-2 Den 9 10 8

Pne-3 Den 4 4 8

Pne-5 Den 4 5 1


Exclusion rules numerator variables l.jpg
Exclusion rules - numerator variables

Number of % of “CDAC % of “Provider

Indicator variable exclusions Dolphins” Dolphins”

AMI-1 Num ASA at admission 2 3 6

AMI-2 Num ASA at discharge 1 2 0

AMI-3 Num BB at admission 3 0 0

AMI-4 Num BB at discharge 1 0 4

AMI-5 Num ACEI at discharge 1 0 18

AMI-6 Num Smoking cessation 1 11 17

HF-1 Num 9 3 4

HF-2 Num 11 2 5

HF-3 Num 1 6 3

Pne-1 Num 2 3 7

Pne-2 Num 8 2 10

Pne-3 Num 4 14 1

Pne-5 Num 2 6 11

Note: HF-4 Num was omitted because it had only 5 cases eligible for numerator analysis





Hypothesis revisited l.jpg
Hypothesis revisited

  • Ho: More exclusion rules (screening criteria) => higher “Provider Dolphin catch” in determining patient eligibility for treatment

    • No signs of such influence on the numerator status of a case

    • No evidence of impact on “CDAC Dolphin catch” (neither for denominator nor numerator status).


Conclusions 1 l.jpg
Conclusions - 1

  • There was no evidence to place the hospitals’ integrity in doubt as far as self-abstracted data are concerned

    • However, public reporting is a whole new bowl of wax


Conclusions 2 l.jpg
Conclusions - 2

  • CDAC to Provider agreement rate ran in the 80% to 86% range (but recall our heavy investment into training and abstractor support).

    • It’s likely to be lower without the upfront training and ongoing support


Conclusions 3 l.jpg
Conclusions - 3

  • Bulk of the disagreement was over the denominator status

    • For AMI, almost 1/3 of decisions were in discord

    • For AMI and HF, a prevailing pattern is one of “Provider Dolphin catch”

    • No clear pattern for Pneumonia

    • “Provider Dolphin catch” increases as the number of exclusion criteria goes up


Conclusions 4 l.jpg
Conclusions - 4

  • Disagreement over the numerator status of a case is less common than over the denominator status (eligibility for an indicator).

    • Also, fewer cases qualify for the numerator


Suggestions 1 l.jpg
Suggestions - 1

  • Minimize the number of exclusion criteria

    • Ideally, keep exclusion rules under 3

      6SOW AMI indicators 6SOW exclusions 7SOW exclusions

      AMI-1 Den ASA at admission 13 6

      AMI-2 Den ASA at discharge 18 8

      AMI-3 Den BB at admission 15 10

      AMI-4 Den BB at discharge 17 5

      AMI-5 Den ACEI at discharge 18 9

      AMI-6 Den Smoking cessation 2 1


Suggestions 2 l.jpg
Suggestions - 2

  • Disregard the “Provider Dolphin catch” in calculation of hospital error rate

    • This is a “productive mistake” as opposed to a “counter-productive mistake”


Roadblocks l.jpg
Roadblocks

  • No JCAHO mandate to prove accuracy

  • No QIO funding to train abstractors

  • Plan for hospital-level (not abstractor-level) tracking of accuracy

  • High turnover rate for abstractors


Contact info l.jpg
Contact info

Andrei Kuznetsov

MissouriPRO

573-893-7900, ext. 163

[email protected]


ad