Reliability of Hospital-Abstracted Data: A Comparison with CDAC Abstraction - PowerPoint PPT Presentation

Reliability of hospital abstracted data a comparison with cdac abstraction l.jpg
Download
1 / 33

  • 289 Views
  • Updated On :
  • Presentation posted in: Pets / Animals

Reliability of Hospital-Abstracted Data: A Comparison with CDAC Abstraction. Andrei Kuznetsov, MA MissouriPRO. 7SOW Measurement. Two parallel measurement processes: State-level Surveillance: CMS Task 1c monitoring through a random sample representative of Medicare discharges

Related searches for Reliability of Hospital-Abstracted Data: A Comparison with CDAC Abstraction

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Reliability of Hospital-Abstracted Data: A Comparison with CDAC Abstraction

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Reliability of hospital abstracted data a comparison with cdac abstraction l.jpg

Reliability of Hospital-Abstracted Data:A Comparison with CDAC Abstraction

Andrei Kuznetsov, MA

MissouriPRO


7sow measurement l.jpg

7SOW Measurement

  • Two parallel measurement processes:

    • State-level Surveillance: CMS Task 1c monitoring through a random sample representative of Medicare discharges

    • Hospital-level tracking: Task 2b, data contributed through ORYX Core Measures and/or CART to the QNet Exchange Clinical data repository


Measurement processes linked l.jpg

Surveillance - 1c

CMS includes record in a surveillance sample

If Yes, use the electronic record from the Repository

Repository - 2b

Hospital contributes an abstracted record

Hospital may save money by not having to copy paper chart

Measurement processes linked

A Match?


2b accuracy required l.jpg

2b: Accuracy Required

  • 7SOW RFP:

    • Hospitals that consistently perform below 80 percent reliability… will be required to provide hardcopy versions of charts when data are requested… Those hospitals performing at or above 80 percent reliability will be allowed to submit electronic versions of abstracted data when requested.

  • Operationalize accurate abstraction: agreement with ‘gold standard’=CDAC


A test case oryx pilot l.jpg

A test case: ORYX Pilot

  • MO was one of the 5 states in the Pilot of ORYX Core Measures

  • HF, AMI and Pneumonia discharges (CY 2001) were reviewed by hospitals

  • JCAHO used 6SOW inclusion/exclusion criteria for the Pilot


Oryx pilot in missouri l.jpg

ORYX Pilot in Missouri

  • 18 hospitals took part

  • AMI and Pne abstraction tools created by MissouriPRO in MedQuest

    • collected all information for the official 6SOW indicators on 2 sides of one sheet

  • HF abstraction tool designed by MPRO (Michigan QIO) as a modification the national tool (NHF)


Oryx pilot steps to ensure accuracy l.jpg

ORYX Pilot: Steps to ensure accuracy

  • Training in use of abstraction tools was conducted up front

  • A support hotline was operated

  • IRR testing was conducted as a condition of admitting an abstractor

  • Kappa=0.4 used as a threshold

  • New abstractors had to submit to IRR


Comparison with cdac data l.jpg

Comparison with CDAC data

  • For Pne (155 charts) and AMI (135 charts), comparisons can only be made at the level of a Numerator/Denominator for an indicator:

    • Example

    • AMI QI-1 Denominator:

    • Eligible for ASA at admission?

    • Hospital: YesCDAC: Yes

    • => Agreement

      • AMI QI-1 Numerator:

      • Received ASA at admission?

      • Hospital: YesCDAC: No

      • => Disagreement


Comparison to cdac cont d l.jpg

Comparison to CDAC, cont’d

  • For 49 variables in the HF module, comparisons could be made directly between CDAC data and hospital-generated data

  • Data available on 90 HF charts


Compare abstraction results ami l.jpg

Compare abstraction results: AMI

CDACProvider

QI-1: ASA at admission84% 89%

QI-2: ASA at discharge91% 79%

QI-3: Beta Blocker at admission61% 79%

QI-4: Beta Blocker at discharge79% 79%

QI-5: ACEI at discharge73% 76%

QI-6: Smoking cessation counseling35% 37%

Data: 155 AMI charts reviewed by CDAC and providers


Compare abstraction results hf l.jpg

Compare abstraction results: HF

CDACProvider

QI-1: Appropriate use of ACEI at discharge87% 86%

QI-2: Appropriate use of ACEI or ARB at disch.88% 87%

QI-3: EF Evaluated before or during admission for pts not admitted on ACEI/ARB73% 63%

QI-4: Discharge on ACEI or documentated reason for no ACEI Rx for pts with LVSD not admitted on ACEI/ARB60%59%

Data: 90 HF charts reviewed by CDAC and Providers


Compare abstraction results pne l.jpg

Compare abstraction results: Pne

CDACProviders

QI-1: Antibiotic within 8 hours91%96%

QI-2: Antibiotic consistent with rec’s79%88%

QI-3: Blood cultures before antibiotics91%75%

QI-5: Pneumococcal immunization screening41%47%

Data: 155 charts reviewed by CDAC and Providers


But agreement is low l.jpg

But agreement is low

  • Heart Failure - 90 charts, 49 variables

    • Kappa=0.40, exact agreement=84%

  • Pneumonia - 155 charts, 8 measures

    • Kappa=0.52, exact agreement=86%

  • AMI - 135 charts, 12 measures

    • Kappa=0.46, exact agreement=80%


Method of further analysis l.jpg

Method of further analysis

  • Separated agreement on denominator (was patient eligible?) from that on numerator (was treatment received?)

  • Used 2x2 tables:

    AMI QI-1 Denominator, ASA at admission

    CDAC: No CDAC:Yes

    Provider: No581

    Provider: Yes4036


Tuna dolphin safe l.jpg

Tuna, dolphin-safe

CDAC: No CDAC:Yes

Provider: No581

Provider: Yes4036

CDAC’s dolphin catch

Agreement: tuna

Provider’s dolphin catch


Disagreements over the denominator status l.jpg

Disagreements over the denominator status

Across multiple indicators...

  • AMI: CDAC and Provider disagreed on denominator status in 31% of cases

    • 28% of 31% were “Provider Dolphins”

  • HF: disagreement in 13% of cases

    • 12% of 13% were “Provider Dolphins”

  • Pne: disagreement in 11% of cases

    • 4% of 11% were “Provider Dolphins”


Disagreements over the numerator status l.jpg

Disagreements over the numerator status

Analyzed only cases where CDAC and Provider agreed that pt was eligible

  • Pne: 14% of cases in disagreement

  • HF: 6% of cases in disagreement

  • AMI: 10% of cases in disagreement

  • =>Disagreements not a huge problem with numerator decisions (plus, the N is smaller)


Working hypothesis l.jpg

Working hypothesis

  • Ho: More exclusion rules (screening criteria) => more opportunities for error and disagreement

  • Example: AMI QI-1, ASA at admission has 13 exclusion rules. Exclude case if:

    • transferred from another acute care hospital

    • transferred from another ER

    • UTD admission source

    • allergy to aspirin

    • bleeding on admission,

    • etc.


Exclusion rules denominator variables l.jpg

Exclusion rules - denominator variables

Number of % of “CDAC% of “Provider

Indicator variable exclusions Dolphins” Dolphins”

AMI-1 Den ASA at admission13 1 30

AMI-2 Den ASA at discharge18 2 47

AMI-3 Den BB at admission15 4 22

AMI-4 Den BB at discharge17 1 33

AMI-5 Den ACEI at discharge18 3 32

AMI-6 Den Smoking cessation2 6 1

HF-1 Den5 3 8

HF-2 Den4 0 8

HF-3 Den6 2 13

HF-4 Den12 0 19

Pne-1 Den4 9 1

Pne-2 Den9 10 8

Pne-3 Den4 4 8

Pne-5 Den4 5 1


Exclusion rules numerator variables l.jpg

Exclusion rules - numerator variables

Number of % of “CDAC % of “Provider

Indicator variable exclusions Dolphins” Dolphins”

AMI-1 Num ASA at admission236

AMI-2 Num ASA at discharge120

AMI-3 Num BB at admission300

AMI-4 Num BB at discharge104

AMI-5 Num ACEI at discharge1018

AMI-6 Num Smoking cessation11117

HF-1 Num934

HF-2 Num1125

HF-3 Num163

Pne-1 Num237

Pne-2 Num8210

Pne-3 Num4141

Pne-5 Num2611

Note: HF-4 Num was omitted because it had only 5 cases eligible for numerator analysis


Cdac dolphin catch l.jpg

CDAC Dolphin Catch


Provider dolphin catch l.jpg

Provider Dolphin Catch


Provider dolphins cont d l.jpg

Provider Dolphins, cont’d


Hypothesis revisited l.jpg

Hypothesis revisited

  • Ho: More exclusion rules (screening criteria) => higher “Provider Dolphin catch” in determining patient eligibility for treatment

    • No signs of such influence on the numerator status of a case

    • No evidence of impact on “CDAC Dolphin catch” (neither for denominator nor numerator status).


Conclusions 1 l.jpg

Conclusions - 1

  • There was no evidence to place the hospitals’ integrity in doubt as far as self-abstracted data are concerned

    • However, public reporting is a whole new bowl of wax


Conclusions 2 l.jpg

Conclusions - 2

  • CDAC to Provider agreement rate ran in the 80% to 86% range (but recall our heavy investment into training and abstractor support).

    • It’s likely to be lower without the upfront training and ongoing support


Conclusions 3 l.jpg

Conclusions - 3

  • Bulk of the disagreement was over the denominator status

    • For AMI, almost 1/3 of decisions were in discord

    • For AMI and HF, a prevailing pattern is one of “Provider Dolphin catch”

    • No clear pattern for Pneumonia

    • “Provider Dolphin catch” increases as the number of exclusion criteria goes up


Conclusions 4 l.jpg

Conclusions - 4

  • Disagreement over the numerator status of a case is less common than over the denominator status (eligibility for an indicator).

    • Also, fewer cases qualify for the numerator


Suggestions 1 l.jpg

Suggestions - 1

  • Minimize the number of exclusion criteria

    • Ideally, keep exclusion rules under 3

      6SOW AMI indicators6SOW exclusions7SOW exclusions

      AMI-1 Den ASA at admission136

      AMI-2 Den ASA at discharge188

      AMI-3 Den BB at admission1510

      AMI-4 Den BB at discharge175

      AMI-5 Den ACEI at discharge189

      AMI-6 Den Smoking cessation21


Suggestions 2 l.jpg

Suggestions - 2

  • Disregard the “Provider Dolphin catch” in calculation of hospital error rate

    • This is a “productive mistake” as opposed to a “counter-productive mistake”


Roadblocks l.jpg

Roadblocks

  • No JCAHO mandate to prove accuracy

  • No QIO funding to train abstractors

  • Plan for hospital-level (not abstractor-level) tracking of accuracy

  • High turnover rate for abstractors


Contact info l.jpg

Contact info

Andrei Kuznetsov

MissouriPRO

573-893-7900, ext. 163

akuznetsov.mopro@sdps.org


  • Login