Massachusetts Quality e-Measure Validation Study (MQeVS)

Massachusetts Quality e-Measure Validation Study(MQeVS) Eric C. Schneider, MD, MSc Brigham and Women’s Hospital Harvard School of Public Health Sponsor: AHRQ (R18 HS017048)

Performance Measurement: Fundamental to Improvement Performance Visibility Performance Rewards Peers Patients Public Purchasers Performance Feedback Payments & Penalties Market Share “Report Cards” “P4P”

Quality Measurement and Reporting: 1997 • Limited public demand • Few standardized quality measures • Few organizations engaged • Few physicians aware • Public disclosure rare • Few patients aware

Standard-setting organizations National Quality Forum (NQF) Ambulatory Quality Alliance (AQA) Engagement of Medical Profession Physician Consortium for Performance Improvement AHRQ’s National Quality Measures Clearinghouse: (www.qualitymeasures.ahrq.gov) Access - 22 measures Outcome - 204 measures Patient Experience - 298 measures Population Health - 33 measures Process - 636 measures Structure - 44 measures Use of Services - 33 measures 2007: Dramatic Increase in Standardized Measures

Implementation of Measurement and Reporting Remains Controversial • Measurement programs based on • Administrative data • Hybrid method (supplemental med record review) • Patient/enrollee surveys • Skepticism about validity of performance results • Burdensome, costly data collection

Colorectal Cancer Screening: Results by Method Schneider et al, Under Review

Envisioning the EHR forPerformance Measurement • Detailed, structured clinical data • Unobtrusive data collection • Performance data aggregated across care settings to enable sophisticated measures (e.g. care coordination, safety) • Performance results at physician group rather than health plan level Schneider et al, Enhancing performance measurement: NCQA’s Roadmap for a Health Information Framework. JAMA 1999;282:1184

Studies of Performance Measurement Using EHR and HIE • Single institution • Single care setting • Single IT platform • Limited number of performance measures • Local, rather than national standards

Massachusetts e-Health Collaborative (MAeHC) • 2004 Demonstration Project • $50 million from Blue Cross Blue Shield of MA • Universal EHR adoption in 3 MA communities • Intra- and inter-community data exchange (HIE) • Integrated performance measurement/reporting • Massachusetts Health Quality Partners (MHQP) • AQA ambulatory performance measure starter set (26 measures) • Quality Data Warehouse • Receives HIE data as needed to calculate measure results

Courtesy of Micky Tripathi

MA Quality e-Measure Validation Study (MQeVS) To compare a quality measurement method using structured, coded EHR data with… 1) a “hybrid method” involving a combination of aggregated claims data and medical record review. 2) a “claims-only method” based on a novel database that aggregates claims data from commercial health plans and Medicare.

MQeVS Highlights • Implementation • MAeHC Pilot (www.maehc.org) • MHQP (www.mhqp.org) • Evaluation • Community ambulatory practices “similar to” U.S. • Broad clinical and research expertise among research partners • Harvard School of Public Health • MHQP • Partners Healthcare • Harvard Medical School • Center for Survey Research (U Mass, Boston)

MQeVS Sample • Two Specific Aims assure that study addresses broad populations and measures • Aim 1: 900 patients with EHR-HIE data, patient survey, medical record review, health plan administrative data • Aim 2: All “measure eligible” patients with EHR-HIE data and health plan administrative data

Data Collection Protocol: Privacy/Confidentiality HSPH (PI) Analysis Team Mass E-Health Collaborative #8 Med Record Extract (Aim 1) Claims Data Extract (Aims 1&2) EHR Quality Measure Extracts (Aims 1 & 2) #9 #10 Med Record Review Team (Partners) Medical Record Requests #6 Physician Offices CSC/ QDW MHQP Medical Record Copies Medical Record Consent Cohort (Study IDs) #5 Survey Data Extract (Aim 1) #7 #3 De-identified Study Cohort List (Aim 1) #1 Opt-out Step Pre-notification EHR Data Survey Team (CSR) Study Consent, Survey, and Med Record Review Consent #2 EHR VENDORS #4 Patients Re-identified Study Cohort: Pt Contact Info Completed Surveys and Consents

Analysis: Availability of Data for Measure Components Five key steps comprise a quality measure calculation algorithm, determining whether a patient meets criteria … 1. for inclusion in the preliminary denominator 2. to be excluded from the preliminary denominator 3. for membership in the final denominator (after exclusions) 4. for membership in the measure numerator 5. for selection for both the numerator and denominator

Quality Measures: Deconstructing Data NeedsE=exclusion criteria; D=denominator inclusion; N=numerator inclusion; Var=varies

Data Adequacy Assessment (1) Data Present, Event Confirmed: a lipid lowering medication is recorded in the patient’s data; (2) Data Present, Event Not Confirmed: there is no lipid lowering medication, but data indicate that the patient is taking other medications; (3) Data Not Present: there are no data regarding medications making it uncertain whether the patient is taking a lipid lowering medication.

Analysis Where: Availability through the EHR = (a+c) / (a+b+c+d) = 92% And: Availability through Hybrid method = (a+b) / (a+b+c+d) = 98%

Challenges • Logistical • HIE implementation • Data sharing (privacy/confidentiality) • Analytic • Lack of a “gold standard” • Complex correlation among data sources • Identifying and interpreting “missing” data • Small sample sizes for some measures

Opportunities • NQF-endorsed national standard measure set • Measure set creates need for broad range of data types • Direct comparison of HIE to two widely-used measurement methods • “Deconstruction” of measure components identifies data problems that may affect future performance measures

“Crossing the Quality Chasm?”

Massachusetts Quality e-Measure Validation Study (MQeVS)