Andrew Ryan, Doctoral Candidate

1. The Relationship between Performance on Medicare’s Process Quality Measures and Mortality: Evidence of Correlation, not Causation Andrew Ryan, Doctoral Candidate

2. Acknowledgements • Support • Agency for Healthcare Research and Quality • Jewish Healthcare Foundation • Dissertation Committee • Stan Wallack • Chris Tompkins • Deborah Garnick • Kit Baum

3. Outline • Measurement of quality in health care • Literature on relationship between process and outcome measures in health care • Methods • Results • Policy implications

4. Measurement of quality in health care • Quality measurement • Central to public quality reporting and pay for performance • Process measures • Assess whether “what is known to be ‘good’ medical care has been applied” (Donabedian, 1966) • Process performance measurement has dominated quality measurement • Hospital Compare • CMS/Premier Hospital Quality Incentive Demonstration • Why? • Providers deemed to have control over performance • Greater statistical power • Provide “actionable” information for improvement (Birkmeyer et al. 2006; Mant 2001) • Fear of inappropriately labeling hospitals as “bad” due to random variation in outcomes • Fear that use of outcome measures will lead to avoidance of higher risk patients

5. Potential problems with process measures • A process measure might: • not appropriately represent a process that is positively associated with patient health • represent a process that is positively associated with patient health but has been rendered obsolete by advances in clinical practice (Porter and Teisberg 2006) • Complicated nature of inpatient care may make process measures an inadequate proxy for quality due to limited scope • Implementation of process measurement may alter the relationship between the observed process measures and patient outcomes • measurement error • enhanced record keeping, or gaming • multitasking

6. CMS/ Joint Commission Starter Set Process Measures • Hospital Compare • hospital care • voluntary, internet-based public quality reporting • implemented by CMS in 2003 • After low rates of voluntary reporting in 2003, CMS made hospitals’ 2004 payment update conditional on reporting for 10 of the 17 indicators-- reporting increased dramatically. • CMS/Joint Commission Starter Set measures • AMI • aspirin at arrival/discharge • β blocker at arrival/discharge • use of ACE inhibitor • Heart failure • use of ACE inhibitor • assessment of left ventricular function • Pneumonia • oxygenation assessment • timing of initial antibiotics • pneumococcal vaccination Exception reporting: hospitals have discretion to exclude patients from calculation of process performance

7. Research Question: Relationship of Process Measures to Outcome For AMI, heart failure, and pneumonia: • Are the CMS/Joint Commission process measures correlated with hospital mortality? • Are the CMS/Joint Commission process measures causally related to hospital mortality?

8. Effects of exception reporting Greater exception reporting may improve process performance Association between process performance and mortality may be weaker for hospitals that report more exceptions Supplemental analysis to examine: Relationship between exception reporting and process performance Effect of exception reporting on relationship between process performance and mortality

9. Previous research finds association between processes and outcomes • Association between process measures and mortality for AMI, heart failure, pneumonia • Bradley et al. 2006; Eagle et al. 2005; Fonarow et al. 2007; Granger et al. 2005; Luthi et al. 2004; Luthi et al. 2003; Peterson et al. 2006; Werner and Bradlow 2006; Jha et al. 2007 • Limitations of studies • Employed cross-sectional designs • Results may be confounded by unobserved factors

10. Data and Methods

11. Data • Medicare fee-for-service inpatient claims and denominator files (2004-2006) • Primary diagnoses for which beneficiaries were admitted • Secondary diagnoses, demographics, and type of admission for risk adjustment • Discharge status to exclude transfer patients • Outcome measure: 30-day mortality • 2006 Medicare hospital characteristics • 2004-2006 Hospital Compare data • Hospital process performance

12. Composite process quality measure • Defined as the z score of the weighted sum of z-scores for process measures corresponding to each condition • Transform each process measure by the z-score to avoid bias • Bias could result from a positive correlation between the likelihood of reporting on a measure and performance on that measure. • Transform the sum of the weighted z-scores by the z-score • Facilitates interpretation • Compute for all hospitals reporting a denominator of at least 10 patients for at least one measure

13. Risk adjustment of 30-day MortalityAMI, Heart Failure, Pneumonia • Hospital-level observed / expected • Expected: estimated from patient-level logit models where mortality was regressed on: • Age, gender, race • Elixhauser comorbidities (Elixhauser et al.1998) • Type of admission (emergency, urgent, elective) • Season of admission

14. Two Specifications for Relationship of Process Measures to Mortality #1: Pooled cross section #2: Add hospital Fixed Effects Then consider impact of hospitals’ exceptions reporting for both specifications

15. Analysis #1:Pooled Cross Section ln (RA Mortality jkt)= b0 + b1Z j + b2year t + b3Z j * year t + δ1Processjkt + δ2Process2jkt + e jkt Where: • j is indexed to hospitals, k is indexed to condition (AMI or heart failure), t is indexed to year (2004-2006) • Z is a vector of hospital characteristics (bed size, ratio of residents / average daily census, urbanicity, ownership, % of Medicare admissions) • Process is the composite quality measure • year is a vector of dummy variables for 2005 and 2006

16. Analysis: #2 Hospital Fixed Effects ln (RA Mortality jkt)= b0 + b2yeart + b3Z j * year t + δ1Processjkt + δ2Process2jkt + h j + e jkt • Where h is a vector of hospital-specific fixed effects • The inclusion of hospital-specific effects controls for unobserved time-invariant factors at the hospital level (e.g. physician skill/experience, technology) that may confound the relationship between processes of care and outcomes

17. Standard error specification • Multiple observations from the same hospitals over time gives rise to group-level heteroskedasticity • Cluster-robust standard errors are estimated (Williams 2000) • Hospital-level RA mortality rates vary in their precision as a result of the number of patients in the denominator of the calculation • Analytical weights (Gould 1994), based on the average number of patients in the denominator of hospitals' RA mortality calculation over the two study years, are employed

18. Results

19. Descriptive statistics Note: table includes data from hospitals that are included in at least one of the regression models Note: data from individual process measures is included if hospitals report at least 10 in the measure denominator Note: composite measures are scaled so that the mean of each measure across all hospitals across all years is 1

20. Marginal effects of process performance on risk-adjusted mortality *** p<0.01, ** p<0.05, * p<0.1 Note: hospital controls include ownership, bed size, teaching status, urbanicity, and ratio of residents to average daily census Note: Marginal effects are evaluated at median of process composite Note: Marginal effects are multiplied by 100 to facilitate interpretation Note: Robust standard errors in parentheses

21. Effects of exception reporting • Not associated with process performance for composite measures for any diagnosis • Higher exception reporting was associated with a substantial and significant increase in one process performance measure: ACE inhibitor for AMI • Did not moderate effect of process performance in expected direction • Did not explain null results from FE model

22. Summary of findings • Pooled cross section models showed CMS/Joint Commission process performance measures arecorrelated with patient 30-day RA mortality for AMI, pneumonia, and heart failure • Consistent with recent evidence • However, hospital fixed effects models showed no evidence of an association between process performance and mortality for any diagnosis • Suggests that CMS/Joint Commission measures are not causally related to mortality • Conflicts with recent evidence • Unobserved improvements in record keeping may be responsible for lack of casual relationship • Higher rate of exceptions does not appear to increase measured process quality nor affect relationship between process performance and mortality

23. Implications for payment system • Correlation between process performance and mortality supports utility of process measures for public reporting • Steer patients toward higher quality providers • Absence of a causal relationship casts serious doubt on the utility of current process performance measures as a metric for hospital quality improvement