part 07 l.
Skip this Video
Loading SlideShow in 5 Seconds..
PART 07 PowerPoint Presentation
Download Presentation

Loading in 2 Seconds...

play fullscreen
1 / 38

PART 07 - PowerPoint PPT Presentation

  • Uploaded on

PART 07 Evaluating Hospital Performance PERFORMANCE MEASURES Patient outcomes Mortality, morbidity, satisfaction with care 30-day mortality among heart attack patients ( Normand et al JAMA 1996, JASA 1997) Process Medication & test administration, costs

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
part 07


Evaluating Hospital Performance

BIO656--Multilevel Models

performance measures

Patient outcomes

  • Mortality, morbidity, satisfaction with care
    • 30-day mortality among heart attack patients (Normand et al JAMA 1996, JASA 1997)


  • Medication & test administration, costs
    • Laboratory costs for diabetic patients
    • Number of physician visits
      • Hofer et al JAMA, 1999
    • Palmer et al. (1996)

BIO656--Multilevel Models

data structure


  • Patients nested in physicians, hospitals, HMOs, ...
  • Providers clustered by health care systems, market areas, geographic areas
  • Covariates at different levels of aggregation:
    • patient, physician, hospital, ...

Variation in variability

  • Statistical stability varies over physicians, hospitals, ..

BIO656--Multilevel Models

mlms are effective
MLMs are Effective

Correlation at many levels

  • Hospital practices may induce a strong correlation among patient outcomes within hospitals even after accounting for patient characteristics

Structuring estimation

  • Stabilizing noisy estimates
  • Balancing SEs
  • Estimating ranks and other non-standard summaries

BIO656--Multilevel Models

the cooperative cardiovascular project ccp
The Cooperative Cardiovascular Project (CCP)
  • Abstracted medical records for patients discharged from hospitals located in Alabama, Connecticut, Iowa, and Wisconsin (June 1992May 1993)
  • 3,269 patients hospitalized in 122 hospitals in four US States for Acute Myocardial Infarction

BIO656--Multilevel Models

  • Identify “aberrant” hospitals with respect to several performance measures
  • Report the statistical uncertainty associated with ranking of the “worst hospitals”
  • Investigate if hospital characteristics explain variation in hospital-specific mortality rates

BIO656--Multilevel Models



  • Mortality within 30-days of hospital admission

Patient characteristics

  • Admission severity index constructed on the basis of 34 patient attributes

Hospital characteristics

  • Urban/Rural
  • (Non academic)/(versus academic)
  • Number of beds

BIO656--Multilevel Models

why adjust for case mix patient characteristics
Why adjust for case mix?(patient characteristics)
  • Irrespective of quality of care, older/sicker patients with multiple diseases have increased need of health care services and poorer health outcomes
  • Without adjustment, physicians/hospitals who treat relatively more of these patients will appear to provide more expensive and lower quality care than those who see relatively younger/healthier patients
  • If there is inadequate case mix adjustment,

evaluations will be unfair

  • But, need to avoid over adjusting

BIO656--Multilevel Models

case mix adjustment
Case-mix Adjustment

Compute hospital-specific, expected mortality by:

    • estimating a patient-level mortality model using

all hospitals

2. averaging the model-produced probabilities for all patients within a hospital

  • Hospitals with “higher-than-expected” mortality rates can be flagged as institutions with potential quality problems, but need to account for uncertainty
  • Need to be careful, if also adjusting for hospital characteristics
    • May adjust away the important signal

BIO656--Multilevel Models


Wrong SEs

  • Test-based

(as we know, very poor approach)

BIO656--Multilevel Models

Hospital Profiling of Mortality Rates Acute Myocardial Infarction Patients(Normand et al. JAMA 1996, JASA 1997)

BIO656--Multilevel Models

hierarchical logistic regression
Hierarchical logistic regression

I: Patient within-provider

  • Patient-level logistic regression model with random intercept & slope

II: Between-provider

  • Hospital-specific random effects are regressed on hospital-specific characteristics
    • Explicit regression

BIO656--Multilevel Models



0 + 1(sevij – sevbar)



BIO656--Multilevel Models


we use b0i + b1i(...)

BIO656--Multilevel Models


b0i = *00 + N(..), etc.

Interpretation of parameters is different for the two levels

BIO656--Multilevel Models

  • Estimates of regression coefficients under three models:
    • Random intercept only
    • Random intercept and random slope
    • Random intercept, random slope, and hospital covariates
  • Hospital performance measures

BIO656--Multilevel Models


Normand et al. JASA 1997

BIO656--Multilevel Models

30 day mortality 2 5 th and 97 5 th percentiles for a patient of average admission severity
30-DAY MORTALITY2.5th and 97.5th percentiles for a patient of average admission severity

Exchangeable model

  • Random intercept and slope, no hospital covariates

log(odds): (-1.87,-1.56)

probability,scale: (0.13, 0.17)

Covariate (non-exchangeable) model

  • Random intercept and slope, with hospital covariates
  • Patient treated in a large, urban academic hospital

log(odds): (-2.15,-1.45)

probability scale: (0.10,0.19)

BIO656--Multilevel Models

effect of hospital characteristics on baseline log odds of 30 day mortality
Effect of hospital characteristics on baseline log-odds of 30-day mortality
  • For an average patient, rural hospitals have a higher odds ratio than urban hospitals
    • Indicates between-hospital differences in the baseline mortality rates
    • Case-mix adjustment may be able to remove some of this difference

BIO656--Multilevel Models

effect of hospital characteristics on association between severity and mortality slopes
Effect of hospital characteristics on association between severity and mortality(slopes)
  • The association between severity and mortality is modified by hospital size
  • Medium-sized hospitals have smaller severity/mortality associations than large hospitals
    • Indicates that the effect of clinical burden (patient severity) on mortality differs across hospitals

BIO656--Multilevel Models

homework is on front table
Homework is on front table

BIO656--Multilevel Models


Observed and risk-adjusted hospital mortality rates

Urban Hospitals

Histogram displays (observed – adjusted)

BIO656--Multilevel Models


Observed and risk-adjusted hospital mortality rates

Rural Hospitals

Histogram displays (observed – adjusted)

Substantial adjustment for severity

BIO656--Multilevel Models

  • There is substantial adjustment for admission severity
  • Generally, urban hospitals are adjusted less than rural
  • There is less variability in observed or adjusted estimated rates for urban hospitals than for rural hospitals

Can you explain why?

BIO656--Multilevel Models


Normand et al. JASA 1997

BIO656--Multilevel Models


Average the probabilities

Don’t average the covariates

BIO656--Multilevel Models


k denotes a draw from the posterior

BIO656--Multilevel Models

plug in the average covariate
Plug in the average covariate

Keep the hospital variation

BIO656--Multilevel Models

comparing measures of hospital performance
Comparing measures of hospital performance

Three measures of hospital performance

  • Probability of a large difference between adjusted and standardized mortality rates
  • Probability of excess mortality for the average patient
  • Z-score

BIO656--Multilevel Models

hospital ranks
Hospital Ranks
  • There was moderate disagreement among the criteria for classifying hospitals as “aberrant”
  • Nevertheless, hospital 1 is ranked worst
  • It is rural, medium sized non-academic with an observed mortality rate of 35%, and adjusted rate of 28%

BIO656--Multilevel Models

adjusting for hospital level charateristics
Adjusting for hospital-level charateristics

Changes the comparison group in “as compared to what?”

  • All hospitals (unadjusted at hospital level)
  • Hospitals of a similar size, urbanicity, ...
  • Percent of physicians who are board certified
  • Hospitals with a similar death rate 

Variance reduction and goodness of fit

should not be the primary considerations

  • “As compared to what?” must dominate

BIO656--Multilevel Models

  • Profiling medical providers is multi-faced and data intensive process with substantial implications for health care practice, management, and policy
  • Major issues include data quality and availability, choice of performance measures, formulation of statistical models (including adjustments), reporting results
  • The ranking approaches and summaries used by Normand and colleagues are very good, but some improvement is possible

BIO656--Multilevel Models

multi level models address key technical conceptual profiling issues including
Multi-level models address key technical & conceptual profiling issues, including
  • Adjusting for patient severity
  • Accounting for within-provider correlations
  • Accounting for differential sample sizes at all levels
  • Stabilize estimates
  • Structure ranking and other, derived comparisons

BIO656--Multilevel Models