Analysis and presentation of quality indicators
Download
1 / 61

Analysis and presentation of quality indicators - PowerPoint PPT Presentation


  • 98 Views
  • Uploaded on

Analysis and presentation of quality indicators. Dr David Harrison Senior Statistician, ICNARC. Analysis and presentation of QIs. Principles of statistical process control Comparison among providers Continuous monitoring over time. Analysis and presentation of QIs.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Analysis and presentation of quality indicators ' - jedidiah-braylon


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Analysis and presentation of quality indicators

Analysis and presentation of quality indicators

Dr David Harrison

Senior Statistician, ICNARC


Analysis and presentation of qis
Analysis and presentation of QIs

  • Principles of statistical process control

  • Comparison among providers

  • Continuous monitoring over time


Analysis and presentation of qis1
Analysis and presentation of QIs

  • Principles of statistical process control

    • Common cause variation

    • Special cause variation

    • Control limits

  • Comparison among providers

  • Continuous monitoring over time


Principles of statistical process control
Principles of statistical process control

  • Common cause variation

    • Variation cannot be eliminated

    • Some variation is inherent to any process

    • This is termed “common cause variation”

    • To reduce common cause variation we need to change the process




They are not identical1
They are not identical…

…but they are all my signature


We could rank them
We could rank them…

1.

2.

3.

4.

5.

…but this doesn’t make much sense!


We could reject some as low quality
We could reject some as low quality…

…but they are still my signature!


This is common cause variation
This iscommon cause variation


Principles of statistical process control1
Principles of statistical process control

  • Special cause variation

    • Some variation is the result of external factors acting on a process

    • This is termed “special cause variation”

    • To reduce special cause variation we need to identify the source and eliminate it



Now we have a sixth signature1
Now we have a sixth signature…

…it’s a good try, but I think you can tell which one is the forgery!


This is special cause variation
This isspecial cause variation


Control limits
Control limits

  • Statistical process control is all about making allowance for common cause variation to detect special cause variation

  • To do this we place control limits around a process

  • Control limits represent the acceptable range of common cause variation


Control limits1
Control limits

  • Typically control limits of 2 and 3 SDs represent “alert” and “alarm”

  • If a system is in control:

    • 95.4% of values within 2 SDs

    • 99.7% of values within 3 SDs


Analysis and presentation of qis2
Analysis and presentation of QIs

  • Principles of statistical process control

  • Comparison among providers

    • League tables

    • Caterpillar plots

    • Funnel plots

    • Over-dispersion

  • Continuous monitoring over time


Comparison among providers
Comparison among providers

  • I’ll assume we have a binary event (e.g. death) and an associated risk estimate (e.g. predicted risk of death)

  • Most common QI is:observed events / expected events

  • (for mortality this is the standardised mortality ratio)

  • How should we compare this QI among providers (e.g. critical care units)?


League tables
League tables

  • Journalists love them

    • High impact

    • Everyone wants to know who is firstand last


Seven deadliest hospitals identified in damning Dr Foster reportDaily Telegraph, 29 November 2009

Twelve NHS trusts slammedThe Sun, 29 November 2009

Patient safety at ScarboroughHospital ‘second worst in country’Scarborough Evening News, 29 November 2009


League tables1
League tables

  • Journalists love them

    • High impact

    • Everyone wants to know who is firstand last

  • Statisticians hate them

    • Overemphasise unimportant differences

    • Even if there is no true difference, someone will be first and someone last

    • No account of role of chance (common cause variation)


Marshall spiegelhalter bmj 1998
Marshall & Spiegelhalter, BMJ 1998

  • League table of 52 IVF clinics ranked on live birth rate

  • Monte Carlo simulation to put 95% CI on ranks



Marshall spiegelhalter bmj 19982
Marshall & Spiegelhalter, BMJ 1998

  • King’s College Hospital – sixth from bottom – is the only one that can reliably be placed in the bottom 25%


Marshall spiegelhalter bmj 19983
Marshall & Spiegelhalter, BMJ 1998

  • BMI Chiltern Hospital – seventh from bottom – may not even be in the bottom 50%


Marshall spiegelhalter bmj 19984
Marshall & Spiegelhalter, BMJ 1998

*

*

*

*

*

  • Five clinics can confidently be placed in the top quarter


Marshall spiegelhalter bmj 19985
Marshall & Spiegelhalter, BMJ 1998

  • Southmead General – ranked sixth from top – may not be in the top 50%


Caterpillar plots or forest plots
Caterpillar plots (or forest plots)

  • Plot of QIs with CIs in rank order

  • Still a league table really

  • But at least acknowledges variation by including CIs



Caterpillar plot anzics
Caterpillar plot – ANZICS

  • SMRs by APACHE III-J for 106 adult ICUs in Australia and New Zealand, 2004(Cook et al. Crit Care Resusc 2008)


Funnel plots
Funnel plots

  • Larger sample = greater precision

  • If you plot QI against sample size, you expect to see a funnel shape

  • We can plot funnel shaped control limits



Funnel plot anzics
Funnel plot – ANZICS

  • SMRs by APACHE III-J for 106 adult ICUs in Australia and New Zealand, 2004(Cook et al. Crit Care Resusc 2008)


Funnel plot anzics1
Funnel plot – ANZICS

  • Note: use of normal distribution can result in negative confidence intervals – better methods exist


Funnel plot anzics2
Funnel plot – ANZICS

  • Note: as SMR is a ratio measure, we would advocate plotting on a log scale (i.e. SMR=2 and SMR=0.5 are equidistant from SMR=1)


Funnel plot sicsag
Funnel plot – SICSAG

  • SMRs by APACHE II for 25 adult ICUs in Scotland, 2009(SICSAG Audit of critical care in Scotland 2010)


Funnel plot sicsag1
Funnel plot – SICSAG

  • Note: as the model is poorly calibrated, most units are “better than average” – the funnel has been centred on the average SMR not 1


Over dispersion
Over-dispersion

  • Variability more than expected by chance

  • Suggests important factors that vary among providers are not being taken into account

  • Too many providers classified as “abnormal” (i.e. outside the funnel)


Over dispersion hospital readmissions
Over-dispersion – hospital readmissions

(Spiegelhalter. Qual Saf Health Care 2005)


Over dispersion what to do
Over-dispersion – what to do…?

  • Don’t use the indicator?

  • Improve risk adjustment

  • Adjust for it

    • Estimate “over-dispersion factor” by “Winsorisation”

  • Use random effects models

    • Assumes each provider has their own true rate from a distribution


Example over dispersion factor
Example – over-dispersion factor

  • SMRs by ICNARC model for 171 adult ICUs in England, Wales & N Ireland, 2009


Example over dispersion factors
Example – over-dispersion factors

  • Over-dispersion factor estimated at 1.4

  • Funnel widened


Analysis and presentation of qis3
Analysis and presentation of QIs

  • Principles of statistical process control

  • Comparison among providers

  • Continuous monitoring over time

    • RAP chart

    • EWMA

    • VLAD

    • R-SPRT

    • CUSUM


Continuous monitoring over time
Continuous monitoring over time

  • Various approaches

  • In general, they consist of…

    • an indicator that is updated for each consecutive patient

    • control limits


Example for continuous monitoring
Example for continuous monitoring

  • Queen Kate Hospital

  • Fictitious critical care unit

  • Random sample of 2000 records from the Case Mix Programme Database

  • After 1000 records, outcomes changed so that an extra 6% of patients (selected at random) die

  • Risk adjustment by the ICNARC (2009) model



Rap chart
RAP chart

  • Risk-adjusted p chart

  • Cohort divided into discrete blocks (e.g. 100 patients)

  • Indicator is observed mortality

  • Control limits are predicted mortality +/- 2 or 3 SDs

  • Pro

    • Displays observed and expected mortality

  • Con

    • Still in blocks, not sensitive



EWMA

  • Exponentially weighted moving average

  • Similar to RAP but uses all data up to the current timepoint

  • Data weighted by a smoothing factor so that most recent data are given most weight


EWMA

  • Pro

    • Displays observed and expected mortality

    • Estimates updated continuously not in arbitrary blocks

  • Con

    • Choice of smoothing factor is important – too little smoothing and plot is unreadable, too much and plot is insensitive to changes



VLAD

  • Variable life adjusted display

  • Cumulative observed minus expected deaths

  • Pro

    • Nice easy interpretation

  • Con

    • Control limits are complex to calculate curved functions



R sprt
R-SPRT

  • Resetting sequential probability ratio test

  • Tests evidence for/against a specific hypothesis (e.g. odds of death are double that predicted by the model)

  • Plot of log likelihood ratio

  • If bottom line is reached (strong evidence against hypothesis) then line resets to zero


R sprt1
R-SPRT

  • Pro

    • Nice statistical properties

    • Control limits are horizontal lines

  • Con

    • Choice of hypothesis to test is arbitrary – should we test for an OR of 2, 1.5,…?



Cusum
CUSUM

  • “Cumulative sum”

  • Log likelihood ratio – same as R-SPRT

  • “Absorbing barrier” at zero (i.e. never goes below zero)


Cusum1
CUSUM

  • Pros/Cons as for the R-SPRT plus…

  • Pro

    • Does not allow credit to build up (as in R-SPRT) so alerts earlier

    • Negative CUSUM (e.g. OR=0.5) can be plotted on the same axes

  • Con

    • Cannot detect evidence against hypothesis



Which method s to use
Which method(s) to use…?

  • Comparison among providers

    • Funnel plot

  • Continuous monitoring over time

    • EWMA

    • or R-SPRT

    • or CUSUM

    • (VLAD can be used as a display in conjunction with, e.g., CUSUM for monitoring)



ad