Alex Bottle robert.bottle@imperial.ac.uk Imperial College London Dr Foster Unit

1. A method for estimating the cost of reducing the false alarm rate in multi-institution performance monitoring using CUSUM charts Alex Bottle robert.bottle@imperial.ac.uk Imperial College London Dr Foster Unit

2. Overview • Background: cumulative sum charts and nationwide NHS mortality monitoring tool • Extent of multiple testing • Factors affecting false alarm rate • Simulation for false alarm and successful detection rates • Estimation of ‘cost’: worked example for AMI • Summary

3. CUSUM chart essentials • Plots one patient at a time • Chart statistic (log-likelihood ratio) goes up if patient dies and down if patient survives • Chart rises faster if low-risk patient dies • If crosses preset threshold, chart ‘signals’ • Threshold choice involves consideration of type I and type II error rates

4. CUSUM charts

5. Mortality monitoring tool • In use in ~100 acute hospitals in England • Compares each hospital’s case-mix adjusted mortality rate with national average • Tests for an odds ratio of at least 2 • Displayed using cumulative sum charts • Data are updated monthly

6. Mortality monitoring tool: opening screen

7. Extent of multiple testing Over time: threshold handles this element But… At each hospital trust each month: • 78 diagnosis groups • >100 procedure groups National monitoring incurs further ‘cost’: • ~150 acute hospital trusts • Consultant-level monitoring?

8. Factors affecting the false alarm rate • Threshold: the higher this is set, the lower the false alarm rate • Length of monitoring: number of patients varies by hospital and diagnosis • Expected mortality rate: e.g. 5% rates will have high FAR than 1% rates • Size of increase (OR) to be detected (not considered here)

9. Research question • A higher chart threshold -> lower FAR but slower detection of high mortality rates • Compared with the conventional 5% false alarm rate, what is the ‘cost’ of having a lower false alarm rate (1% or 0.1%) to deal with all the multiple testing?

10. Simulation: FAR and SDR • For FAR, generate 5,000 artificial hospitals with mortality rate p • Do this for various p, p=0.1% to 30% • Calculate FAR after t patients, t in steps of 5 from 5 to 20,000 • Do this for different thresholds h, h=0.5 to 15 • For SDR, generate hospitals with rate 2p/(1-p)

11. Using the simulation to estimate ‘cost’ • For each dx, work out the threshold h needed for FAR of 5% at average hosp • Find number of monitored patients t needed for SDR of 80% using threshold h • Knowing the dx’s expected death rate and OR to be detected, convert t into a number of deaths • Repeat for FAR of 1% and 0.1% • Find the difference in number of deaths between the pairs of FAR values

12. ‘Cost’ calculation for AMI in England (1) • National death rate=11.8%. Average number of AMIs per hospital=467 • For FAR=5%, h=5.2 -> t=185 for SDR=80% • At rate p, this means 21.8 deaths • At rate 2p/(1-p), this means 39.0 deaths • ‘Excess’ deaths: 39.0 – 21.8 = 17.2

13. ‘Cost’ calculation for AMI in England (2) • For FAR=0.1%, h=8.6 -> t=305 for SDR=80% • At rate p, this means 36.0 deaths • At rate 2p/(1-p), this means 64.4 deaths • ‘Excess’ deaths: 64.4 – 36.0 = 28.4 • ‘Cost’ of lowering FAR to 0.1% = 28.4 – 17.2 = 11.2 extra deaths at average hosp

14. Findings for AAA repair and CABG • AAA repair: less common but high risk • ‘Cost’=6.3 for FAR=0.1%, 2.4 for FAR=1% • CABG: common but low risk • ‘Cost’= 6.8 for FAR=0.1%, 2.8 for FAR=1% • These are all figures for an average hospital

15. Summary • Multiple testing can be addressed by lowering the false alarm rate: raise the threshold for CUSUM charts • Other approaches include minimising ‘loss function’ or maximising ‘desirability function’ • The proposed measure of ‘cost’ depends on mortality rate and hospital volume • The ‘cost’ can be derived from simulation and is intuitive to less-technical users