html5-img
1 / 58

Explanation of HSMR and experience in Britain

Explanation of HSMR and experience in Britain . Dr Paul Aylin Professor Sir Brian Jarman Dr Alex Bottle p.aylin@imperial.ac.uk. Contents. History English Hospital Statistics Case-mix adjustment Administrative vs Clinical datasets Latest methods Presentation of performance data

Faraday
Download Presentation

Explanation of HSMR and experience in Britain

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Explanation of HSMR and experience in Britain Dr Paul Aylin Professor Sir Brian Jarman Dr Alex Bottle p.aylin@imperial.ac.uk

  2. Contents History English Hospital Statistics Case-mix adjustment Administrative vs Clinical datasets Latest methods Presentation of performance data • League tables • Bayesian ranking • Statistical process Control Charts • Beyond HSMRs

  3. Hospital Standardised Mortality Ratio Originally developed by Brian Jarman Jarman et al. “Explaining Differences in English Hospital Death Rates Using Routinely Collected Data,” BMJ 1999;318:1515-1520 Indirect standardisation using top 80 diagnoses (using 3 digit ICD9 code) leading to 80% of all in-hospital deaths and adjusted for diagnosis, age, sex, emergency/elective and LOS

  4. Florence Nightingale

  5. The Areas of the blue, red, & black wedges are each measured from the centre as the common vertex. The blue wedges measured from the centre of the circle represent area for area the deaths from Preventable or Mitigable Zymotic diseases, the red wedges measured from the centre the deaths from wounds, & the black wedges measured from the centre the deaths from all other causes. The black line across the red triangle in Nov. 1854 marks the boundary of the deaths from all other causes during the month. In October 1854, & April 1855, the black area coincides with the red, in January & February 1855, the blue coincides with the black. The entire areas may be compared by following the blue, the red, & the black lines enclosing them. Source: Nightingale, Florence. Notes on Matters Affecting the Health, Efficiency and Hospital Administration of the British Army, 1858. http://www.nightingaledeclaration.net/index.php?option=com_content&task=view&id=37&Itemid=25

  6. Florence Nightingale Uniform hospital statistics would: “Enable us to ascertain the relative mortality of different hospitals as well as of different diseases and injuries at the same and at different ages, the relative frequency of different diseases and injuries among the classes which enter hospitals in different countries, and in different districts of the same country” Nightingale 1863

  7. Hospital Episode Statistics Electronic record of every inpatient or day case episode of patient care in every NHS (public) hospital 14 million records a year 300 fields of information including Patient details such as age, sex, address Diagnosis using ICD10 Procedures using OPCS4 Admission method Discharge method

  8. Why use Hospital Episode Statistics Comprehensive – collected by all NHS trusts across country on all patients Coding of data separate from clinician Access Updated monthly through NHS clearing service

  9. Limitations of HES Some data quality issues in terms of completeness, but quality appears to be improving year on year Episode based In-hospital death only Limited case-mix factors HES vs Clinical databases

  10. In-hospital vs. all deaths Likely to make more difference for diagnoses like cancer, less so for acute disease and emergency surgical admissions Possible solution is HES/death certificate information linked data

  11. Case mix adjustment Limited within HES? Age Sex Emergency/Elective

  12. Risk adjustment models using HES on 3 index procedures CABG AAA Bowel resection for colorectal cancer

  13. Risk factors

  14. ROC curve areas comparing ‘simple’, ‘intermediate’ and ‘complex’ models derived from HES with models derived from clinical databases for four index procedures Aylin P; Bottle A; Majeed A. Use of administrative data or clinical databases as predictors of risk of death in hospital: comparison of models. BMJ 2007;334: 1044

  15. Calibration plots for ‘complex’ HES-based risk prediction models for four index procedures showing observed number of deaths against predicted based on validation set Aylin P; Bottle A; Majeed A. Use of administrative data or clinical databases as predictors of risk of death in hospital: comparison of models. BMJ 2007;334: 1044

  16. Comparison of HES vs clinical databases Isolated CABG HES around 10% fewer cases compared to National Cardiac Surgical Database Fifth National Adult Cardiac Surgical Database Report 2003. The Society of Cardiothoracic Surgeons of Great Britain and Ireland. Dendrite Clinical Systems Ltd. Henley-Upon-Thames. 2004. Vascular surgery HES = 32,242 National Vascular Database = 8,462 Aylin P; Lees T; Baker S; Prytherch D; Ashley S. (2007) Descriptive study comparing routine hospital administrative data with the Vascular Society of Great Britain and Ireland's National Vascular Database. Eur J Vasc Endovasc Surg 2007;33:461-465 Bowel resection for colorectal cancer HES 2001/2 = 16,346 ACPGBI 2001/2 = 7,635 ACPGBI database, 39% of patients had missing data for the risk factors Garout M, Tilney H, Aylin, P. Comparison of administrative data with the Association of Coloproctology of Great Britain and Ireland (ACPGBI) colorectal cancer database. International Journal of Colorectal Disease (in press)

  17. Current HSMRs Same risk models used as for all risk adjusted analyses (including log-likelihood CUSUMs) carried out within the unit. Risk of death calculated for each individual admission using binary logistic regression Use 56 diagnosis (CCS) groups contributing to 80% of all in-hospital deaths Sum of probability of death = expected deaths Divide observed by expected for each trust to give HSMR Plotted on funnel plot with 95% and 99.8% CIs (2 and 3 sigma)

  18. Current casemix adjustment model Adjusts for age sex emergency status socio-economic deprivation diagnosis (56 CCS groups) and subgroups (3 digit ICD10) co-morbidity – Charlson index number of prior emergency admissions palliative care year month of admission

  19. Current ROC (based on 1996/7-2006/7 HES data) for in-hospital mortality 56 Clinical Classification System diagnostic groups leading to 80% of all in-hospital deaths 7 CCS groups 0.90 or above • Includes cancer of breast (0.94) and biliary tract disease (0.91) 27 CCS groups 0.80 to 0.89 • Includes aortic, peripheral and visceral anuerysms (0.87) and cancer of colon (0.83) 19 CCS groups 0.7 to 0.79 • Includes septicaemia (0.77) and acute myocardial infarction (0.74) 3 CCS groups 0.60 to 0.69 • Includes COPD (0.69) and congestive heart failure (0.65)

  20. Presentation of clinical outcomes “Even if all surgeons are equally good, about half will have below average results, one will have the worst results, and the worst results will be a long way below average” Poloniecki J. BMJ 1998;316:1734-1736

  21. RR of death following CABG HES data 1999/00 to 2001/02

  22. Criticisms of ‘league tables’ Spurious ranking – ‘someone’s got to be bottom’ Encourages comparison when perhaps not justified 95% intervals arbitrary No consideration of multiple comparisons Single-year cross-section – what about change? 

  23. Bayesian ranking Bayesian approach using Monte Carlo simulations can provide confidence intervals around ranks Can also provide probability that a unit is in top 10%, 5% or even is at the top of the table See Marshall et al. (1998). League tables of in vitro fertilisation clinics: how confident can we be about the rankings? British Medical Journal, 316, 1701-4.

  24. RR of death following CABG - 1999/00 to 2001/02 Top 15 only

  25. Rankings for CABG mortality 1999/00 to 2001/02

  26. RR of death following CABG - 1999/00 to 2001/02 Top 15 only

  27. Statistical Process Control (SPC) charts Shipman: Aylin et al, Lancet (2003) Mohammed et al, Lancet (2001) Spiegelhalter et al, J Qual Health Care (2003) Surgical mortality: Poloniecki et al, BMJ (1998) Lovegrove et al, CHI report into St George’s Steiner et al, Biostatistics (2000) Public health: Terje et al, Stats in Med (1993) Vanbrackle & Williamson, Stats in Med (1999) Rossi et al, Stats in Med (1999) Williamson & Weatherby-Hudson, Stats in Med (1999)

  28. Common features of SPC charts Need to define: in-control process (acceptable/benchmark performance) out-of-control process (that is cause for concern) Test statistic difference between observed and benchmark performance calculated for each unit at each time point Pre-defined alarm threshold minimise false alarms but remain sensitive to true signals

  29. Control chart for HSMRs by Provider Trust 2004/5 based on simulated deaths using binomial distribution

  30. Funnel plots No ranking Visual relationship with volume Takes account of increased variability of smaller centres

  31. Sample

  32. Sample

  33. Sample

  34. Prospective SPC charts Cumulative sums of outcomes accumulate information on performance over time formal assessment of sensitivity and specificity different ways of deriving test statistic • Log-likelihood CUSUM (our preferred method) • Sequential Probability Ratio Test (SPRT) • Exponentially Weighted Moving Average (EWMA)

  35. Risk-adjusted Log-likelihood CUSUM charts STEP 1: estimate pre-op risk for each patient, given their age, sex etc. This may be national average or other benchmark STEP 2: Order patients chronologically by date of operation STEP 3: Choose chart threshold(s) of acceptable “sensitivity” and “specificity” (via simulation) STEP 4: Plot function of patient’s actual outcome v pre-op risk for every patient, and see if – and why – threshold(s) is crossed

  36. More details Based on log-likelihood CUSUM to detect a predetermined increase in risk of interest Taken from Steiner et al (2000); pre-op risks derived from logistic regression of national data The CUSUM statistic is the log-likelihood test statistic for binomial data based on the predicted risk of outcome and the actual outcome Model uses HES data and adjusts for age, sex, emergency status, socio-economic deprivation etc. Bottle A, Aylin P. Intelligent Information: a national system for monitoring clinical performance. Health Services Research (in press).

  37. AAA mortality monitoring

  38. Intelligent Information: a national system for monitoring clinical performance. Bottle A, Aylin P. Intelligent Information: a national system for monitoring clinical performance. Health Services Research 2008;43:10-31

  39. Relevant publications Bottle A, Aylin P. Intelligent Information: a national system for monitoring clinical performance. Health Services Research 2008;43:10-31 Aylin P; Bottle A; Majeed A. (2007) Use of administrative data or clinical databases as predictors of risk of death in hospital: comparison of models. BMJ 2007;334: 1044 Aylin P, Bottle A. Are hospital league tables calculated correctly? A commentary. Public Health (Sept 3007) Aylin P; Lees T; Baker S; Prytherch D; Ashley S. (2007) Descriptive study comparing routine hospital administrative data with the Vascular Society of Great Britain and Ireland's National Vascular Database. Eur J Vasc Endovasc Surg 2007;33:461-465 Wright J, Dugdale B, Hammond I, Jarman B, Neary M et al. Learning from death: a hospital mortality reduction programme. JR Soc Med, June 2006; 99:303-308. Bottle A, Aylin P. Mortality associated with delay in operation after hip fracture: observational study. BMJ 2006;332:947-951 Aylin P, Williams S, Jarman B, Bottle A. Variation in operation rates by primary care trust. BMJ, Sep 2005; 331: 539 Jarman, B, Bottle, A, Aylin, P, Browne, M. Monitoring changes in hospital standardised mortality ratios. BMJ 2005, 330: 329 Aylin P, Bottle B, Jarman B, Elliott P. paediatric cardiac surgical mortality in England after Bristol: Descriptive analysis of hospital episode statistics 1991-2002. BMJ 2004;329:825 Marshall EC, Best NG, Bottle A, Aylin P. Statistical issues in the prospective monitoring of health outcomes at multiple units. J R Stat Soc Series A 2004;167(3):541-559. Jarman B, Aylin P, Bottle A. Acute stroke units and early CT scans are linked to lower in-hospital mortality rates. BMJ 2004;328:369. Spiegelhalter DJ, Aylin P, Best N, Evans SJW, Murray GD. Commissioned analysis of surgical performance using routine data: lessons from the Bristol Inquiry. J R Stat Soc Series A 2002;165:Part 2 pp1-31 Jarman et al. Explaining Differences in English Hospital Death Rates Using Routinely Collected Data. BMJ 1999;318:1515-1520

  40. Acknowledgements Imperial College Brian Jarman Alex Bottle Dr Foster Intelligence Steve Middleton Simon Jones

  41. Criticisms of HSMR HSMRs could be sensitive to admission thresholds more patients admitted more frequently, will increase denominator and decrease HSMR HSMRs do not take into account deaths occurring outside hospital Hospitals who tend to send patients home to die will tend to have lower HSMRs

  42. Calculate HSMR using three different methods Traditional, using in-hospital deaths Use patients’ last spell of care in financial year to counter multiple admissions for each patient Use ONS/HES linked mortality field to include all deaths occurring in 30 days includes in and out of hospital deaths

More Related