1 / 34

WHICH RISK ADJUSTMENT MODEL SHOULD WE USE? A FINNISH POINT OF VIEW

WHICH RISK ADJUSTMENT MODEL SHOULD WE USE? A FINNISH POINT OF VIEW. 16.3.2011 Matti Reinikainen North Karelia Central Hospital, Joensuu. THE FINNISH INTENSIVE CARE CONSORTIUM. 1994. 2007. So far, benchmarking in the Finnish Intensive Care Consortium has been mainly based on SAPS II

tocho
Download Presentation

WHICH RISK ADJUSTMENT MODEL SHOULD WE USE? A FINNISH POINT OF VIEW

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WHICH RISK ADJUSTMENT MODEL SHOULD WE USE? A FINNISH POINT OF VIEW 16.3.2011 Matti Reinikainen North Karelia Central Hospital, Joensuu

  2. THE FINNISH INTENSIVE CARE CONSORTIUM 1994 2007

  3. So far, benchmarking in the Finnish Intensive Care Consortium has been mainly based on SAPS II • Based on “The Severity Study” • 13 152 patients (720 from 7 Finnish hospitals) • Le Gall JR, Lemeshow S, Saulnier F. A new Simplified Acute Physiology Score (SAPS II) based on a European/North American multicenter study. JAMA 1993; 270: 2957-63. • APACHE II data is also collected

  4. APACHE II vs. SAPS II • same basic principle, values of physiologic parameters from the first 24 hrs in the ICU • APACHE II (Acute Physiology And Chronic Health Evaluation II): the diagnostic category weight is added to the logit • SAPS II (Simplified Acute Physiology Score II): the diagnosis is not needed; instead the type of admission (scheduled surgical, unscheduled surgical, medical) affects the score

  5. ARE THE OLD MODELS GOOD ENOUGH? • APACHE II - from 1985 - not always easy to choose the right diagnostic category • SAPS II - from 1993 - advantage: no diagnosis needed - disadvantage: does not take into account the diagnosis

  6. DOES THE RISK PREDICTED BY SAPS II REFLECT REALITY? • A patient example: • HR 110/min • SAPs 84 mmHg • Tc 38 ºC • consciousness, renal function, blood cell counts, electrolytes quite OK • HCO3-18 mmol/l • age 65 years • no difficult chronic diseases • a medical admission • respiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33.3 kPa) • PROBABILITY OF IN-HOSPITAL DEATH ?

  7. DOES THE RISK PREDICTED BY SAPS II REFLECT REALITY? • A patient example: • HR 110/min • SAPs 84 mmHg • Tc 38 ºC • consciousness, renal function, blood cell counts, electrolytes quite OK • HCO3-18 mmol/l • age 65 years • no difficult chronic diseases • a medical admission • respiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33,3 kPa) • SAPS II score 32 points → probability 0.128

  8. SAPS II –score 32 → probability 0.128 • the database of the Finnish Consortium, 1998-2007, readmissions excluded: 2319 patients, with a SAPS II score of 32 points • hospital mortality 8.4%

  9. SAPS II –score 32 → probability 0,128 • the database of the Finnish Consortium, 1998-2007, readmissions excluded: 2319 patients, with a SAPS II score of 32 points • hospital mortality 8.4% • diabetic ketoacidosis (n = 26): mort 0% • drug intoxication (n = 108): mort 0.9% • congestive heart failure (n = 49): mort 22.4%

  10. CAN SAPS II STILL BE USED? • It overestimates the risk of death – leads to ”grade inflation” • If most intensive care units are graduating with honors, is it genuine quality or grade inflation? Popovich MJ, Crit Care Med 2002 • Recalibrations are needed

  11. SMR 1998 – 2007, FINNISH INTENSIVE CARE CONSORTIUM SMR based on new calibration SMR based on original SAPS II model Päivitetty 09.04.2008

  12. CAN SAPS II STILL BE USED? • It can be used for monitoring changes in a unit’s own results • Can be used for benchmarking purposes if the units to be compared have similar case-mix • Should not be used to compare results of units with major differences in case-mix

  13. SAPS 3 WAS CONSIDERED IN FINLAND TOO - IS IT A GOOD ALTERNATIVE? • Values of physiological parameters ± 1 h of ICU admission • Reason for ICU admission documented more precisely than in SAPS II • Takes into account pre-ICU care • Prognostic performance? • Quality of data collected??

  14. The SAPS 3 StudyMetnitz et al ICM 2005: 31:1336-1344. (Part 1)Moreno et al. ICM 2005: 31:1345-1355. (Part 2) • At first 22,791 admissions • Exclusions: readmissions (1455), < 16 yrs (628), those without ICU admission or discharge data (1074) and those that lacked an entry in the field ”ICU outcome” (57) - SAPS 3 basic cohort: 19,577 patients

  15. The SAPS 3 StudyMetnitz et al ICM 2005: 31:1336-1344. (Part 1)Moreno et al. ICM 2005: 31:1345-1355. (Part 2) • SAPS 3 basic cohort: 19,577 patients • More exclusions: patients with a missing entry in the field of ”vital status at hospital discharge” (2540) and those still in hospital (253) • SAPS 3 Hospital outcome cohort: 16,784 patients • Quality of data? – at first, 5.5% of patients excluded because of missing data; then 13% of the remaining population excluded because of missing data on vital status

  16. The SAPS 3 StudyMetnitz et al ICM 2005: 31:1336-1344. (Part 1)Moreno et al. ICM 2005: 31:1345-1355. (Part 2) • How about data completeness? • ”Data completeness was found to be satisfactory with 1 [0-3] SAPS II parameter missing per patient” • How many SAPS 3 parameters were missing? • ??? • Were the physiological values obtained within ± 1 h?

  17. SAPS 3 – even if data quality in the study was less than perfect, does it work?

  18. Ledoux D et al. SAPS 3 admission score: an external validation in a general intensive care population. Intensive Care Med 2008; 34: 1873-7. single-centre (Belgium), 802 patients “the SAPS 3 … model customised for Central and Western Europe … was not significantly better than the SAPS II.” Poole D et al. External validation of the Simplified Acute Physiology Score (SAPS) 3 in a cohort of 28,357 patients from 147 Italian intensive care units. Intensive Care Med 2009; 35: 1916-24. “…the SAPS 3 score calibrates inadequately in a large sample of Italian ICU patients and thus should not be used for benchmarking, at least in Italian settings”

  19. Sakr Y et al. Comparison of the performance of SAPS II, SAPS 3, APACHE II, and their customized prognostic models in a surgical intensive care unit. Br J Anaesth 2008; 101: 798-803. • single-centre (Germany), 1851 patients • “… the performance of SAPS 3 was similar to that of APACHE II and SAPS II. Customization improved the calibration of all prognostic models.” • Metnitz B, Schaden E, Moreno R, Le Gall JR, Bauer P, Metnitz PG; ASDI Study Group. Austrian validation and customization of the SAPS 3 Admission Score. Intensive Care Med 2009; 35: 616-22. • 22 ICUs in Austria, 2060 patients • “The SAPS 3 … general equation can be seen as a framework … For benchmarking purposes, region-specific or country-specific equations seem to be necessary...”

  20. 2 ICUs in Norway, 1862 patients • “The performance of SAPS 3 was satisfactory, but not markedly better than SAPS II.”

  21. SAPS II showed better discrimination • SAPS 3 equations showed better calibration • “…in our experience the scoring process is more time-comsuming and complex than that for SAPS II.”

  22. SAPS 3, CONCLUSION: Does it work? – Yes! However, prognostic performance is NOT better than that of SAPS II the scoring process is more time-comsuming and complex than that for SAPS II (experience from Norway) on the other hand: according to many studies, the calibration of SAPS II is poor and customisation is needed

  23. QUESTION DISCUSSED IN FINLAND: Should we implement a new risk-adjustment model (SAPS 3) that is not better than the old ones is more time-consuming would require customisation Or should we go on with one of the old models (that also require customisation)?

  24. FINNISH (at least temporary) SOLUTION: OWN CUSTOMISED PREDICTION MODEL • One objective: no need to exclude patient groups for benchmarking • neuro- and cardiac surgical patients are not excluded • We did not want to increase the burden of data collection – no new parameters added • SAPS II –based data collection preserved • possible to compare the results with those of previous years • possible to describe the population using a well-known scoring system

  25. OWN CUSTOMISED MODEL - M Reinikainen, P Mussalo, V Kiviniemi, V Pettilä, E Ruokonen • Based on patients treated in 2007-2008 • Readmissions excluded • Age ≥ 18 yrs • Those discharged to another ICU excluded • n = 25 801

  26. OWN CUSTOMISED MODEL • Outcome variable (to be predicted) ”DEATH IN HOSPITAL” • Explaining covariates: • Emergency admission or planned beforehand • Surgical postoperative or medical • SAPS II score without admission type points • ln ((SAPS II score without admission type points) + 1) • Diagnostic groups having an independent impact on the probability of death • First a binary variable (0,1) was made of every APACHE III –dg group; everyone of these was tested separately • 31 dg groups with an independent effect were included in the model

  27. LOGISTIC REGRESSION ANALYSIS • logit = β0 + β1X1 + β2X2 + … + βiXi • - the regression analysis produces the constant β0 and the coefficients βi • the logit can be calculated when the parameter values Xi are known • the logit (log odds) can also be expressed as and thus

  28. CALCULATING THE RISK R

  29. LOGIT = -7,796 + 0,049 x (SCORE_SAPS_WITHOUT_ADM_TYPE_POINTS) + 1,013 x (ln(SAPS_WITHOUT_ADM_TYPE_POINTS + 1)) + 0,767 (if emergency admission) - 0,219 (if post-operative admission) + 1,229 (if DG_NONOP_CARDIOGENIC_SHOCK) + 0,364 (if DG_NONOP_CARDIAC_ARREST) – 0,796 (if DG_NONOP_RHYTHM_DISTURBANCE) + 0,348 (if DG_NONOP_ACUTE_MYOCARDIAL INFARCTION) + 0,422 (if DG_NONOP_BACTERIAL_OR_VIRAL_PNEUMONIA) – 1,619 (if DG_NONOP_MECHANICAL_AIRWAY_OBSTRUCTION) + 0,306 (if DG_NONOP_OTHER_RESP_DISEASES) + 0,795 (if DG_NONOP_HEPATIC_FAILURE) + 0,703 (if DG_NONOP_GI_PERFORATION_OR_OBSTRUCTION) + 0,643 (if DG_NONOP_GI BLEEDING_DUE_TO_VARICES) + 0,431 (if DG_NONOP_OTHER_GI_DISEASES) + 0,790 (if DG_NONOP_INTRACEREBRAL_HAEMORRHAGE) + 0,654 (if DG_NONOP_SUBARACHNOID_HAEMORRHAGE) + 0,400 (if DG_NONOP_STROKE) – 1,427 (if DG_NONOP_NEUROLOGIC_INFECTION) - 1,266 (if DG_NONOP_SEIZURE) – 0,486 (if DG_NONOP_OTHER_NEUROLOGIC_DISEASES) - 0,679 (if DG_NONOP_MULTIPLE TRAUMA_WITHOUT_HEAD_TRAUMA) – 0,658 (if DG_NONOP_METABOLIC_COMA) – 2,126 (IF DG_NONOP_DIABETIC_KETOACIDOSIS) – 2,245 (if DG_NONOP_DRUG_OVERDOSE) – 1,150 (if DG_NONOP_OTHER_METABOLIC_DISEASES) – 0,752 (if DG_NONOP_OTHER MEDICAL_DISEASES) + 0,340 (if DG_POSTOP_DISSECTING_OR_RUPTURED_AORTA) – 0,701 (if DG_POSTOP_CABG) + 0,701 (if DG_POSTOP_PERIPH_ARTERY_BYPASS_GRAFT) + 0,470 (if DG_POSTOP_GI_PERFORATION_OR_RUPTURE) + 0,411 (if DG_POSTOP GI_OBSTRUCTION) – 0,522 (if DG_POSTOP_SUBDURAL_OR_EPIDURAL_HAEMATOMA) – 0,885 (if DG_POSTOP_CRANIOTOMY_FOR_NEOPLASM) – 1,620 (if DG_POSTOP_OTHER_RENAL_DISEASES) PROB = EXP(LOGIT) / (1 + EXP(LOGIT))

  30. PATIENT EXAMPLE A patient example: • HR 110/min • SAPs 84 mmHg • Tc 38 ºC • consciousness, renal function, blood cell counts, electrolytes quite OK • HCO3-18 mmol/l • age 65 years • no difficult chronic diseases • a medical admission • respiratory insufficiency, need for mechanical venti-lation, PaO2/FIO2 250 mmHg (33.3 kPa) • SAPS II score 32 points → probability 0.128

  31. PATIENT EXAMPLE • SAPS II score 32 → probability 0.128 • New customised model: • If none of the diagnoses included in the model: probability 0.082 • dg bacterial pneumonia: probability 0.12 • dg drug intoxication: probability 0.0094

  32. AUROC • APACHE II: 0.84 • SAPS II: 0.84 • new customised model: 0.87 • H-L test for new model: p = 0.127

  33. CONCLUSIONS SAPS 3 works, but its prognostic performance is not better than that of SAPS II If you want to use SAPS 3, you should probably customise it If you want to use SAPS II, you should probably customise it Idea for future research: to create a Nordic risk adjustment model, predicting 6-month or 1-year mortality

More Related