External Validation of clinical prediction models - Statswork

1. External Validation of clinical prediction models Dr. Nancy Agnes, Head, Technical Operations, Statswork info@statswork.com Validation, particularly external validation, II. FACTORS INFLUENCES AFFECT is a crucial part of developing a predictive EXTERNAL VALIDATION DATA model. External validation is needed to The sample size for external validation ensure that a prediction model is data for the implementation of the generalizable to patients other than those prediction model is affected by the number in the derivative cohort. External of events and predictors. validation can be done by testing the External validation of the prediction model model's output in data that isn't the same as requires a minimum of 100 events and/or the data used to create the model. As a non-events, according to simulation consequence, it is carried out after the studies, and a systematic analysis found creation of a prediction model. that small external validation studies are I. EXTERNAL VALIDATION ineffective and inaccurate.Example: External validation can take many forms, Radiology imaging is often treated as including validation in the field such as effective predictive parameters and temporal, geographical and independent researchers often validate the findings validation. For external validation studies, using clinical prediction model. Every the sample size calculation estimates prediction model is based on the regression analysis. The most common based on statistical power considerations have not been extensively investigated. predictive model or the regression model However, in order to achieve adequate used for the clinical prediction model are model output in the validation set, a large linear regression if the dependent variable sample size is needed to validate the is continuous in nature, logistic regression prediction model. model if the dependent variable is binary, and Cox-proportional model if the dependent variable is time-to-event in Copyright © 2021 Statswork. All rights 1

2. nature. Al-Ameri et al (2020) presented a identified the validity using the calibration slope and the sample articles are detailed review on clinical prediction models for liver transplantation study. presented in the following table Further, Ratna et al (2020) discussed the quality of clinical prediction model in vitro fertilisation and human reproduction. Validation of model has been carried out using re-sampling technique and measured the accuracy using AUC, calibration plot as shown in figure 1, c-index, and Hosmer- Lemeshow test statistic. Figure 1: Slope of Calibration plot (Source: Stevens and Poppe (2020)) In addition, Stevens and Poppe (2020) Table1.Stated Interpretation of the “Calibration Slope” Source: Stevens and Poppe (2020) suggested the Cox- calibration slope using logistic regression model instead of using Arjun et al (2020) considered the pandemic mortality study of COVID19 and discussed the development and validation of clinical prediction model. simply the calibration slope for the predictive model. This suggestion has been made after the scrutiny of around 33 research articles and found that most of the validation are external validation and Copyright © 2021 Statswork. All rights 2

3. II. FUTURE SCOPE Though many literature suggests several validation techniques for the predictive model, there is no such proper technique which can be suitable for all the clinical datasets. Further, proper adjustment has to be made for the calibration index to validate the prediction model suitable for all clinical datasets. References: 1. Stevens, R. J. and Poppe, K. K. (2020). Validation of Clinical Prediction Models: What does the "Calibration Slope" Really Measure?. Journal of clinical epidemiology, 118, pp. 93– 99. Adibi, A., Sadatsafavi, M., Ioannidis, J. P. A. (2020). Validation and Utility Testing of Clinical Prediction Models: Time to Change the Approach. JAMA. 2020; 324(3):235–236. Ratna, M. B., Bhattacharya, S., Abdulrahim, B. and McLernon, D. L. (2020). A Systematic Review of the Quality of Clinical Prediction Models in Vitro Reproduction, 35(1), pp. 100–116 Arjun S Yadaw., Yan-chak Li., Sonali Bose., Ravi Iyengar., Supinda Bunyavanich., Gaurav Pandey. (2020). Clinical Features of COVID- 19 Mortality: Development and Validation of a Clinical Prediction Model, The Lancet Digital Health, 2(10), pp. 516-525. Al‐Ameri, A.A.M., Wei, X., Wen, X., Wei, Q., Guo, H., Zheng, S. and Xu, X. (2020), Systematic review: risk prediction models for recurrence of hepatocellular carcinoma after liver transplantation. Transpl Int, 33, pp. 697- 712. 2. 3. Fertilisation, Human 4. 5. Copyright © 2021 Statswork. All rights 3