There is limited research comparing the calibration performance (evaluated by the ICI) of prediction models for binary outcomes. Furthermore, when developing clinical prediction models, variable selection methods are used, but previous comparisons between analysis methods that optimize ICI are insufficient. The aim of this study was to compare the calibration performance of different variable selection methods (linear probability model (LPM), logistic model, and classification and regression tree) in two different clinical datasets of different sizes for predictive modeling. The results showed that although the optimal variable selection method for ICI differs depending on the size of the dataset, LPM showed the best calibration performance compared to the other methods. Our findings are expected to provide important information when developing clinical prediction models with an emphasis on calibration.

clinical prediction model, calibration index, integrated calibration index, variable selection.

Received: May 14, 2021; Accepted: June 17, 2021; Published: June 28, 2021

How to cite this article: Yuki Shiko, Ikumi Takashima, Ippeita Dan and Yohei Kawasaki, Comparison of variable selection methods for optimizing the calibration of clinical prediction model, JP Journal of Biostatistics 18(2) (2021), 269-294. DOI: 10.17654/JB018020269

This Open Access Article is Licensed under Creative Commons Attribution 4.0 International License

Reference:

[1] E. Steyerberg, Clinical Prediction Models: A Practical Approach to Development, Validation and Updating, New York: Springer-Verlag New York, 2019.
[2] F. E. H. Jr, K. L. Lee and D. B. Mark, Multivariable Prognostic Models: Issues in Developing Models, Evaluating Assumptions and Adequacy, and Measuring and Reducing Errors, Published online (1996), 361-387. doi: https://doi.org/10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168> 3.0.CO;2-4
[3] F. Harrell Jr., Regression Modeling Strategies: With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis, New York: Springer-Verlag New York, 2015.
[4] G. S. Collins, J. A. de Groot and S. Dutton, et al., External validation of multivariable prediction models: a systematic review of methodological conduct and reporting, BMC Med Res Methodol. 14(1) (2014), 40. doi:10.1186/1471-2288-14-40
[5] G. S. Collins and D. G. Altman, Predicting the 10 year risk of cardiovascular disease in the United Kingdom: independent and external validation of an updated version of QRISK 2, BMJ 344(1) (2012), e4181-e4181. doi:10.1136/bmj.e4181
[6] P. C. Austin and E. W. Steyerberg, The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models, Stat Med. 38(21) (2019), 4051-4065. doi:10.1002/sim.8281
[7] G. W. Brier, Verification of Fore Casts Express Ed in terms, Mon Weather Rev. 78(1) (1950), 1-3.
[8] D. W. Hosmer and S. Lemesbow, Goodness of fit tests for the multiple logistic regression model, Commun Stat-Theory Methods 9(10) (1980), 1043-1069. doi:10.1080/03610928008827941
[9] Y. Wang, L. Li and C. Dang, Calibrating Classification Probabilities with Shape-Restricted Polynomial Regression, IEEE Trans Pattern Anal Mach Intell. 41(8) (2019), 1813-1827. doi:10.1109/TPAMI.2019.2895794
[10] Y. Huang, W. Li, F. Macheret, R. A. Gabriel and L. Ohno-Machado, A tutorial on calibration measurements and calibration models for clinical prediction models, J Am Med Inform Assoc. 27(4) (2020), 621-633. doi:10.1093/jamia/ocz228
[11] D. R. Cox, Two Further Applications of a Model for Binary Regression, Biometrika 45(3/4) (2021), 562-565. doi:https://doi.org/10.2307/2333203
[12] D. W. Hosmer, T. Hosmer, S. L. Cessie and S. Lemeshow, A Comparison of Goodness-of-1-fit tests for the logistic regression model, Stat Med. 16(9) (1997), 965-980. doi:https://doi.org/10.1002/(SICI)1097-0258(19970515)16:9<965::AIDSIM509>3.0.CO;2-O
[13] J. Wang, C. Li, J. Li, et al., Development and internal validation of risk prediction model of metabolic syndrome in oil workers, BMC Public Health. 20(1) (2020), 1828. doi:10.1186/s12889-020-09921-w
[14] S. Hong, S. Lee, J. Lee, W. C. Cha and K. Kim, Prediction of Cardiac Arrest in the Emergency Department Based on Machine Learning and Sequential Characteristics: Model Development and Retrospective Clinical Validation Study, JMIR Med Inform. 8(8) (2020), e15932. doi:10.2196/15932
[15] Y. Chen, Y. Cai, X. Kang, et al., Predicting the risk of sarcopenia in elderly patients with patellar fracture: development and assessment of a new predictive nomogram, Peer J. 8 (2020), e8793. doi:10.7717/peerj.8793
[16] L. Sun, J. Shang, J. Xiao and Z. Zhao, Development and validation of a predictive model for end-stage renal disease risk in patients with diabetic nephropathy confirmed by renal biopsy, Peer J. 8 (2020), e8499. doi:10.7717/peerj.8499
[17] S. B. Kotsiantis, Decision trees: a recent overview, Artif Intell Rev. 39 (2013), 261-283.
[18] R. Tibshirani, Regression Shrinkage and Selection Via the Lasso, J R Stat Soc Ser B Methodol. 58(1) (1996), 267-288. doi:10.1111/j.2517-6161.1996.tb02080.x
[19] W. S. Cleveland and E. Grosse, Computational methods for local regression, Stat Comput. 1(1) (1991), 47-62. doi:10.1007/BF01890836
[20] P. C. Austin and E. W. Steyerberg, Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers, Stat Med. 33(3) (2014), 517-535. doi:10.1002/sim.5941
[21] P. Royston, K. G. Moons, A. D. Altman and Y. Vergouwe, Prognosis and prognostic research: Developing a prognostic model, BMJ 338 (2009), b604. doi: https://doi.org/10.1136/bmj.b604
[22] K. G. M. Moons, D. G. Altman, J. B. Reitsma, et al., Transparent reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): Explanation and Elaboration, Ann Intern Med. 162(1) (2015), W1. doi:10.7326/M14-0698
[23] F. Bagherzadeh-Khiabani, A tutorial on variable selection for clinical prediction models: feature selection methods in data mining could improve the results, J Clin Epidemiol. Published online 2016:10.
[24] K. G. M. Moons, A. P. Kengne, M. Woodward, et al., Risk prediction models: I. Development, internal validation, and assessing the incremental value of a new (bio)marker, Heart 98(9) (2012), 683-690. doi:10.1136/heartjnl-2011-301246
[25] N. Mantel, Why Stepdown Procedures in Variable Selection, Technometrics 12(3) (1970), 621-625. doi:10.1080/00401706.1970.10488701
[26] T. van der Ploeg and E. W. Steyerberg, Feature selection and validated predictive performance in the domain of Legionella pneumophila: a comparative study, BMC Res Notes 9(1) (2016), 147. doi:10.1186/s13104-016-1945-2
[27] L. N. Sanchez-Pinto, L. R. Venable, J. Fahrenbach and M. M. Churpek, Comparison of variable selection methods for clinical predictive modeling, Int. J Med Inf. 116 (2018), 10-17. doi:10.1016/j.ijmedinf.2018.05.006
[28] Z. Bursac, C. H. Gauss, D. K. Williams and D. W. Hosmer, Purposeful selection of variables in logistic regression, Source Code Biol Med. 3(1) (2008), 17. doi:10.1186/1751-0473-3-17
[29] T. van der Ploeg, P. C. Austin and E. W. Steyerberg, Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints, BMC Med Res Methodol. 14(1) (2014), 137. doi:10.1186/1471-2288-14-137