There is limited research comparing the calibration performance (evaluated by the integrated calibration index (ICI)) of prediction models for binary outcomes. Furthermore, there is no research comparing the calibration performance using the ICI between penalized regression methods for high-dimensional settings. This study compared the parsimony for calibration of penalized logistic regression methods (LASSO, adaptive LASSO, adaptive double LASSO, and adaptive elastic-net) in two clinical datasets with different events per variable (EPV). The results revealed that the adaptive double LASSO achieved the best parsimony for calibration and discrimination in Study 1 (EPV = 0.98) and Study 2 (EPV = 1.6). These results demonstrate that adaptive double LASSO is a powerful variable selection method that emphasizes parsimony for calibration in high-dimensional settings. Our findings are expected to provide important information when developing clinical prediction models, emphasizing parsimony for calibration.

clinical prediction model, binary outcome, calibration index, integrated calibration index, high-dimensional settings.

Received: September 2, 2021; Accepted: October 7 2021; Published:  October 20, 2021

How to cite this article: Yuki Shiko, Ippeita Dan and Yohei Kawasaki, Comparison of penalized regression methods for optimizing the parsimonyfor calibration performance of clinical prediction model in high-dimensional settings, JP Journal of Biostatistics 18(3) (2021), 437-457. DOI: 10.17654/BS018010437

This Open Access Article is Licensed under Creative Commons Attribution 4.0 International License

Reference:

[1] J. A. N. Dorresteijn et al., Estimating treatment effects for individual patients based on the results of randomised clinical trials, BMJ 343(1) (2011), d5888-d5888. doi: 10.1136/bmj.d5888.
[2] R. A. Hayward, D. M. Kent, S. Vijan and T. P. Hofer, Multivariable risk prediction can greatly enhance the statistical power of clinical trial subgroup analysis, BMC Med. Res. Methodol. 6(1) (2006), 18. doi: 10.1186/1471-2288-6-18.
[3] E. Steyerberg, Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating, Springer-Verlag, New York, 2019.
[4] F. E. Harrell, Jr., K. L. Lee and D. B. Mark, Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors, Stat. Med. 15 (1996), 361-387. doi: https://doi.org/10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168 >3.0.C O;2-4.
[5] P. C. Austin and E. W. Steyerberg, The Integrated Calibration Index (ICI) and related metrics for quantifying the calibration of logistic regression models, Stat. Med. 38(21) (2019), 4051-4065. doi: 10.1002/sim.8281.
[6] P. C. Austin, F. E. Harrell and D. Klaveren, Graphical calibration curves and the integrated calibration index (ICI) for survival models, Stat. Med. 39(21) (2020), 2714-2742. doi: 10.1002/sim.8570.
[7] D. W. Hosmer and S. Lemeshow, Goodness of fit tests for the multiple logistic regression model, Comm. Statist. Theory Methods 9(10) (1980), 1043-1069. doi: 10.1080/03610928008827941.
[8] D. W. Hosmer, T. Hosmer, S. L. Cessie and S. Lemeshow, A comparison of goodness-of-fit tests for the logistic regression model, Stat. Med. 16(9) (1997), 965-980. doi: https://doi.org/10.1002/(SICI)1097-0258(19970515)16:9<965::AID-SIM509 =>3.0.C O;2-O.
[9] Y. Huang, W. Li, F. Macheret, R. A. Gabriel and L. Ohno-Machado, A tutorial on calibration measurements and calibration models for clinical prediction models, J. Am. Med. Inform. Assoc. 27(4) (2020), 621-633. doi: 10.1093/jamia/ocz228.
[10] J. Wang et al., Development and internal validation of risk prediction model of metabolic syndrome in oil workers, BMC Public Health 20(1) (2020), 1828. doi: 10.1186/s12889-020-09921-w.
[11] S. Hong, S. Lee, J. Lee, W. C. Cha and K. Kim, Prediction of cardiac arrest in the emergency department based on machine learning and sequential characteristics: model development and retrospective clinical validation study, JMIR Med. Inform. 8(8) (2020), e15932. doi: 10.2196/15932.
[12] Y. Shiko, I. Takashima, I. Dan and Y. Kawasaki, Comparison of variable selection methods for optimizing the calibration of clinical prediction model, JP J. Biostat. 18(2) (2021), 269-294.
[13] L. Chen, H. Zheng and S. Wang, Prediction model of emergency mortality risk in patients with acute upper gastrointestinal bleeding: a retrospective study, Peer J. 9 (2021), e11656. doi: 10.7717/peerj.11656.
[14] Y. Chen et al., Predicting the risk of sarcopenia in elderly patients with patellar fracture: development and assessment of a new predictive nomogram, Peer J. 8 (2020), e8793. doi: 10.7717/peerj.8793.
[15] R. Tibshirani, Regression shrinkage and selection via the Lasso: a retrospective, J. R. Stat. Soc. Ser. B Stat. Methodol. 73(3) (2011), 273-282. doi: 10.1111/j.1467-9868.2011.00771.x.
[16] H. Zou, The adaptive Lasso and its oracle properties, J. Amer. Statist. Assoc. 101(476) (2006), 1418-1429. doi: 10.1198/016214506000000735.
[17] S. Tibshirani and H. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed., Springer-Verlag, New York, 2009.
[18] H. Zou and T. Hastie, Regularization and variable selection via the elastic net, J. R. Stat. Soc. Ser. B Stat. Methodol. 67(2) (2005), 301-320. doi: 10.1111/j.1467-9868.2005.00503.x.
[19] W. S. Cleveland and E. Grosse, Computational methods for local regression, Stat. Comput. 1(1) (1991), 47-62. doi: 10.1007/BF01890836.
[20] L. N. Sanchez-Pinto, L. R. Venable, J. Fahrenbach and M. M. Churpek, Comparison of variable selection methods for clinical predictive modeling, Int. J. Med. Inf. 116 (2018), 10-17. doi: 10.1016/j.ijmedinf.2018.05.006.
[21] F. Bagherzadeh-Khiabani, A tutorial on variable selection for clinical prediction models: feature selection methods in data mining could improve the results, J. Clin. Epidemiol. 71 (2016), 76-85.
[22] G. S. Collins, J. B. Reitsma, D. G. Altman and K. G. M. Moons, Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement, BMJ 350 (2015), g7594-g7594. doi: 10.1136/bmj.g7594.
[23] N. Meinshausen, Relaxed Lasso, Comput. Stat. Data Anal. 52(1) (2007), 374-393. doi: 10.1016/j.csda.2006.12.019.
[24] C. J. Greenwood, G. J. Youssef, P. Letcher, et al., A comparison of penalized regression methods for informing the selection of predictive markers, Y. Kotozaki, ed., PLOS ONE 15(11) (2020), e0242730. doi: 10.1371/journal.pone.0242730.
[25] M. M. Vasquez, C. Hu, D. J. Roe, Z. Chen, M. Halonen and S. Guerra, Least absolute shrinkage and selection operator type methods for the identification of serum biomarkers of overweight and obesity: simulation and application, BMC Med. Res. Methodol. 16(1) (2016), 154. doi: 10.1186/s12874-016-0254-8.
[26] M. Pavlou, G. Ambler, S. Seaman, M. De Iorio and R. Z. Omar, Review and evaluation of penalised regression methods for risk prediction in low-dimensional data with few events, Stat. Med. 35(7) (2016), 1159-1177. doi: 10.1002/sim.6782.