nach oben

Journal of Nuclear Cardiology

Erschienen in:

24.05.2022 | Theme Articles

Statistical methods for validation of predictive models

verfasst von: Marcio Augusto Diniz, Ph.D

Erschienen in: Journal of Nuclear Cardiology | Ausgabe 6/2022

Einloggen, um Zugang zu erhalten

Abstract

Predictive models are widely used in clinical practice. Despite of the increasing number of published AI systems, recent systematic reviews have identified lack of statistical rigor in the development and validation of predictive models. This work reviewed the current literature for predictive performance measures and resampling methods. Furthermore, common pitfalls were discussed.

Nur mit Berechtigung zugänglich

Wilson PW, D’Agostino RB, Levy D, Belanger AM, SIlbershatz H, Kannel WB. Prediction of coronary heart disease using risk factor categories. Circulation 1998;97:1837‐47. https://doi.org/10.1161/01.cir.97.18.1837.CrossRef

Malinchoc M, Kamath PS, Gordon FD, Peine CJ, Rank J, ter Borg PCJ. A model to predict poor survival in patients undergoing transjugular intrahepatic portosystemic shunts. Hepatology 2000;31:864‐71. https://doi.org/10.1053/he.2000.5852.CrossRef

Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a severity of disease classification system. Crit Care Med 1985;13:818‐29.CrossRef

Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med 2019;25:44‐56. https://doi.org/10.1038/s41591-018-0300-7.CrossRef

Otaki Y, Miller RJH, Slomka PJ. The application of artificial intelligence in nuclear cardiology. Ann Nucl Med 2022;36:111‐22. https://doi.org/10.1007/s12149-021-01708-2.CrossRef

Kim DW, Jang HY, Kim KW, Shin Y, Park SH. Design characteristics of studies reporting the performance of artificial intelligence algorithms for diagnostic analysis of medical images: results from recently published papers. Korean J Radiol 2019;20:405. https://doi.org/10.3348/kjr.2019.0025.CrossRef

Kelly CJ, Karthikesalingam A, Suleyman M, Corrado G, King D. Key challenges for delivering clinical impact with artificial intelligence. BMC Med 2019;17:195. https://doi.org/10.1186/s12916-019-1426-2.CrossRef

Moons KGM, Altman DG, Reitsma J, Ioannidis JPA, Macaskill P, Steyerberg EW. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD Statement. Ann Intern Med 2015;67:1142‐51. https://doi.org/10.1016/j.eururo.2014.11.025.CrossRef

Yusuf M, Atal I, Li J, Smith P, Ravaud P, Fergie M. Reporting quality of studies using machine learning models for medical diagnosis: a systematic review. BMJ Open 2020. https://doi.org/10.1136/bmjopen-2019-034568.CrossRef

10.

Andaur Navarro CL, Damen JAA, Takada T, Nijman SWJ, Dhiman P, Ma J, et al. Completeness of reporting of clinical prediction models developed using supervised machine learning: a systematic review. BMC Med Res Methodol 2022;22:12. https://doi.org/10.1186/s12874-021-01469-6.CrossRef

11.

Collins GS, Dhiman P, Andaur Navarro CL, Ma J, Hooft L, Reitsma Navarro JB, et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open 2021. https://doi.org/10.1136/bmjopen-2020-048008.CrossRef

12.

Vergouwe Y, Moons KGM, Steyerberg EW. External validity of risk models: use of benchmark values to disentangle a case-mix effect from incorrect coefficients. Am J Epidemiol 2010;172:971‐80. https://doi.org/10.1093/aje/kwq223.CrossRef

13.

Poldervaart JM, Reitsma JB, Koffijberg H, Backus BE, Six AJ, Doevendans PA, et al. The impact of the HEART risk score in the early assessment of patients with acute chest pain: design of a stepped wedge, cluster randomised trial. BMC Cardiovasc Disord 2013;13:77. https://doi.org/10.1186/1471-2261-13-77.CrossRef

14.

Hutchings HA, Evans BA, Fitzsimmons D, Harrison J, Heaven M, Huxley P, et al. Predictive risk stratification model: a progressive cluster-randomised trial in chronic conditions management (PRISMATIC) research protocol. Trials 2013;14:301. https://doi.org/10.1186/1745-6215-14-301.CrossRef

15.

Kappen TH, Moons KGM, van Wolfswinkel L, van Wolfswinkel L, Kalkman CJ, Vergouwe Y, et al. Impact of risk assessments on prophylactic antiemetic prescription and the incidence of postoperative nausea and vomiting: a cluster-randomized trial. Anesthesiology 2014;120:343‐54. https://doi.org/10.1097/ALN.0000000000000009.CrossRef

16.

Long E, Lin H, Liu Z, Wu X, Wang L, Jiang J, et al. An artificial intelligence platform for the multihospital collaborative management of congenital cataracts. Nat Biomed Eng 2017;1:0024. https://doi.org/10.1038/s41551-016-0024.CrossRef

17.

Brocklehurst P, Field D, Greene K, Juszczak E, Keith R, Kenyon S, et al. Computerised interpretation of fetal heart rate during labour (INFANT): a randomised controlled trial. The Lancet 2017;389:1719‐29. https://doi.org/10.1016/S0140-6736(17)30568-8.CrossRef

18.

Titano JJ, Badgeley M, Schefflein J, Pain M, Su A, Cai M, et al. Automated deep-neural-network surveillance of cranial images for acute neurologic events. Nat Med 2018;24:1337‐41. https://doi.org/10.1038/s41591-018-0147-y.CrossRef

19.

Lin H, Li R, Liu Z, Chen J, Yang Y, Chen H, et al. Diagnostic efficacy and therapeutic decision-making capacity of an artificial intelligence platform for childhood cataracts in eye clinics: a multicentre randomized controlled trial. EClinicalMedicine 2019;9:52‐9. https://doi.org/10.1016/j.eclinm.2019.03.001.CrossRef

20.

Wu L, Zhang J, Zhou W, An P, Shen L, Liu J, et al. Randomised controlled trial of WISENSE, a real-time quality improving system for monitoring blind spots during esophagogastroduodenoscopy. Gut 2019;68:2161‐9. https://doi.org/10.1136/gutjnl-2018-317366.CrossRef

21.

Gneiting T, Raftery AE. Strictly proper scoring rules, prediction, and estimation. J Am Stat Assoc 2007;102:359‐78. https://doi.org/10.1198/016214506000001437.CrossRef

22.

Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol 2016;74:167‐76. https://doi.org/10.1016/j.jclinepi.2015.12.005.CrossRef

23.

Brier GW. Verification of Forecasts expressed in terms of probability. Monthly Weather Rev 1950;78:1‐3. https://doi.org/10.1175/1520-0493(1950)078%3c0001:VOFEIT%3e2.0.CO;2.CrossRef

24.

Murphy AH, Epstein ES. A note on probability forecasts and “hedging.” J Appl Meteor 1967;6:1002‐4. https://doi.org/10.1175/1520-0450(1967)006%3c1002:ANOPFA%3e2.0.CO;2.CrossRef

25.

Hosmer DW, Hosmer T, Le Cessie S, Lemeshow S. A comparison of goodness-of-fit tests for the logistic regression model. Stat Med 1997;16:965‐80. https://doi.org/10.1002/(SICI)1097-0258(19970515)16:9%3c965::AID-SIM509%3e3.0.CO;2-O.CrossRef

26.

Tsiatis AA. A note on a goodness-of-fit test for the logistic regression model. Biometrika 1980;67:250‐1. https://doi.org/10.1093/biomet/67.1.250.CrossRef

27.

le Cessie S, van Houwelingen JC. A goodness-of-fit test for binary regression models based on smoothing methods. Biometrics 1991;47:1267. https://doi.org/10.2307/2532385.CrossRef

28.

Royston P. The use of cusums and other techniques in modelling continuous covariates in logistic regression. Statist Med 1992;11:1115‐29. https://doi.org/10.1002/sim.4780110813.CrossRef

29.

Cox DR. Two further applications of a model for binary regression. Biometrika 1958;45:562‐5. https://doi.org/10.1093/biomet/45.3-4.562.CrossRef

30.

Copas JB (1983) Plotting p against x. Applied Statistics 32:25. https://doi.org/10.2307/2348040

31.

Cleveland WS. Robust locally weighted regression and smoothing scatterplots. J Am Stat Assoc 1979;74:829‐36. https://doi.org/10.1080/01621459.1979.10481038.CrossRef

32.

Austin PC, Steyerberg EW. Graphical assessment of internal and external calibration of logistic regression models by using loess smoothers. Stat Med 2014;33:517‐35. https://doi.org/10.1002/sim.5941.CrossRef

33.

Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Mak 2006;26:565‐74. https://doi.org/10.1177/0272989X06295361.CrossRef

34.

Van Calster B, Vickers AJ. Calibration of risk prediction models: impact on decision-analytic performance. Med Decis Mak 2015;35:162‐9. https://doi.org/10.1177/0272989X14547233.CrossRef

35.

Vickers AJ, van Calster B, Steyerberg EW. A simple, step-by-step guide to interpreting decision curve analysis. Diagn Progn Res 2019;3:18. https://doi.org/10.1186/s41512-019-0064-7.CrossRef

36.

Pencina MJ, D’Agostino RB, RS, Vasan. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 2008;27:157‐72. https://doi.org/10.1002/sim.2929.CrossRef

37.

Pencina MJ, D’Agostino RB, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med 2011;30:11‐21. https://doi.org/10.1002/sim.4085.CrossRef

38.

Kerr KF, Wang Z, Janes H, McClelland RL, Psaty BM, Pepes MS. Net reclassification indices for evaluating risk prediction instruments. Epidemiology 2014;25:114‐21. https://doi.org/10.1097/EDE.0000000000000018.CrossRef

39.

Hilden J, Gerds TA. A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index. Stat Med 2014;33:3405‐14. https://doi.org/10.1002/sim.5804.CrossRef

40.

Pepe MS, Fan J, Feng Z, Gerds T, Hilden J. The net reclassification index (NRI): a misleading measure of prediction improvement even with independent test data sets. Stat Biosci 2015;7:282‐95. https://doi.org/10.1007/s12561-014-9118-0.CrossRef

41.

Pencina MJ, Steyerberg EW, D’Agostino RB. Net reclassification index at event rate: properties and relationships. Stat Med 2017;36:4455‐67. https://doi.org/10.1002/sim.7041.CrossRef

42.

Steyerberg EW. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. Journal of Clinical Epidemiology 2001;8:774‐81.CrossRef

43.

Braga-Neto UM, Dougherty ER. Is cross-validation valid for small-sample microarray classification? Bioinformatics 2004;20:374‐80. https://doi.org/10.1093/bioinformatics/btg419.CrossRef

44.

Kim J-H. Estimating classification error rate: repeated cross-validation, repeated hold-out and bootstrap. Comput Stat Data Anal 2009;53:3735‐45. https://doi.org/10.1016/j.csda.2009.04.009.CrossRef

45.

Borra S, Di Ciaccio A. Measuring the prediction error. A comparison of cross-validation, bootstrap and covariance penalty methods. Comput Stat Data Anal 2010;54:2976‐89. https://doi.org/10.1016/j.csda.2010.03.004.CrossRef

46.

Picard RR, Berk KN. Data splitting. Am Stat 1990;44:140‐7. https://doi.org/10.1080/00031305.1990.10475704.CrossRef

47.

Stone M. Cross-validatory choice and assessment of statistical predictions. J R Stat Soc: Ser B (Methodol) 1974;36:111‐33. https://doi.org/10.1111/j.2517-6161.1974.tb00994.x.CrossRef

48.

Allen DM. The relationship between variable selection and data agumentation and a method for prediction. Technometrics 1974;16:125‐7.CrossRef

49.

Geisser S. The predictive sample reuse method with applications. J Am Stat Assoc 1975;70:320‐8. https://doi.org/10.1080/01621459.1975.10479865.CrossRef

50.

Kohavi R. A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Ijcai. Montreal, Canada, 1995, p. 1137-1145

51.

Burman P. A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods. Biometrika 1989;76:503‐14.CrossRef

52.

Varma S, Simon R. Bias in error estimation when using cross-validation for model selection. BMC Bioinform 2006;7:91. https://doi.org/10.1186/1471-2105-7-91.CrossRef

53.

Vabalas A, Gowen E, Poliakoff E, Casson AJ. Machine learning algorithm validation with a limited sample size. PLoS ONE 2019;14:e0224365. https://doi.org/10.1371/journal.pone.0224365.CrossRef

54.

Wainer J, Cawley G. Nested cross-validation when selecting classifiers is overzealous for most practical applications. Expert Syst Appl 2021;182:115222.CrossRef

55.

Bates S, Hastie T, Tibshirani R. Cross-validation: what does it estimate and how well does it do it? arXiv:210400673 [math, stat], 2021

56.

Efron B, Gong G. A leisurely look at the bootstrap, the jackknife, and cross-validation. Am Stat 1983;37:36‐48. https://doi.org/10.1080/00031305.1983.10483087.CrossRef

57.

Harrell J. Regression modeling strategies: with applications to linear models, logistic and ordinal regression, and survival analysis. 2nd ed. Cham: Springer; 2015.CrossRef

58.

Efron B. Estimating the error rate of a prediction rule: improvement on cross-validation. J Am Stat Assoc 1983;78:316‐31. https://doi.org/10.1080/01621459.1983.10477973.CrossRef

59.

Breiman L. Out-of-bag estimation, ftp. stat. berkeley. edu/pub/users/breiman. OOBestimation ps 199, 1996

60.

Efron B, Tibshirani R. Improvements on cross-validation: the 632+ bootstrap method. J Am Stat Assoc 1997;92:548‐60. https://doi.org/10.1080/01621459.1997.10474007.CrossRef

61.

Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW On behalf of Topic Group ‘Evaluating diagnostic tests and prediction models’ of the STRATOS initiative. Calibration: the Achilles heel of predictive analytics. BMC Med 2019;17:230. https://doi.org/10.1186/s12916-019-1466-7

62.

Krstajic D, Buturovic LJ, Leahy DE, Thomas S. Cross-validation pitfalls when selecting and assessing regression and classification models. J Cheminform 2014;6:10. https://doi.org/10.1186/1758-2946-6-10.CrossRef

63.

Hastie TT. The elements of statistical learning. Math Intell 2009;27:83‐5. https://doi.org/10.1007/b94608.CrossRef

Titel: Statistical methods for validation of predictive models
verfasst von: Marcio Augusto Diniz, Ph.D
Publikationsdatum: 24.05.2022
Verlag: Springer International Publishing
Erschienen in: Journal of Nuclear Cardiology / Ausgabe 6/2022
Print ISSN: 1071-3581
Elektronische ISSN: 1532-6551
DOI: https://doi.org/10.1007/s12350-022-02994-7

Neu im Fachgebiet Kardiologie

Nach Herzinfarkt mit Typ-1-Diabetes schlechtere Karten als mit Typ 2?

29.05.2024 Herzinfarkt Nachrichten

Bei Menschen mit Typ-2-Diabetes sind die Chancen, einen Myokardinfarkt zu überleben, in den letzten 15 Jahren deutlich gestiegen – nicht jedoch bei Betroffenen mit Typ 1.

Erhöhtes Risiko fürs Herz unter Checkpointhemmer-Therapie

28.05.2024 Nebenwirkungen der Krebstherapie Nachrichten

Kardiotoxische Nebenwirkungen einer Therapie mit Immuncheckpointhemmern mögen selten sein – wenn sie aber auftreten, wird es für Patienten oft lebensgefährlich. Voruntersuchung und Monitoring sind daher obligat.

GLP-1-Agonisten können Fortschreiten diabetischer Retinopathie begünstigen

24.05.2024 Diabetische Retinopathie Nachrichten

Möglicherweise hängt es von der Art der Diabetesmedikamente ab, wie hoch das Risiko der Betroffenen ist, dass sich sehkraftgefährdende Komplikationen verschlimmern.

TAVI versus Klappenchirurgie: Neue Vergleichsstudie sorgt für Erstaunen

21.05.2024 TAVI Nachrichten

Bei schwerer Aortenstenose und obstruktiver KHK empfehlen die Leitlinien derzeit eine chirurgische Kombi-Behandlung aus Klappenersatz plus Bypass-OP. Diese Empfehlung wird allerdings jetzt durch eine aktuelle Studie infrage gestellt – mit überraschender Deutlichkeit.

Update Kardiologie

Bestellen Sie unseren Fach-Newsletter und bleiben Sie gut informiert.

Newsletter bestellen

Live-Webinar "Urologie und Sexualmedizin in der Praxis"

Springer Medizin

Statistical methods for validation of predictive models

Abstract

Neu im Fachgebiet Kardiologie

Nach Herzinfarkt mit Typ-1-Diabetes schlechtere Karten als mit Typ 2?

Erhöhtes Risiko fürs Herz unter Checkpointhemmer-Therapie

GLP-1-Agonisten können Fortschreiten diabetischer Retinopathie begünstigen

TAVI versus Klappenchirurgie: Neue Vergleichsstudie sorgt für Erstaunen

Update Kardiologie

Live-Webinar "Urologie und Sexualmedizin in der Praxis"

Springer Medizin

Abstract

Bitte loggen Sie sich ein, um Zugang zu diesem Inhalt zu erhalten

Weitere Artikel der Ausgabe 6/2022

Incremental value of 18F-FDG cardiac PET imaging over dobutamine stress echocardiography in predicting myocardial ischemia in patients with suspected coronary artery disease

Mediastinum-type lung cancer on 68Ga-FAPI PET/CT with cardiac insufficiency as the first symptom

A new perspective for phase analysis of radionuclide angiocardiography

Semi-quantitative assessment of ischemia with rubidium-82 PET myocardial perfusion imaging

Right ventricular ischemia by myocardial perfusion imaging in a 6-year-old child with pulmonary hypertension after ventricular septal defect repair

Deep learning-based attenuation map generation and correction; could it be useful clinically?

Neu im Fachgebiet Kardiologie

Nach Herzinfarkt mit Typ-1-Diabetes schlechtere Karten als mit Typ 2?

Erhöhtes Risiko fürs Herz unter Checkpointhemmer-Therapie

GLP-1-Agonisten können Fortschreiten diabetischer Retinopathie begünstigen

TAVI versus Klappenchirurgie: Neue Vergleichsstudie sorgt für Erstaunen

Update Kardiologie