Skip to main content

01.12.2018 | Letter | Ausgabe 1/2018 Open Access

Critical Care 1/2018

Trustworthy or flawed clinical prediction rule?

Critical Care > Ausgabe 1/2018
Anders Granholm, Anders Perner, Aksel Karl Georg Jensen, Morten Hylander Møller
Wichtige Hinweise
This comment refers to the article available at: https://​doi.​org/​10.​1186/​s13054-017-1888-6.
Extracorporeal membrane oxygenation
We read with interest the recently published paper by Hilder et al. [ 1], where the authors present the PRESET-Score, a new clinical prediction rule for patients with acute respiratory distress syndrome treated with extracorporeal membrane oxygenation (ECMO). While the topic is clinically relevant and interesting, we are worried that spurious findings, biased results, and overstated findings are presented.
First, both the new and the four existing scores assessed are at high risk of being underpowered. Multivariable risk prediction models should be based on an effective sample size (lowest number of events/non-events) of at least 10, often more, per predictor variable assessed [ 2, 3]. Using 11 variables and 41 non-events (3.7 per predictor) results in overfitting of the development sample and inflated performance estimates [ 2]. This will be evident upon use of the score in other populations.
Second, comparing the performance of the new score with four existing scores using the development dataset is against recommendations [ 2], as this is biased to favor the new score due to overfitting. For comparison with other scores, an independent cohort not used to develop any of the scores must be used [ 2].
Third, internal validation is performed to quantify overfitting, and should be done by bootstrap resampling of the development dataset [ 2]. The authors state that they used logistic regression analysis to “reassess” the score, which essentially is a recalibration resulting in a new model generating new predictions. This is neither internal nor external validation, which requires assessment of predictions made by the score without modifications in a new sample [ 2].
Fourth, it is recommended to assess calibration by graphical methods or regressions of the predicted versus observed outcomes [ 2, 4], not by the Hosmer-Lemeshow Ĉ-test, as P > 0.05 is more likely to indicate lack of power than proper model fit when used on small samples.
While we agree that clinical prediction rules may be valuable for clinicians considering ECMO, it is a prerequisite that such scores are developed and validated using appropriate methodology [ 2] and sufficient sample sizes, and that all relevant features are transparently reported with adequate discussion of the limitations [ 5]. Developing and sufficiently validating a clinical prediction rule for this highly selected patient group likely requires a large, multicentre collaboration to ensure trustworthy predictions that will benefit patients and relatives, the healthcare system, researchers, and society.


Not applicable.



Availability of data and materials

Not applicable.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://​creativecommons.​org/​licenses/​by/​4.​0/​), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver ( http://​creativecommons.​org/​publicdomain/​zero/​1.​0/​) applies to the data made available in this article, unless otherwise stated.
Über diesen Artikel

Weitere Artikel der Ausgabe 1/2018

Critical Care 1/2018 Zur Ausgabe

Neu im Fachgebiet AINS

Mail Icon II Newsletter

Bestellen Sie unseren kostenlosen Newsletter Update AINS und bleiben Sie gut informiert – ganz bequem per eMail.