Multinational development and validation of an early prediction model for delirium in ICU patients
verfasst von:
A. Wassenaar, M. van den Boogaard, T. van Achterberg, A. J. C. Slooter, M. A. Kuiper, M. E. Hoogendoorn, K. S. Simons, E. Maseda, N. Pinto, C. Jones, A. Luetz, A. Schandl, W. Verbrugghe, L. M. Aitken, F. M. P. van Haren, A. R. T. Donders, L. Schoonhoven, P. Pickkers
Delirium incidence in intensive care unit (ICU) patients is high and associated with poor outcome. Identification of high-risk patients may facilitate its prevention.
Purpose
To develop and validate a model based on data available at ICU admission to predict delirium development during a patient’s complete ICU stay and to determine the predictive value of this model in relation to the time of delirium development.
Methods
Prospective cohort study in 13 ICUs from seven countries. Multiple logistic regression analysis was used to develop the early prediction (E-PRE-DELIRIC) model on data of the first two-thirds and validated on data of the last one-third of the patients from every participating ICU.
Results
In total, 2914 patients were included. Delirium incidence was 23.6 %. The E-PRE-DELIRIC model consists of nine predictors assessed at ICU admission: age, history of cognitive impairment, history of alcohol abuse, blood urea nitrogen, admission category, urgent admission, mean arterial blood pressure, use of corticosteroids, and respiratory failure. The area under the receiver operating characteristic curve (AUROC) was 0.76 [95 % confidence interval (CI) 0.73–0.77] in the development dataset and 0.75 (95 % CI 0.71–0.79) in the validation dataset. The model was well calibrated. AUROC increased from 0.70 (95 % CI 0.67–0.74), for delirium that developed <2 days, to 0.81 (95 % CI 0.78–0.84), for delirium that developed >6 days.
Conclusion
Patients’ delirium risk for the complete ICU length of stay can be predicted at admission using the E-PRE-DELIRIC model, allowing early preventive interventions aimed to reduce incidence and severity of ICU delirium.
The online version of this article (doi:10.1007/s00134-015-3777-2) contains supplementary material, which is available to authorized users.
Take home message: The early delirium prediction model is the first model that enables clinicians to identify those patients likely to develop delirium immediately after ICU admission. Identification of these high-risk patients is paramount to facilitate delirium prevention.
Introduction
Delirium is an acute organic brain dysfunction characterised by disturbances of attention and cognition with a fluctuating course, as a direct consequence of an underlying medical condition [1]. Delirium occurs frequently in intensive care unit (ICU) patients [2, 3], and is associated with poor outcome [3‐6].
Systematic delirium assessment in ICU patients is important to deliver adequate patient care by allowing clinicians to detect and treat delirium at an early stage [7‐9]. Several promising interventions to prevent delirium are available that target cognitive impairment, sleep deprivation, immobility, and visual and hearing impairment [10‐15]. Although one study found no beneficial effects of haloperidol as prophylaxis and treatment for ICU delirium [16, 17], other studies have shown that prophylactic haloperidol may exert beneficial effects in critically ill patients [18, 19].
Anzeige
As part of a process to improve delirium care and to further facilitate prevention of ICU delirium, identification of high-risk patients is paramount [20], since these are the most fragile patients who require maximum preventive efforts [21]. Also, when preventive measures can be restricted to high-risk patients, both the number of patients who will be unnecessarily exposed to potential harmful side effects [22] and costs are likely less. Moreover, the effectiveness of prevention seems more pronounced in patients with a higher risk for delirium [18]. Finally, prediction of the ICU delirium risk provides relevant information for family members and caregivers and will facilitate stratification of patients in future delirium prevention studies. This stratification will positively influence the efficient use of research resources.
Currently, an ICU delirium prediction model is available [23, 24]. However, the usefulness of this model is limited by the fact that it requires predictors obtained during the first 24 h of ICU admission. A relevant number of ICU patients develop delirium within 24 h following admission [25, 26], and preventive measures should ideally be applied as early as possible. This calls for a prediction model that can reliably predict delirium at the time of ICU admission. Therefore, the objective of this multinational study was to develop and validate a model based on data available at ICU admission to predict the development of delirium during a patient’s complete ICU stay, and to determine the predictive value of this model in relation to the time of development of delirium.
Some results of this study have been previously presented at the ISICEM congress in Brussels, March 2014, the ESICM congress in Barcelona, October 2014 [27], and the EDA congress in Cremona, November 2014 [28].
Methods
Design and study population
A multinational prospective cohort study was carried out in 13 ICUs from seven countries (Australia, Belgium, England, Germany, Spain, Sweden, and the Netherlands) (Table E1 online data supplement). Each participating ICU included all consecutive patients aged ≥18 years (surgical, medical, neurology/neurosurgical, or trauma patients) during a continuous period of approximately three months between October 2011 and June 2012. Patients were excluded if: they were delirious at ICU admission; had an ICU stay shorter than one day; were unable to reliably assess for delirium (e.g., coma during entire ICU stay; unable to understand the language spoken; severely mentally disabled; serious receptive aphasia; serious auditory or visual disorders); or if the compliance rate of the CAM-ICU for delirium screening was <80 % during an individual patient’s ICU stay (Fig. 1).
Fig. 1
Study flow chart
×
Anzeige
This study was approved by the medical ethical research committee Arnhem-Nijmegen region, The Netherlands (CMO Region Arnhem-Nijmegen, no. 2010/365), and conducted in accordance with the applicable rules concerning the review of research ethics committees and informed consent. All participating centres obtained ethics approval from their committees to conduct the study.
Data collection
Data of 18 candidate delirium predictors were collected at ICU admission in an electronic clinical report form using a secured website. Candidate predictors were selected based on the PRE-DELIRIC model [23, 24], a systematic review about risk factors for delirium [29] and on expert opinion [gathered during the annual meeting of the working group Postoperative Delirium and Cognitive Dysfunction (PoDeCoD) of the European Society of Intensive Care Medicine (ESICM)]. A proposal with candidate predictors, based on the PRE-DELIRIC model and the systematic review, was presented and discussed. Candidate predictors were removed from/added to the final list based on consensus of the members of this working group.
Candidate predictors available at time of ICU admission were: age, gender, history of cognitive impairment (history of dementia, mild cognitive impairment or delirium), history of alcohol, nicotine and drugs abuse, history of vascular disease, Glasgow Coma Scale (GCS) score, diabetes, blood urea nitrogen (BUN), use of opiates in the 24 h before ICU admission, use of anti-psychotics before ICU admission, admission category (surgery, medical, trauma, neurology/neurosurgery), urgent admission, mean arterial blood pressure (MAP), (strong suspicion of) infection, use of corticosteroids, or respiratory failure (see Table E2 online data supplement for exact definitions). Data on sedation and illness severity (APACHE) were not collected, as these are not available at ICU admission.
Delirium assessment
All consecutive patients were screened for delirium by trained ICU nurses at least once every 8- or 12-h shift using the CAM-ICU. In accordance with previous prediction studies [23, 24], delirium was diagnosed when the CAM-ICU was positive [7, 30], and/or when a patient was treated with haloperidol or other anti-psychotics because of delirium (see Table E2). Patients were screened using the CAM-ICU as soon as delirium assessment was possible after awakening from sedation. To exclude a potential source of bias, assessors of the CAM-ICU were not made aware of the fact that they were collecting data for this study [31].
In order to check the quality of the delirium assessments, compliance with the CAM-ICU was per participating centre calculated monthly as the percentage of assessments performed per day in relation to the total number of assessments that should have been performed. Also, inter-rater reliability (IRR) measurements of the CAM-ICU were performed by duplicate measurements of the CAM-ICU for all admitted ICU patients, on one given day per month per centre, by comparing CAM-ICU scores assessed by the ICU nurse with the scores assessed by a CAM-ICU expert. CAM-ICU experts and ICU nurses were blinded to each others delirium assessments. If per centre, the CAM-ICU screening compliance was above 80 % and the Cohen’s kappa was above 0.80, their data were considered reliable. Centres that did not meet these criteria were per centre excluded from the analysis, after which re-analysis was performed. If exclusion of a centre did not significantly affect the performance of the E-PRE-DELIRIC model, the centre was included in the final analysis, as the data were considered to be of sufficient quality [23, 24].
Sensitivity and specificity of the delirium assessments were calculated using the CAM-ICU outcomes obtained by CAM-ICU experts compared with those of the ICU nurses.
Outcome definition
The primary outcome of this study was the development of ICU delirium, defined as a positive assessment for delirium during a patient’s complete ICU stay.
Statistical analysis
Statistical analyses, including information regarding missing values, are described in Supplemental file 1 online data supplement. The model was developed using multiple logistic regression analysis on data of the first two-thirds of the patients of all participating hospitals. Selection of the final predictors of the E-PRE-DELIRIC model took place through manual backward selection of the candidate predictors using a P value of >0.10 for exclusion. The model was validated on data of the last one-third of the patients. Discriminative power was assessed using the area under the receiver operating characteristic curve (AUROC) [31]. Calibration was assessed graphically by plotting the observed outcome frequencies against the mean predicted outcome probabilities or risks, within subgroups of patients that were ranked by increasing estimated probability [32]. To evaluate the predictive value of the E-PRE-DELIRIC model related to the moment at which delirium first occurred, the database was divided into four groups based on the quartiles of the time to development of delirium at: days 0–1; day 2; days 3–6; >day 6.
Anzeige
Data were analyzed using IBM® SPSS® Statistics 20.01 and R statistics R 3.0.1 [33]. In this study, we used the Transparent Reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD), a strict methodology for data reporting, to check our adherence to suggested optimal levels of transparency and completeness of reporting of a prediction model [34].
Sample size calculation
The E-PRE-DELIRIC model was compiled from 18 candidate predictors. For each candidate predictor, at least ten patients with delirium were needed for the validation and calibration of the model [35]. With an anticipated delirium incidence of 15–30 %, an expected attrition of 25 %, we aimed to enrol at least (18 × 10/0.15)/0.75 = 1600 patients.
Results
In total, 5352 patients were screened, of whom 2438 patients did not fulfil the inclusion criteria (Fig. 1). The cohort consisted of 2914 patients of whom 1962 patients were included in the development dataset. The ICU delirium incidence was 24.5 %. The remaining 952 patients were included in the validation dataset with an ICU delirium incidence of 21.8 %. The demographic characteristics of both groups were comparable (Table 1).
Table 1
Patient characteristics
Variable
Development dataset (n = 1962)
Validation dataset (n = 952)
Whole study population (n = 2914)
Age in years, mean (Q1–Q3, min/max)
61.7 (53–74, 18/95)
60.6 (51–73, 18/94)
61.3 (52–73, 18/95)
Male, n (%)
1166 (59.4)
550 (57.8)
1716 (58.9)
Admission category, n (%)
Surgery
1019 (51.9)
476 (50.0)
1495 (51.3)
Medical
683 (34.8)
338 (35.5)
1021 (35.0)
Trauma
90 (4.6)
44 (4.6)
134 (4.6)
Neurology/neurosurgery
170 (8.7)
94 (9.9)
264 (9.1)
Urgent admission, n (%)
1163 (59.3)
570 (59.9)
1733 (59.5)
LOS-ICU in days, median (Q1–Q3, min/max)
2.0 (1–6, 1/133)
2.0 (1–5, 1/125)
2.0 (1–5, 1/133)
Delirium, n (%)
481 (24.5)
208 (21.8)
689 (23.6)
CAM-ICU compliance and inter-rater reliability
As exclusion of the centres with a CAM-ICU screening compliance <80 % and/or a Cohen’s kappa <0.80 did not significantly affect the performance of the E-PRE-DELIRIC model, all centres were subsequently included in the final analysis. The overall CAM-ICU compliance was 83 % (Q1–Q3: 78–93). In total, 648 inter-rater CAM-ICU measurements were performed. The mean IRR measurements were 0.83 [95 % confidence interval (CI) 0.78–0.88] Cohen’s kappa (Table E3 online data supplement). The CAM-ICU had a sensitivity of 0.82 (95 % CI 0.76–0.87) and a specificity of 0.98 (95 % CI 0.96–0.99).
Anzeige
Discrimination and calibration of the E-PRE-DELIRIC model
The developed E-PRE-DELIRIC model consists of nine predictors: age, history of cognitive impairment, history of alcohol abuse, BUN at time of ICU admission, admission category, urgent admission, MAP at the time of ICU admission, use of corticosteroids, and respiratory failure (Table 2). The discriminative power (AUROC) was 0.76 (95 % CI 0.73–0.78) in the development dataset. This database was divided in groups based on the quartiles of the predicted probabilities for delirium development: very low (0–10 %), low (10–20 %), moderate (20–35 %), and high risk for delirium (>35 %). For each of these groups, the sensitivity, specificity and likelihood ratios were calculated (Fig. 2). At a cut-off of 24.5 %, which was the ICU delirium incidence in the development dataset, the sensitivity and specificity were 71 and 69 %, respectively. The AUROC was 0.75 (95 % CI 0.71–0.79) in the validation dataset (see Figure E1 online data supplement for the calibration plot of the validation dataset). Fitting the model Log(Odds) = b0 + b1 × LP on the observed data showed that recalibration was not necessary. The slope of the model was 0.961 with an intercept of −0.094.
Table 2
Variables of the E-PRE-DELIRIC model and regression coefficients and formula for the E-PRE-DELIRIC model
Variable
Regression coefficient
Odds ratio (95 % CI)
Age
0.025
1.025 (1.017–1.033)
History of cognitive impairment
0.878
2.406 (1.700–3.404)
History of alcohol abuse
0.505
1.657 (1.170–2.347)
Admission category
Surgery
RC
RC
Medical
0.370
1.448 (1.058–1.982)
Trauma
1.219
3.384 (1.997–5.735)
Neurology/neurosurgery
0.504
1.655 (1.064–2.575)
Urgent admission
0.612
1.843 (1.326–2.563)
MAP at the time of ICU admission
−0.006
0.994 (0.988–1.001)
Use of corticosteroids
0.283
1.327 (0.996–1.768)
Respiratory failure
0.982
2.670 (2.099–3.396)
BUN at time of ICU admission
0.018
1.018 (1.006–1.031)
Data development set. Formula for the E-PRE-DELIRIC model: Risk of delirium = 1/[1 + exp − (−3.907 + 0.025 × age + 0.878 for history of cognitive impairment + 0.505 for history of alcohol abuse + 0 for surgery or 0.370 for medical or 1.219 for trauma or 0.504 for neurology/neurosurgery + 0.612 for urgent admission − 0.006 × MAP at the time of ICU admission + 0.283 for use of corticosteroids + 0.982 for respiratory failure + 0.018 x BUN in mmol/l at time of ICU admission)]
RC reverence category
Fig. 2
Area under the receiver operating characteristic curve (AUROC) of development dataset (AUROC = 0.76)
×
Prediction of early and late development of delirium
In patients who developed delirium at days 0–1 the AUROC was 0.70 (95 % CI 0.67–0.74) with a sensitivity and specificity of 62 and 67 %, respectively. This increased to 0.76 (95 % CI 0.72–0.80) (development of delirium at day 2), 68 and 67 %; 0.77 (95 % CI 0.74–0.80) (development of delirium at days 3–6), 72 and 68 %; and 0.81 (95 % CI 0.78–0.84) (development of delirium after day 6), 78 and 68 %, respectively. The predictive value of the model improved significantly over time (Fig. 3).
Fig. 3
Area under the receiver operating characteristic curve (AUROC) during ICU stay. Error bars 95 % CIs
×
Discussion
In this prospective, international multicentre study, we developed and validated a model allowing early prediction of patients likely to develop delirium during ICU admission. Using nine predictors that are readily available at ICU admission, it is now possible to stratify ICU patients for delirium risk for the complete ICU length of stay with a high discriminative power. In contrast to the earlier prediction model [23, 24], which reliably predicts delirium using data available only after 24 h in the ICU, our E-PRE-DELIRIC model has a moderate-to-good performance using data available immediately at ICU admission. Of interest, the predictive accuracy of the model is higher for delirium that develops later compared to delirium that develops early following ICU admission.
In view of the high prevalence of delirium in ICU patients [2, 3] and its serious consequences [3‐6], prediction of ICU delirium is clearly of clinical relevance as early risk identification may serve different purposes. Both caregivers and patients families can be informed early about the risk of developing delirium in the ICU, and this may, for example, assist them in taking decisions related to the use of preventive measures. In addition, the model enables stratification for the risk to develop delirium, which may be helpful in future trials that aim to investigate preventive interventions. Our group previously reported the effects of prevention with low-dose haloperidol in critically ill patients using a prediction model and demonstrated that patients with the highest risk appear to benefit most from prevention [18]. With the E-PRE-DELIRIC model, stratification is possible immediately after ICU admission. Various other preventive measures that target modifiable delirium risk factors have been found beneficial in non-critically ill patients [8, 10, 11] and need confirmation in ICU patients. In these confirmation trials, stratification of the a priori risk to develop delirium should be determined to demonstrate that treatment groups are adequately randomized. As a consequence, possible interventions that may be expensive and/or labour intensive can then be provided only to those patients with highest risk. In this way, improved quality and efficiency of care may be achieved in a cost-effective manner [36].
Anzeige
The predictors that build the E-PRE-DELIRIC-model are largely consistent with previously reported risk factors for delirium, including age, pre-existing dementia, history of alcoholism, and a high severity of illness at admission [8, 37, 38]. Other factors that confirm previously reported predictors include admission category, urgent admission, BUN [23, 24], and respiratory failure [37‐39]. While use of corticosteroids was not reported as a relevant factor in another study [2], in our study the use of corticosteroids was a significant predictor for delirium. We only collected data about corticosteroids use at ICU admission, in contrast, the authors of [2] collected daily data about corticosteroids use during the whole study period. Of importance, as relevant differences between countries may exist, the current study was a multinational study using data of ICUs from different countries for the development of a delirium prediction model, which increases its generalisability. Interestingly, while the delirium incidence remains stable over time in the four groups in this study, the predictive value of the model increases over time. However, examining differences between early and late delirium did not fall within the scope of this study, and therefore should be studied in future research.
Several limitations of this study need to be addressed. First, despite the fact that availability of candidate predictors at ICU admission and the effort it would take to collect the data in daily clinical practice were leading in determining candidate predictors, three of the predictors from the E-PRE-DELIRIC model (BUN, history of alcohol abuse, and cognitive impairment) had a fairly high rate of missing values. Probably due to unawareness of the importance of these predictors for delirium prediction, currently these predictors are not collected in a standardised manner in daily practice. When health professionals need the information about these predictors to be able to use the E-PRE-DELIRIC model, they will (most often) be able to collect the information as they are readily available. We used imputation techniques to handle missing data. Multiple imputation helps to reduce bias and increases precision by including variation of data [31]. Second, the performance of the CAM-ICU in daily practice is not optimal [40‐42]. However, in contrast to the evaluation study of Van Eijk et. al [40], in the current study ICU patients were screened for delirium at least two times daily and during the whole ICU length of stay. This increases the chance of correct delirium scores, because of the fluctuating course of delirium. The scores were determined in the same way in all participating ICUs, positively influencing possible measurement bias. In addition, data on the use of haloperidol and other anti-psychotics for delirium were used [23]. This usage was consistent among the different centres. Therefore, we may assume that sensitivity of the bedside nurse to recognize delirium anywhere during ICU admission is high. The quality of data of the CAM-ICU screening was further improved by determination of compliance and inter-rater reliability in each ICU. Third, although the full model approach to include all 18 candidate predictors in the predictive model is likely to avoid overfitting and selection data [43], we chose to perform manual backward selection of predictors to improve the clinical feasibility. Fourth, based on the predetermined exclusion criteria, a large number of patients had to be excluded. One of these exclusion criteria was an ICU stay shorter than one day. Since the E-PRE-DELIRIC model will be used prospectively following ICU admission, one may not be able to anticipate a patient’s ICU length of stay (LOS). Therefore, for each patient admitted to the ICU, the E-PRE-DELIRIC score will be calculated. Predictions for patients with an ICU LOS shorter than one day might be different than those for patients with an ICU LOS longer than one day. Therefore, in future research, it is recommended to include these patients to determine the influence on the performance of the E-PRE-DELIRIC model. Finally, it should be taken into account that selection of candidate predictors in the E-PRE-DELIRIC model might be biased. However, the candidate predictors were based on multiple scientific sources, including the previously published PRE-DELIRIC model [23, 24], a systematic review about delirium risk factors [29], and expert opinion. Although external validation of the E-PRE-DELIRIC model in new centres was not performed, this prediction model was internationally developed and prospectively validated in 13 ICUs from seven different countries. The E-PRE-DELIRIC model is a result of a relevant sample from these ICUs, and therefore widely applicable and probably generalisable to other countries. ICUs should consider, however, that the prediction of delirium derived from the E-PRE-DELIRIC model is an approximation; there can always be some over- or underestimation in the predicted delirium risk. Therefore, for future research, it is recommended to test the external validity of the E-PRE-DELIRIC model in other centres to further improve its generalisability.
In conclusion, in this multinational study, we developed and validated an early ICU delirium prediction model that revealed sufficient validity. It enables the clinician to identify those patients likely to develop delirium following ICU admission using only nine predictors. The model allows early delirium preventive interventions in ICU patients with a high risk of delirium.
The authors would like to thank all co-investigators, their research nurses, and their intensive care teams at each of the participating sites for their help in collecting the patient data. There was no external funding for this study.
Conflicts of interest
The authors declare that they have no conflict of interest.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Mit e.Med Innere Medizin erhalten Sie Zugang zu CME-Fortbildungen des Fachgebietes Innere Medizin, den Premium-Inhalten der internistischen Fachzeitschriften, inklusive einer gedruckten internistischen Zeitschrift Ihrer Wahl.
Mit e.Med Anästhesiologie erhalten Sie Zugang zu CME-Fortbildungen des Fachgebietes AINS, den Premium-Inhalten der AINS-Fachzeitschriften, inklusive einer gedruckten AINS-Zeitschrift Ihrer Wahl.
Multinational development and validation of an early prediction model for delirium in ICU patients
verfasst von
A. Wassenaar M. van den Boogaard T. van Achterberg A. J. C. Slooter M. A. Kuiper M. E. Hoogendoorn K. S. Simons E. Maseda N. Pinto C. Jones A. Luetz A. Schandl W. Verbrugghe L. M. Aitken F. M. P. van Haren A. R. T. Donders L. Schoonhoven P. Pickkers
Ein Frauenanteil von mindestens einem Drittel im ärztlichen Op.-Team war in einer großen retrospektiven Studie aus Kanada mit einer signifikanten Reduktion der postoperativen Morbidität assoziiert.
Sie sei „ethisch geboten“, meint Gesundheitsminister Karl Lauterbach: mehr Transparenz über die Qualität von Klinikbehandlungen. Um sie abzubilden, lässt er gegen den Widerstand vieler Länder einen virtuellen Klinik-Atlas freischalten.
Um die Langzeitfolgen eines Delirs bei kritisch Kranken zu mildern, wird vielerorts auf eine Akuttherapie mit Antipsychotika gesetzt. Eine US-amerikanische Forschungsgruppe äußert jetzt erhebliche Vorbehalte gegen dieses Vorgehen. Denn es gibt neue Daten zum Langzeiteffekt von Haloperidol bzw. Ziprasidon versus Placebo.
Gesundheitsminister Lauterbach hat die vom Bundeskabinett beschlossene Klinikreform verteidigt. Kritik an den Plänen kommt vom Marburger Bund. Und in den Ländern wird über den Gang zum Vermittlungsausschuss spekuliert.
Update AINS
Bestellen Sie unseren Fach-Newsletter und bleiben Sie gut informiert.
