Delirium prediction in the intensive care unit: comparison of two delirium prediction models

Delirium, defined as acute brain dysfunction featured by disturbances of attention, awareness, and cognition with a fluctuating course caused by an underlying medical condition [1], occurs frequently in the intensive care unit (ICU), is associated with impaired patient outcome, and substantially increases healthcare costs [2, 3]. Given these deleterious consequences, delirium prevention is crucial.

Delirium preventive measures are important for all ICU patients. However, a delirium prediction model may facilitate early recognition of the patients who may benefit the most from delirium prevention [4]. In the case of limited resources, non-pharmacologic reduction strategies and medication-based strategies may be most relevant for patients who have an increased risk of developing delirium. Prediction models may aid clinical decision making and setting of priorities regarding the use of delirium preventive measures. For instance, when deciding which patient should be admitted to the available room with adequate natural daylight and which patient to the room without it, preferably the patient with the highest delirium risk should be admitted to the room with adequate natural daylight. Also, the use of a delirium prediction model facilitates patient selection for studies on delirium prevention, which is not only efficient in terms of reducing waste, but it may also increase the chance of finding an effect, which ultimately might improve ICU patients’ outcomes. Furthermore, family members can be informed about the patient’s risk of developing delirium and be engaged to help provide strategies to reduce delirium (e.g. cognitive activities) [5]. Involvement of family in patient care in the ICU is stimulated by many ICU societies worldwide [6, 7] and might even increase the prevalence of interventions for delirium prevention and treatment in the ICU [8].

Two delirium prediction models have been validated for use in critically ill adults admitted to the ICU [9‐11]. The prediction model for delirium in ICU patients (PRE-DELIRIC model) was developed and validated in a large cohort of Dutch ICU patients [9]. This model, which was recently recalibrated in a multinational cohort [10], reliably predicts ICU patients’ risk of delirium using ten predictors obtained within the first 24 h of ICU admission [10]. However, given that up to 25% of critically ill adults develop delirium within the first 24 h of ICU admission [12, 13], and delirium prevention strategies should be deployed as early as possible, an early prediction model (E-PRE-DELIRIC) was developed to predict the risk of delirium the moment a patient is admitted to the ICU [11]. This E-PRE-DELIRIC model was developed and validated in a multinational cohort and uses nine predictors to predict ICU patients’ risk of delirium [11].

It remains unclear which ICU delirium prediction model might be recommended for daily clinical practice, because the comparative predictive performance of the PRE-DELIRIC and the E-PRE-DELIRIC models and clinicians’ preferences have not been assessed [14]. Therefore, the objective of this study was to compare the predictive performance and user convenience of the PRE-DELIRIC and E-PRE-DELRIC models. Second, we sought to determine the value of the use of both models in a two-stage calculation of patients’ risk of delirium in the ICU (i.e. the E-PRE-DELIRIC model immediately after ICU admission with an updated delirium risk score after 24 h of ICU admission using the PRE-DELIRIC model) to see if we could expand on current models, since it is well-known that dynamic variables assessed over time as opposed to variables assessed at admission only tend to perform better in prediction models.

A total of 2802 patients were screened for inclusion; 2178 patients (78%) were included. Among the 624 patients excluded, inability to reliably assess for delirium (46.3% (289/624)) and delirium at the time of ICU admission (25.9% (162/624)) were the most common reasons for exclusion (see study flowchart, Fig. 1). Among the 83 patients that were excluded for other reasons the most common reasons were severe neurological injury and confidential files. Patients were 62.1 ± 15.2 years old, 60.8% male, and had a baseline APACHE-II score of 17.4 ± 7.1. During their ICU stay 21.4% of patients (467/2178) developed delirium. Patient characteristics are presented in Table 1. For patient and hospital characteristics per participating ICU see Additional file 2: Table S1.

This large, multinational prospective cohort study provides insight into the comparative performance of two available ICU delirium prediction models (i.e. the E-PRE-DELIRIC model that estimates the risk of delirium at the time of ICU admission and the PRE-DELIRIC model used to estimate the risk of delirium 24 h later) [9‐11]. Both models had a moderate-to-good statistical performance. Although the predictive accuracy of the E-PRE-DELIRIC model was somewhat lower, its user convenience appeared to be better compared to the PRE-DELIRIC model. To allow for optimal implementation of a delirium prediction model in daily practice, involvement and the opinion on user convenience of the target group, i.e. the ICU physicians, is very important [28, 29]. Based on these results, the E-PRE-DELIRIC model is likely the model that can be implemented most successfully in daily ICU practice. Moreover, our analysis indicates that when the E-PRE-DELIRIC model predicts a low risk of delirium, an additional calculation using the PRE-DELIRIC model after 24 h in the ICU increases the model’s sensitivity to detect patients that will develop delirium who are incorrectly identified as low-risk patients. Thus, this method will prevent deprivation of delirium-preventive measures in patients with a false negative rating (i.e. the patients with predicted probability <0.30 who develop delirium during ICU admission).

The routine use of delirium preventive measures in the ICU is widely endorsed given the high prevalence of delirium, its deleterious effects on patient outcome [30, 31], and the high costs related to these effects [32]. The routine use of a delirium prediction model may facilitate early recognition of those patients at greatest risk of delirium who may benefit the most from delirium preventive measures [32]. Importantly, when resources are limited, the use of delirium risk stratification to target high-risk patients makes wider implementation of multicomponent non-pharmacological interventions aimed at preventing delirium more feasible [32]. Preventive measures should be initiated as soon as possible after ICU admission; therefore an early ICU delirium prediction model is preferred.

The performance of a prediction model outside the development sample determines its generalisability in clinical practice [33]. External validation of many other clinical prediction models is lacking [34]. Of interest, both the E-PRE-DELIRIC and recalibrated PRE-DELIRIC models are validated externally and had moderate-to-good statistical performance in independent data sets, allowing for generalisation to non-study ICUs around the world [[35], Wassenaar et al. 2017]. “External validation of two models to predict delirium in intensive care unit patients. Unpublished data.” The delirium incidence, estimated based on new positive delirium assessments after ICU admission, is important for the performance and thus the generalisation of both delirium prediction models. One might argue that the delirium incidence in our study cohort was relatively low. However, multiple previous studies have reported comparable delirium incidence rates [36‐38].

Our study has important strengths. The use of a cohort design without strict eligibility criteria helped boost its generalisability [14], and its prospective nature allowed us to carefully measure and document the predictors and outcomes, thereby improving its applicability and reproducibility in non-study ICUs [14, 20, 39]. The large number of patients enrolled, their mixed characteristics, and the multinational character of our study allows the results to be applied in the vast majority of ICUs in the developed world. Of note, when generalising to high-intensity ICUs it should be taken into account that their patient group will probably have more severe illness in comparison with our study cohort. For future research it might be of interest to study the performance of both models in patients with APACHE and SOFA scores in the higher ranges. The proportion of ICU physicians responding was better than response rates shown in other physician surveys [40]. No imputation techniques were used to handle missing data, as we wanted to determine the clinical performance of both delirium prediction models and the use of a prediction model in daily clinical practice does not allow for imputation. Our efforts to provide clear definitions and instruction manuals to all study sites resulted in the exclusion of only fourteen patients due to missing values for the predictors.

Several limitations are also present. It might be possible that the two delirium prediction models evaluated might need to be updated in the future as new risk factors for delirium in the ICU may emerge. Of course, this also offers an opportunity to further improve the discriminative performance of each model, which in particular could benefit the E-PRE-DELIRIC. For an update, referred to as model revision, one needs to have insight into new risk factors for delirium, both available at ICU admission or within 24 h of ICU admission. Subsequently, a new prediction study is needed to determine which of the new risk factors improves the performance of the models and should be used for model revision [41]. It is important to realize that when a model is used to predict a patient’s risk of an event, it should always be considered an approximation no matter how strong the documented predictive accuracy. This is particularly important in the case of medical decision making. Two well-validated delirium screening-instruments (i.e. the CAM-ICU and the ICDSC) were used in this study. Naturally, the sensitivity and specificity of each instrument differs [42]. Realizing that the sensitivity of either screening tool is not 100%, we also defined delirium to be present when haloperidol or an atypical antipsychotic was administered for the treatment of delirium. While each ICU had similar delirium treatment protocols, we cannot exclude that antipsychotic therapy may have been initiated in patients who did not have delirium.

It is shown that ICU clinicians’ predictions are less accurate than those of an ICU delirium prediction model [9]. We believe that routine prognostic delirium evaluation in the ICU is important in the clinical setting to identify those patients who may benefit the most from early preventive measures and in the research setting to ensure that delirium risk is well-characterized and stratified in controlled studies. We want to emphasize that the predicted risk score is an estimation of the chance of developing delirium during ICU admission that may facilitate early clinical decisions on delirium prevention and personalized care to ICU patients and their family members. However, the rationing of critical care resources should not be based on the predicted risk score for delirium over the first 24 h of ICU admission alone as the predicted risk score does not take into account changes in the health status of ICU patients. For future controlled studies on the effect of both non-pharmacological and pharmacological interventions on ICU delirium, we suggest stratifying patients based on their risk of delirium and restricting delirium prevention to those patients at high risk of delirium.

To achieve the best predictive performance and user convenience currently possible, we suggest a two-stage calculation using the E-PRE-DELIRIC model in all patients admitted to the ICU to predict patients’ risk of delirium immediately after ICU admission and to update the risk scores of the patients at low risk of delirium after 24 h using the PRE-DELIRIC model. This way, the chance of missing a patient that will develop delirium during ICU admission is further attenuated. Still, a substantial minority of the patients that develop delirium during ICU admission will score a low predicted risk for delirium using both delirium prediction models in a two-stage calculation. Of interest, in this study cohort the delirium incidence was 21%. Based on the fact that the sensitivity and specificity of a prediction model are most optimal at the cutoff of the incidence level of the outcome of interest, in this case delirium, we expect that the sensitivity in a population with around 30% incidence of delirium will be better compared to the sensitivity shown in our study. Of importance, the acceptability of the suggested two-stage calculation by ICU physicians is not yet assessed. Future research should focus on the usefulness of both delirium prediction models in clinical practice and their impact on clinical outcomes, since this is the only way to determine whether their use improves usual care [41]. In addition, such an impact analysis also provides the opportunity to study the acceptance of the models in daily practice [41], in which it is interesting to also take the experiences of ICU patients and their families into account.

This study shows that statistically both ICU delirium prediction models have moderate-to-good performance. Although the predictive accuracy of the PRE-DELIRIC is greater, the E-PRE-DELIRIC model scores significantly better on user convenience. Moreover, the PRE-DELIRIC model needs data obtained during a period of 24 h, while the E-PRE-DELIRIC can be obtained at ICU admission, allowing direct preventive measures and stratified randomization in studies at the time of ICU admission. In patients who appear to be at low risk of delirium at ICU admission, it is of additional advantage to update their predicted risk scores using the PRE-DELIRIC model after 24 h in the ICU.