Machine learning approach for the prediction of 30-day mortality in patients with sepsis-associated encephalopathy

This retrospective study was conducted on the Medical Information Mart for Intensive Care (MIMIC)-IV version 1.0 [14]. Specifically, the MIMIC-IV database contained comprehensive, de-identified data of patients who have been admitted to the ICUs at the Beth Israel Deaconess Medical Center in Boston, Massachusetts, between 2008 and 2019, containing data from 383,220 admissions (single center). One author (CP) has obtained access to both databases and was responsible for data extraction (Certification number: 41657645). This study was approved by the Institutional Review Boards of Beth Israel Deaconess Medical Center (Boston, MA). Requirement for individual patient consent was waived due to the fact that all protected health information was deidentified.

Inclusion criteria were patients with a diagnosis of sepsis in accordance with the Third International Consensus Definitions for Sepsis (Sepsis-3) [15]. People with an age of younger than 16 years old, ICU stays less than 48 h, primary brain injury (traumatic brain injury, ischemic stroke, hemorrhagic stroke, epilepsy, or intracranial infection), pre-existing liver or kidney failure affecting consciousness, chronic alcohol or drug abuse, and severe electrolyte imbalances were excluded from the study. In addition, for patients with multiple ICU admissions, only data of the first ICU admission of the first hospitalization were included in the analysis.

In this study, the data extracted from MIMIC-IV included age, gender, race, and coexisting disorders. Hereafter, the Charlson comorbidity index (CCI) was calculated from its component variables [myocardial infarction, congestive heart failure, peripheral vascular disease, cerebrovascular disease, dementia, chronic pulmonary disease, rheumatic disease, peptic ulcer disease, diabetes, paraplegia, renal disease, malignant cancer, severe liver disease, metastatic solid tumor and acquired immunodeficiency syndrome (AIDS)]. Additionally, we retrospectively extracted the following data: vital signs, laboratory findings, injury types, different therapy strategies and scoring systems on the first day of ICU admission. Since values were missing at random, we used multiple imputation to deal with missing data. Details of missing data are shown in Supplementary Table 1 (Table S1).

Values were presented as means with standard deviations (if normal) or medians with interquartile ranges (IQR) (if non-normal) for continuous variables, and total numbers (%) for categorical variables. Continuous variables were compared by the t test or Wilcoxon rank sum test while proportions were compared using χ² test or Fisher exact tests, if appropriate.

Recursive feature elimination (RFE), a resource selection method, was utilized to select the most relevant variables. In a word, RFE recursively fits into a model based on smaller resource sets until a specified termination criterion is reached. In each loop, characteristics are classified in accordance with their importance in the trained model. Ultimately, highly correlated and collinear variables were eliminated. The characteristics were then considered in groups of 15/25/35/45/ALL (ALL = 56 variables, as represented in Fig. 1) organized by the ranks obtained after the method of selection of the characteristics. In order to find the optimal hyperparameters, fivefold cross-validation was used as the resampling method. In each iteration, every four folds were used as a training subset, and the remaining one-fold was processed to adjust the hyperparameters. This training-test process was repeated thirty times. Thus, each sample would be involved in both the training model and the testing model, so that all data were used as much as possible.

In this study, we employed nine different ML algorithms to develop models, including artificial neural network (NNET), bayes naive (NB), logistic regression (LR), gradient boosting machine (GBM), adaptating boosting (Ada), random forest (RF), bagged trees (BT), eXtreme Gradient Boosting (XGB) and CatBoost. Firstly, the population was divided into development set and validation set. As for internal validation, bootstrap resampling technique with 100 iterations was employed. Median and 95% confidence intervals of area under the curve (AUC) were calculated. Other evaluation indicators, such as, accuracy, sensitivity, specificity, negative predictive value and positive predictive value were also calculated. Moreover, the calibration curve was employed by the Hosmer–Lemeshow test of good adaptation. More precisely, the chi-square value was calculated based on the actual observed and predicted value of the model for each group and, subsequently, the corresponding p value was obtained. Ultimately, the “Shiny” package in R was used to build a visual data analysis platform. All analyses were performed by the statistical software packages R version 4.0.2 (http://​www.​R-project.​org, The R Foundation). In our study, we also used the “Caret” R packages and “Shiny” R packages to achieve the process. P values less than 0.05 (two-sided test) were considered as statistically significant.

In accordance with the inclusion and exclusion criteria, 6994 patients were finally included in the dataset. The process of data extraction, training preparation, data testing by diverse ML algorithms is demonstrated in Fig. 1. The characteristics of the participants are depicted in Table 1. People who died were more likely to be older, with more comorbidities (myocardial infarction, congestive heart failure, chronic pulmonary disease, rheumatic disease, mild liver disease, renal disease, malignant cancer, severe liver disease, metastatic solid tumor), higher heart rate, higher respiratory rate, higher white blood cell (WBC), higher mean corpuscular volume (MCV), higher red blood cell volume distribution width (RDW), longer activated partial thromboplastin time (APTT), longer prothrombin time (PT), higher international normalized ratio (INR), higher lactate, higher buffer excess (BE), higher anion gap, higher potassium, higher creatinine (CRE), higher blood urea nitrogen (BUN), higher vasopressor, higher sepsis related organ failure assessment (SOFA), higher acute physiology score III (APSIII), and higher systemic inflammatory response syndrome (SIRS). Furthermore, they were more likely to have lower temperature, lower mean artery pressure (MAP), lower red blood cell (RBC), lower mean corpuscular hemoglobin concentration (MCHC), lower platelet (PLT), lower hematocrit (HCT), lower pH, lower bicarbonate, lower PaO₂, lower chloride, lower sodium, and lower Glasgow coma score (GCS).

Based on the threshold measure of importance, a total of 15 important predictors were selected by the RFE algorithm. (Fig. 2) These variables included APSIII, GCS, SOFA, CCI, RDW, BUN, age, respiratory rate, PaO₂, temperature, lactate, CRE, malignant cancer, metastatic solid tumor, and PLT. Then, these 15 variables were used in all the subsequent analysis for all models in both training and testing sets.

The discriminatory abilities of all models for the prediction of 30-day mortality in SAE patients are shown in Fig. 3 and Table 2. Within the training set, the NNET, NB, LR, GBM, Ada, RF, BT, XGB, and CatBoost models were established, and the testing set obtained AUCs of 0.833, 0.816, 0.833, 0.824, 0.834, 0.825, 0.804, 0.830, and 0.830, respectively. Comparatively, the BT had the lowest discriminative ability (AUC 0.804, 95% CI 0.786 to 0.820) while the other eight models had a relatively higher discriminative ability (Table S2). In terms of the calibration performance, the NNET, LR, Ada models had a good calibration—namely, a high accuracy of prediction, with P-values of 0.831, 0.119, and 0.129, respectively (Fig. 4).

In the Fig. 5, fifth predictor variables in the ML are demonstrated. Each variable incorporated in the study had varying importance over SAE depending on the ML approach. In general, APSIII was the variable with greatest importance across all ML algorithms, followed by GCS, RDW, and so forth.

The Shiny package analyzed the entire training set, demonstrating the impact of each variable on predicting SAE (Fig. 6). For example, the information of one patient was input into the model: no metastatic solid tumor, no malignant cancer, APSIII (121), GCS (3), CCI (6), SOFA (16), age (92), temperature (32 ℃), respiratory rate (19 per/min), RDW (17.5%), PLT (158 × 10⁹/L), lactate (4.6 mmol/L), BUN (20 mg/dL), CRE (1.1 mg/dL), PaO₂ (85 mmHg). The model analyzed that the risk of in-hospital mortality in this patient was 84.20%, indicating that the 30-day mortality for this SAE patient was relatively high, and precaution measures were recommended. In order to better apply this model, we also made a web-based calculator (https://​pengchi2009.​shinyapps.​io/​Mortality_​of_​sepsis_​associated_​encephalopathy/​).