Machine learning for the prediction of volume responsiveness in patients with oliguric acute kidney injury in critical care

A large US-based critical care database named Medical Information Mart for Intensive Care (MIMIC-III) was analyzed [12]. The MIMIC-III database is an integrated, de-identified, comprehensive clinical dataset containing all the patients admitted to the ICUs of Beth Israel Deaconess Medical Center in Boston, MA, from June 1, 2001, to October 31, 2012. There were 53,423 distinct hospital admissions for adult patients (aged 16 years or above) admitted to the ICUs during the study period. Since the study was an analysis of a third-party anonymized publicly available database with pre-existing institutional review board (IRB) approval, IRB approval from our institution was exempted. The study was reported according to the recommendations of the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement [13].

Patient eligibility was considered when urine output was less than 0.5 ml/kg/h for the first 6 h after ICU admission. This definition was consistent with the urine output component of the KDIGO criteria [4]. To examine the impact of fluid resuscitation on subsequent urine output response, only patients with substantial fluid intake (> 5 l) within 6 h following the initial 6 h for assessing oliguria were eligible (Fig. 1). Fluid intake and urine output were extracted from the nursing chart system, which more accurately reflected the actual volume intake or output than the medical order system. Patients who left ICU during the observation period were excluded. Furthermore, patients receiving any diuretics and/or renal replacement therapy (RRT) on day 1 were excluded.

The urine output within an 18-h period following the initial 6 h for defining oliguria was used as the outcome. Patients were considered as VR-AKI if he/she had urine output greater than 0.65 ml/kg/h, corresponding to a 30% increase as compared with the baseline value. Otherwise, they were defined as VU-AKI.

Routinely collected clinical and laboratory variables obtained within the first 6 h of ICU admission were assessed for their ability to predict volume responsiveness. For some variables with multiple measurements, both the maximum and minimum values were assessed. Age, gender, ethnicity, admission type, elective surgery, type of ICU and presence of infection, and vital signs including respiratory rate, blood pressure, heart rate, and temperature were analyzed. In addition, laboratory data including glucose, white blood cell count (WBC), hematocrit, chloride, potassium, sodium, lactate, creatinine, blood urea nitrogen (BUN), coagulation profile, PaO2, PaCO2, and pH were included.

Because this was a hypothesis-generating epidemiological study, no attempt was made to estimate the sample size of the study; instead, all eligible patients in the database were included to maximize the statistical power of the predictive model. Because missing data may create bias, variables with > 70% missing values were excluded from further analysis. Other variables with a lesser degree of missing values were analyzed using multiple imputation method [14].

Clinical characteristics between VR-AKI and VU-AKI groups were compared using either Student t test or rank-sum test as appropriate. Chi-square test or Fisher’s exact test was employed to compare the differences of the categorical variables [15]. A stepwise logistic regression model was used to select variables which were predictive of volume responsiveness. Both forward selection and backward elimination were used, testing at each step for variables to be included or excluded. Akaike Information Criterion (AIC) was used as the selection criteria to eliminate the predictors [16].

Extreme gradient boosting (XGBoost) combined with decision trees was employed to predict VR versus VU. A classification tree was used as the weak learner, and the learning objective function was binary logistic. The boosting method works by iteratively refitting a weak classifier (decision tree) to residuals of previous models. Each successive classifier focused more on misclassified observations during the previous round of fitting [17]. In this study, we employed 300 rounds of iterations for cross-validation process, which were expected to result in a powerful ensemble classifier with superior predictive accuracy. Overfitting can be a major problem in using machine learning techniques. The ability to understand the complex relationship in data while avoiding overfitting requires fine-tuned hyperparameters. XGBoost hyperparameters included learning rate, minimum loss reduction required to make a further partition on a leaf node of the tree, maximum depth of a tree, subsample ratio of the training instance. The original dataset was randomly partitioned into 5 equal-sized subsamples for bootstrap validation (BV). Specifically, 4 subsamples were used to train the model, which was then validated in the remaining 1 subsample. Hyperparameters were considered to be sufficiently tuned if (1) the BV training log-loss decreased as the number of trees increased and (2) BV testing log-loss was less than 0.693 and only slightly greater than the training log-loss (e.g., a log-loss of 0.693 is the performance of a binary classifier that performs no better than chance: − log 0.5 ≈ 0.693). We used a loop function (grid search) to select the hyperparameters that the minimum training log-loss should be greater than the 85th percentile, and the minimum testing log-loss should be less than the 8th percentile. After choosing the hyperparameters, the BV process was run for 100 times to determine the number of trees required for the final model. The number of trees in the final XGBoost model was determined by the minimum BV testing log-loss in each of the last 100 BV iterations and computed the 5th percentile of that distribution. This was a conservative approach to the selection of the number of trees, which was also described in the literature [18].

Of the 10,795 patients with urine output < 0.5 ml/kg/h for the first 6 h after ICU admission, 7491 patients (69.4%) received fluid intake > 5 l within the following 6 h. A number of 809 patients were excluded because they received diuretics and/or RRT on the first day. A total of 6682 patients were included in our analysis; 2456 patients had VR-AKI, and 4226 patients had VU-AKI on day 1 in ICU (Fig. 2).

The differences in characteristics between VR and VU groups are described in Table 1. VR group had more patients of elective surgery prior to ICU admission than the VU group (18.2% vs. 13.6%; p < 0.001). The maximum serum creatinine concentration (1.74 ± 1.50 vs. 1.26 ± 0.98 μmol/l; p < 0.001) was higher, and the minimum bicarbonate concentration (20.82 ± 5.41 vs. 22.24 ± 4.32 mmol/l; p < 0.001) and maximum hematocrit (35.14 ± 5.69 vs. 35.93 ± 5.79%; p < 0.001) were lower in the VU group. The VR group had higher urinary pH (5.78 ± 0.86 vs. 5.66 ± 0.79; p < 0.001), lower urinary creatinine (111.61 ± 73.01 vs. 132.51 ± 79.89 mg/dl; p < 0.001), higher systolic blood pressure (89.16 ± 16.46 vs. 85.09 ± 18.03 mmHg; p < 0.001), higher albumin (3.03 ± 0.63 vs. 2.83 ± 0.66; p < 0.001), lower rate of mechanical ventilation (31.3% vs. 42.7%; p < 0.001), vasopressor use (22.8% vs. 30.0%; p < 0.001), and infection (47.6% vs. 62.4%; p < 0.001) than the VU group (Table 1).

The results of stepwise logistic regression model are shown in Table 2. As expected, advanced age (odds ratio [OR] for a 20-year increase, 0.69; 95% confidence interval [CI], 0.63 to 0.75), a higher serum creatinine (OR for each 0.1 mg/dl increase, 0.98; 95% CI, 0.97 to 0.99), and lactate (OR for each 1-mmol/l increase, 0.91; 95% CI, 0.88 to 0.94) were associated with decreased probability of volume responsiveness. On the contrary, a greater value of albumin (OR for each 1-g/dl increase, 1.53; 95% CI, 1.38 to 1.71), systolic BP (OR for each 20-mmHg increase, 1.13; 95% CI, 1.04 to 1.23), and minimum bicarbonate (OR for each 5-mmol/l increase, 1.12; 95% CI, 1.02 to 1.22) were associated with increased probability of volume responsiveness (Table 2).

The hyperparameters used in our analysis were as follows (determine by grid search): learning rate = 0.04, minimum loss reduction = 10, maximum tree depth = 9, subsample = 0.6, and number of trees = 300. With these hyperparameters, bootstrap validation (BV) training log-loss decreases as the number of trees in an ensemble increases, and the BV testing log-loss was less than 0.693 and only slightly more than BV training log-loss as the tree grows (Fig. 3). Feature importance was calculated by the sum of the decrease in error when split by a variable, which reflects the contribution each variable makes in classifying VR versus VU. The urinary creatinine was the most important variable to distinguish VR and VU group, followed by maximum BUN, age, albumin, and maximum temperature (Fig. 4).

Model discrimination was assessed using the area under receiver operating characteristic curve (AU-ROC). The XGBoost had a significantly greater AU-ROC than the logistic regression model (AU-ROC, 0.860; 95% CI, 0.842 to 0.878 vs. 0.728; 95% CI, 0.703 to 0.753, respectively; Fig. 5). Table 3 describes the classification or confusion matrix for the two models in identifying the VR and VU status.