SEA-MAKE score as a tool for predicting major adverse kidney events in critically ill patients with acute kidney injury: results from the SEA-AKI study

This was a prospective multicenter observational cohort study or also known as the Southeast Asia AKI cohort study that is conducted in 17 ICUs at 16 hospitals from various regions across Thailand. The study sites were composed of university, regional, and provincial hospitals. All patients aged older than 15 years who were admitted to the participating ICUs from February 2013 to July 2015 were enrolled into the study. Patients with ESRD on chronic dialysis were excluded. Only data from the first admission were used if the patients have been admitted to the hospital several times during the study period. Formal sample size was not calculated because it was fixed to the Southeast Asia AKI cohort [10]. All available data were used to maximize the power and generalizability of the proposed model.

The study protocol was reviewed by the Institutional Review Board at each participating site and the informed consent was waived.

All data were collected in a web-based platform after the patient was registered online at the study sites. Demographic, clinical and laboratory data were collected. The following demographic data were collected: age, body mass index, sex, co-morbidity diseases (hypertension, diabetes, coronary artery, cerebrovascular disease, malignancy, and chronic kidney disease), timing of hospital and ICU admission, and the patient’s principle diagnosis at ICU admission. Clinical data at ICU admission were collected: the patient’s vital signs, fluid balance status, whether or not the patient used vasopressor or mechanical ventilation, renal replacement therapy, Acute Physiology and Chronic Health Evaluation (APACHE) II score, and Sequential Organ Failure Assessment (SOFA) scores for the first 3 days after being admitted to the hospital. Laboratory data such as complete blood count, serum creatinine, and blood urea nitrogen levels, if available, were obtained. The data were collected every day for the first 7 days and then weekly until day 28. However, only parameters from the first day were used as predictors to create the model.

The primary outcome was patients who met one or more criteria for Major Adverse Kidney Events within 28 days (MAKE28): death, provision of dialysis, and/or sustained loss of the kidney function. Hospital death was defined as death from any cause before hospital discharge within 28 days after ICU admission. Provision of dialysis was defined as receipt of any modality of renal replacement therapy before hospital discharge within 28 days after ICU admission. Sustained loss of kidney function was defined as a final serum creatinine value before hospital discharge within 28 days after ICU admission that was 200% or more than the baseline creatinine value.

AKI was diagnosed by KDIGO criteria [3]. For the baseline serum creatinine, the most recent available value before hospital admission, within 1 year, was used. If the true baseline creatinine level was not available and the patients had no previous history of CKD, we estimated the baseline serum creatinine using the lowest value between the serum creatinine value at the time of hospital admission and the back calculation of serum creatinine from the Modification of Diet in Renal Disease (MDRD) equation using a glomerular filtration rate of 75 mL/min/1.73 m² [11] as recommended in the KDIGO guideline [3]. For the urine output criteria, it was modified to using 24-h urine output. The patients who had urine output > 0.5 mL/kg/h were defined as no AKI, 0.3–0.5 mL/kg/h were defined as KDIGO stage 2, and < 0.3 mL/kg/h were defined as KDIGO stage 3 [11]. MAKE was defined as a composite endpoint of death, provision of dialysis, and sustained loss of kidney function [4].

The International Classification of Disease, 10th Revision coding was used to classify the diagnosis of ICU admission. The primary diagnosis for ICU admission and etiologies of AKI were determined by the study team. The definition of “renal-related disease” was AKI, glomerular disease, electrolyte imbalance related to renal disorders and/or stones in the urinary tract system. Hypertension was defined according to the seventh Report of the Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure (JNC 7) [12]. Chronic kidney disease was diagnosed according to KDIGO 2012 Clinical Practice Guideline for the Evaluation and Management of Chronic Kidney Disease [13].

The actual body weight value before hospital admission was used to calculate the rate of urine flow, if unavailable, the ideal body weight was calculated from the following formula: height (cm)—100 for male and height (cm)—110 for female. Body mass index (BMI) class was defined using the World Health Organization Expert Consultation criteria: optimal BMI was 18.5–22.9 kg/m², overweight had BMI > 23 kg/m², and obesity had BMI > 27.5 kg/m² [14]. Almost all of the patients in this study were Asian.

Categorical data were presented as counts and percentages. Continuous data were presented as mean and standard deviation (SD), if normally distributed, or median with interquartile range, if non-normally -distributed. Comparison between MAKE and non-MAKE groups was performed using a Pearson’s Chi Square asymptotic test and a Fisher’s exact test for categorical variables. Meanwhile, a Student’s t test (if data were normally distributed) or Kruskal–Wallis one-way analysis of variance by ranks (if data were non-normally distributed) was performed for continuous variables. The normality was assessed by graph visualization, and the similarity between the mean and the median was compared. Statistical analyses were done using IBM SPSS Statistics version 21. P value < 0.05 was considered to be statistically significant.

In the development cohort, all potential clinical factors were initially considered. Variables that were statistically significant in the univariate logistic regression method were inputted into the multivariate logistic regression method. In the multivariable analysis models, each clinical factor was adjusted for age, sex, body mass index, and co-morbidities such as hypertension, diabetes, coronary artery disease, cerebrovascular disease, malignancy, and chronic kidney disease to examine the potential predictability. Variables that were statistically significant in the multivariate logistic regression method were selected as predictors of MAKE. Then, significant predictors were included in the final analysis. Candidate variables with more than 25% missing data were eventually excluded. Data that were not missing completely at random, multiple imputations (n = 10) for the missing candidate variables were performed in the final model. Multivariable binary logistic regression analysis with adjustment for co-variates was operated under stepwise Forward Wald selection method. Parameter estimate values were obtained from pooled data of 10 imputations. The full model was converted to a simplified score points using Schneeweiss’s scoring system [15]. The variance inflation factors (VIFs) and tolerance coefficients were computed to test multicollinearity among the co-variates. Value of VIF exceeding 10 are often regarded as indicating multicollinearity, but in weaker models, which is often the case in logistic regression, value above 2.5 may be a cause for concern [16]. A tolerance value below 0.1 indicated that there was a serious collinearity problem and if the value was less than 0.2, this indicated that there was a potential collinearity problem [17, 18].

Internal validation was performed using bootstrapping method [19]. A total of 2000 bootstrap samples were drawn, and the final model containing all included variables was fitted in each bootstrap sample. This was followed by re-simplification of the score and point assignment using the Schneeweiss’s scoring system [15]. The apparent performance of the score was evaluated using the following performance measures: Nagelkerke’s R² (overall performance) [20], area under the receiver operating characteristic curve (AUC) (able to differentiate between MAKE and non-MAKE patients), and a calibration plot [19, 21]. If AUC = 1, then this indicated that there was a perfect discrimination. If the AUC = 0.5, then the discrimination was no better than chance. Calibration intercept α and slope β from the linear regression of the observed outcomes (dependent variable) versus the predicted risks (independent variable) were acquired to show that the predictions were on average, close to the average observed outcome. In the calibration curve analysis, the intercept α = 0 and slope β = 1 implied that the models were well-calibrated.

We also tried to compare the discrimination (AUC) of the SEA-MAKE score to APACHE II, SOFA day 1, and SOFA-Kidney day 1 in this internal validation analysis even though we relegalized that the limitation of using internal validation might overestimate the performance of SEA-MAKE score over the other score and the comparison should be operated under a true external validation circumstance.

In the development cohort, univariate analysis for comparisons of clinical and laboratory characteristics between MAKE and non-MAKE at the time of enrollment is shown in Table 1. Many parameters such as diabetes, CKD, Glasgow Coma scale, body temperature, heart rate, respiratory rate, mean arterial pressure, hematocrit, platelet count, serum sodium, serum potassium, blood urea nitrogen, serum creatinine, total bilirubin, PaO₂/FiO₂ ratio, arterial pH, urine output, net fluid balance, mechanical ventilation, and vasopressor use were found to be significantly different between MAKE and non-MAKE patients. However, age, body mass index, sex, hypertension, coronary artery disease, cerebrovascular disease, malignancy and chronic health points were not found to be significantly different between the two groups.

In the logistic regression analysis, adjusted for age, sex, body mass index and co-morbidity (hypertension, diabetes coronary artery disease, cerebrovascular disease, malignancy and chronic kidney disease), it was determined that MAKE had a significantly lower Glasgow coma scale [OR (95% CI) 4.27 (3.48, 5.23)], lower mean arterial pressure [OR (95% CI) 2.25 (1.73, 2.93)], lower hematocrit [OR (95% CI) 2.27 (1.91, 2.69)], lower urine output [OR (95% CI) 1.78 (1.52, 2.10)], and lower platelet count [OR (95% CI) 2.22 (1.87, 2.65)]. Conversely, it was determined that MAKE had a significantly higher body temperature [OR (95% CI) 1.74 (1.30, 2.32)], higher heart rate [OR (95% CI) 0.78 (1.51, 2.08)], higher respiratory rate [OR (95% CI) 2.29 (1.94, 2.70)], higher positive fluid balance [OR (95% CI) 2.34 (1.98, 2.76)], higher blood urea nitrogen [OR (95% CI) 3.97 (3.30, 4.78)], and especially higher proportion of serum creatinine rising ≥ 3 times from the baseline level [OR (95% CI) 10.18 (7.34, 14.12)] (Fig. 2).

In creating the SEA-MAKE score using multivariable regression-based method, after multiple imputations were performed for the missing values, the candidate predictor variables were included in the analysis under the Forward Wald selection with adjustment for the co-variates. Nine predictors were significant in the final model: low Glasgow coma scale [OR (95% CI) 2.72 (2.17, 3.40)], tachypnea [OR (95% CI) 1.55 (1.29, 1.86)], vasopressor use [OR (95% CI) 1.56 (1.30, 1.88)], on mechanical ventilation [OR (95% CI) 1.81 (1.45, 2.27)], oliguria [OR (95% CI) 1.78 (1.48, 2.14)], serum creatinine rising ≥ 3 times [OR (95% CI) 4.77 (3.38, 6.73)], high blood urea nitrogen [OR (95% CI) 2.33 (1.89, 2.86)], low hematocrit [OR (95% CI) 1.81 (1.50, 2.19)], and thrombocytopenia [OR (95% CI) 1.40 (1.15, 1.70)] (Table 2). The simplified score points (Schneeweiss’s scoring system) of low Glasgow coma scale was 3, tachypnea was 1, vasopressor use was 1, on mechanical ventilation was 2, oliguria was 2, serum creatinine rising ≥ 3 times was 5, high blood urea nitrogen was 3, low hematocrit was 2, and thrombocytopenia was 1. All of the variance inflation factors (VIFs) were less than 1.3 and the tolerance coefficients exceeded 0.79, meaning that there was no potential multicollinearity present among the co-variates.

The overall performance of the score (Nagelkerke’s R²) was 0.329. Discrimination (AUC) of apparent performance was 0.797 (95% CI 0.780–0.813), bias-corrected bootstrap performance was 0.799 (95% CI 0.780–0.813) (Additional file 1: Figure S1), and optimism-corrected performance was 0.795. When we compared the discrimination of SEA-MAKE score to SOFA day 1, APACHE II, and SOFA kidney day 1, the AUC (95% CI) of those scores were 0.75 (0.73, 0.77), 0.74 (0.72, 0.75) and 0.70 (0.68, 0.72), respectively (Fig. 3a). We also found that SEA-MAKE score had higher discriminatory power when applied to patients with AKI on the first day (n = 2154) compared to after the first day (n = 702) with AUC (95%) of 0.82 (0.80, 0.84) and 0.73 (0.70, 0.77), respectively (Fig. 3b). Based on the simplified model, SEA-MAKE score can be calculated by adding the simplified score point of each predictor. When the cut-off value was 7 for AKI in the first day group (AKI D1), the sensitivity was 0.75, specificity was 0.76, positive likelihood ratio was 3.10, and the positive predictive value was 0.84. When the cut-off value was 5 in AKI after the first day group (AKI after D1), the sensitivity was 0.71, specificity was 0.65, positive likelihood ratio was 2.03, and the positive predictive value was 0.71 (Additional file 1: Table S2).

In the calibration curve analyses, the X axis represented the predicted risk for MAKE, the Y axis represented the observed risk for MAKE, the blue straight line represented a perfect fit, and the red dots represented the average MAKE of bias-corrected bootstrap samples in each bins. The calibration intercept α = 0 and calibration slope β = 1.001 are shown in Additional file 1: Figure S2.