Background
Methods
Literature search and eligibility criteria
Data extraction and critical appraisal
Descriptive analyses
Results
Study characteristics
Quality assessment
Study (country) | Scoring systems | Sample size (age in years) | (Outcome) and model performance measures | Study (country) | Scoring system | Sample size (age in years) | (Outcome) and model performance measures |
---|---|---|---|---|---|---|---|
Abhinandan and Vedavathi, 2013 (India) [60] | APACHE II | 50 (18–90, mean 48.36) | (Unspecified mortality) D: not reported; C: not reported; CM: not reported; comparison of means | Liu et al., 2016 (China) [61] | APACHE II | 137 (mean 69.53 ± 9.28) | (Hospital mortality) D: AUROC; C: not reported; CM: sensitivity, specificity |
Aggarwal et al., 2006 (India) [7] | APACHE II SAPS II MPM II 0 MPM II 24 | 459 (16–80) | (Hospital mortality) D: AUROC; C: HL C and H statistics; CM: sensitivity, specificity, NPV, PPV, accuracy | Mohan et al., 2015 (India) [46] | APACHE II SAPS II SAPS 3 | 100 (mean: survivors 44.29 ± 15.53, non-survivors 57.37 ± 20.42) | (ICU mortality) D: not reported; C: not reported; CM: sensitivity, specificity; comparison of means |
Ahluwalia et al., 1999 (India) [29] | APACHE II 48 hours (TA) New Scorea | 79 (13–80, mean 46) | (Hospital mortality) D: AUROC (CI not reported); C: not reported; CM: sensitivity, specificity | Nair et al., 2016 (India) [62] | SAPS | 48 (mean: survivors 53.56, non-survivors 47.78) | (ICU mortality) D: AUROC; C: not reported |
Celik et al., 2014 (Turkey) [63] | SAPS II | 545 (>18) | (ICU mortality) D: not reported; C: not reported; CM: not reported; comparison of means | Naqvi et al., 2016 (Pakistan) [33] | APACHE II SAPS | 96 (mean 32.93 ± 16.61) | (ICU mortality) D: AUROC; C: HL |
Chang et al., 2006 (Taiwan) [64] | APACHE II | 1263 (13–92, mean 56) | (Hospital mortality) D: AUROC; C: HL | Nassar et al., 2012 (Brazil) [42] | APACHE IV SAPS 3 MPM III | 5780 (median 66, IQR 47–79) | (Hospital mortality) D: AUROC; C: HL C statistic |
Chiavone and Rasslan, 2005 (Brazil) [65] | APACHE II | 94 (16–97) | (Hospital mortality) D: AUROC; C: calibration curve, stratified in 10% risk bands, R2 | Naved et al., 2011 (Pakistan) [34] | APACHE II (48 hours) | 253 (15–84, mean 51.26 ± 17.9) | (ICU mortality) D: not reported; C: chi-square test |
Eapen et al., 1997 (India) [30] | APACHE II (VA) | 500 (13–84) | (Unspecified mortality) D: AUROC (CI not reported); C: not reported | Nimgaonkar et al., 2004 (India) [35] | APACHE II ANN22a ANN15a | 2962 (mean 37.5 ± 16.1) | (Hospital mortality) D: AUROC; C: HL |
Evran et al., 2016 (Turkey) [47] | APACHE II APACHE IV SAPS 3 | 487 (18–96, mean 58.58 ± 18 | (ICU mortality) D: AUROC (as figure); C: not reported; CM: sensitivity, specificity, accuracy, PPV and NPV | Nouira et al., 1998 (Tunisia) [66] | APACHE II SAPS II MPM 0 MPM 24 | 1325 (mean 46.6 ± 20.4) | (Hospital mortality) D: AUROC; C: HL C and H statistics; CM: NPV, PPV |
Fadaizadeh et al., 2012 (Iran) [67] | APACHE II SAPS II | 415 (mean 49.28 ± 0.94) | (ICU mortality) D: AUROC; C: HL C statistic; CM: sensitivity, specificity | Ratanarat et al., 2005 (Thailand) [68] | APACHE II SAPS I | 482 (mean 56 ± 21) | (ICU mortality) D: AUROC; C: not reported CM: sensitivity, specificity |
Faruq et al., 2013 (Bangladesh) [37] | APACHE II SAPS II | 194 (mean 61.06 ± 15.42 | (ICU mortality) D: AUROC; C: HL C statistic | Riviello et al., 2016 (Rwanda) [43] | MPM III (0) R-MPMa Simplified R-MPMa | 427 (median 34, IQR 25–47) | (Hospital mortality) D: AUROC; C: HL |
Galal et al., 2013 (Egypt) [49] | APACHE II SAPS II | 105 (0–88, mean 54.59 ± 15.75) | (ICU mortality) D: AUROC (CI not reported); C: HL; CM: sensitivity, specificity, accuracy | Sathe and Bapat, 2014 (India) [69] | APACHE II SAPS II | 1543 (mean 53.9 ± 16.8) | (Hospital mortality) D: AUROC; C: HL |
Gilani et al., 2014 (Iran) [70] | APACHE II APACHE III SAPS II | 202 (14–85 mean 53.1 ± 20.3) | (Hospital mortality) D: AUROC (CI not reported); C: HL; CM: sensitivity, specificity, accuracy, NPV, PPV | Sekulic et al., 2015 (Serbia) [48] | APACHE II MPM II (0) MPM II (24) MPM II (48) MPM II (72) MPM II (7 days) SAPS II | 60 (mean 59.0 ± 15.8) | (ICU mortality) D: AUROC (as figure for MPM II (24)); C: HL; CM: sensitivity, specificity |
Godinjak et al., 2016 (Bosnia and Herzegovina) [31] | APACHE II SAPS II | 174 (19–87, mean 61.7 ± 16.3) | (Hospital mortality) D: AUROC; C: not reported; CM: sensitivity, specificity | Shoukat et al., 2016 (Pakistan) [71] | APACHE IV | 155 (13–70, mean 38.39 ± 13.61) | (Unspecified mortality) D: not reported; C: not reported; CM: not reported; comparison of means |
Gupta and Arora, 2004 (India) [72] | APACHE II | 330 (mean 43.32 ± 16.22) | (ICU mortality) D: AUROC (CI not reported); C: HL; CM: accuracy | Shrestha et al., 2011 (Nepal) [73] | APACHE III | 117 (16–84, mean 43.18 ± 18.49) | (ICU mortality) D: AUROC; C: HL; CM: sensitivity, specificity, NPV, PPV, accuracy |
Haidri et al., 2011 (Pakistan) [74] | APACHE II | 142 (15–90, mean 53.16 ± 19.29) | (ICU mortality) D: not reported; C: not reported; comparison of means | Silva Junior et al., 2010 (Brazil) [55] | SAPS 3 | 1310 (mean 67.1 ± 15.3) | (Hospital mortality) D: AUROC; C: HL C statistic; CM: sensitivity, specificity |
Halim et al., 2009 (Indonesia) [75] | APACHE II | 144 (15–85, mean 47.33 ± 17.17) | (ICU mortality) D: AUROC (CI not reported); C: HL; CM: sensitivity, specificity, accuracy, NPV, PPV | Soares and Salluh, 2006 (Brazil) [39] | SAPS II SAPS 3 SAPS 3 (CSA) | 952 (>18, mean 58.3 ± 23.1) | (Hospital mortality) D: AUROC; C: HL C statistic |
Hamza et al., 2009 (Egypt) [76] | APACHE IV SAPS II MPM II0 | 265 (mean 57.07 ± 16.6) | (ICU mortality) D: AUROC; C: HL H statistic | Soares et al., 2004 (Brazil) [40] | APACHE II APACHE III SAPS II, MPM II 0 MPM II 24 | 1257 (18–93, mean 56.0 ± 16.7) | (Hospital mortality) D: AUROC; C: HL H statistic |
Hashmi et al., 2016 (Pakistan) [77] | APACHE II APACHE II (MTA) | 213 (mean 46.31 ± 18.43) | (Unspecified mortality) D: AUROC; C: HL; CM: sensitivity, specificity, accuracy | Soares et al., 2010 (Brazil) [41] | SAPS II SAPS 3 SAPS 3 (CSA) MPM III (24) | 717 (mean 61.2 ± 15.4) | (Hospital mortality) D: AUROC; C: HL C statistic |
Hernandez et al., 2014 (Philippines) [78] | SAPS 3 | 2426 (mean 62 ± 17) | (ICU mortality) D: AUROC (CI not reported); C: HL C statistic | Sutheechet, 2009 (Thailand) [79] | SAPS II MPM II (24) | 639 (18–100, mean 56.9 ± 20.6) | (Hospital mortality) D: AUROC; C: HL; CM: sensitivity, specificity, accuracy, NPV, PPV |
Hosseini and Ramazani, 2015 (Iran) [80] | APACHE II | 150 (3–97) | (ICU mortality) D: AUROC; C: HL; CM: sensitivity, specificity, accuracy | Teoh et al., 1991 (Malaysia) [81] | APACHE II | 100 (1 month–82 years) | (Hospital mortality) D: not reported; C: not reported; CM: not reported; correlation |
Juneja et al, 2012 (India) [1] | APACHE II APACHE III APACHE IV SAPS II SAPS 3 MPM II (0) MPM III (0) | 653 (mean 58.48 ± 18.6) | (ICU mortality) D: AUROC; C: HL; CM: sensitivity, specificity | Turner et al., 1989 (South Africa) [36] | APACHE II | 728 (12–88, mean 43) | (Unspecified mortality) D: not reported; C: not reported; CM: not reported; correlation |
Khan et al., 2015 (India) [32] | APACHE II APACHE II (48 hours) SAPS II SAPS II (48 hours) | 85 (mean 39.14 ± 17.3) | (Hospital mortality) D: AUROC; C: not reported; CM: sensitivity, specificity | Wilairatana et al., 1995 (Thailand) [82] | APACHE II APACHE III SAPS II | 209 (16–93, mean 55.36 ± 17.44) | (Hospital mortality) D: AUROC; C: not reported; CM: sensitivity, specificity, accuracy |
Khawannimit and Geater, 2007 (Thailand) [38] | APACHE II SAPS II | 1316 (mean 55.6 ± 18.2) | (Hospital mortality) D: AUROC; C: HL C and H statistics; CM: sensitivity, specificity, NPV, PPV, accuracy | Xing et al., 2015 (China) [52] | APACHE II APACHE IV SAPS 3 | 981 (mean 64.8 ± 12.1) | (Hospital mortality) D: AUROC; C: HL C statistic |
Khwannimit and Bhurayanontachai, 2011 (Thailand) [51] | APACHE II SAPS 3 SAPS 3 (AUS) SAPS II Customised APACHE II (recalibration) Customised SAPS II (recalibration) Customised SAPS 3 (recalibration) | 2022 (median 62, IQR 49–73) | (Hospital mortality) D: AUROC; C: HL H and C statistic | Yamin et al., 2011 (Pakistan) [28] | APACHE IV | 162 (9–90, mean 38.024) | (ICU mortality) D: not reported; C: not reported; CM: accuracy; comparison of means |
Kiatboonsri and Charoenpan, 1995 (Thailand) [45] | APACHE II | 334 (15–98, mean 56.01 ± 18.23) | (Hospital mortality) D: AUROC not reported; C: not reported; CM: sensitivity, specificity, PPV, accuracy | Zhao et al., 2013 (China) [50] | SAPS II (initial) SAPS II (48 hours) SAPS II (24 hours) SAPS II (72 hours) Simplified SAPS II (VA) | 1684 (18–98, mean 58.93 ± 18.30) | (ICU mortality) D: AUROC; C: HL; CM: sensitivity, specificity, accuracy; correlation |
Study | Scoring system/s | Missing value handling |
---|---|---|
Exclusion | ||
Celik et al. (2014) [63] | SAPS II | 178 (21.1%) were excluded due to lack of data, and 46 (5.55%) patients were excluded due to archival documentation problems. No information on admission source, attached devices, PaO2/FiO2 and Glasgow Coma Score was available for the excluded group in the computerised medical records. The included group survey did not differ from the excluded group regarding age, gender, admission time and admission day. The prevalence of trauma and intoxication was higher among the excluded group than the included group (trauma 15%, intoxication 30%) |
Chiavone et al. (2005) [65] | APACHE II | One patient excluded |
Godinjak et al. (2016) [31] | APACHE II SAPS II | 15 patients (7.9%) who died in the first 24 hours after admission to the MICU |
Haidri et al. (2011) [74] | APACHE II | All patients with incomplete records and missing variables including laboratory investigations or who were not followed up due to any reason were excluded |
Hernandez et al. (2014) [78] | SAPS 3 | 159 (6.6%) were excluded for incomplete SAPS 3 data |
Naqvi et al. (2016) [33] | APACHE II SAPS | 29 patients (23.6%) with incomplete information of scoring system in case records |
Naved et al. (2011) [34] | APACHE II | Patients with incomplete records not included (numbers not reported) |
Willairatna et al. (1995) [82] | APACHE II APACHE III SAPS II | When scores could not be derived due to an incomplete set of physiological data, patients were excluded |
Normal value imputation | ||
Faruq et al. (2013) [37] | APACHE II SAPS II | GCS attributed as normal |
Khwannimit and Geater (2007) [38] | APACHE II SAPS II | GCS attributed as normal Missing physiological variables were found in only 6% for APACHE II (excluding bilirubin, which was missing in 76.5% of the presented data records) and 6.3% for SAPS II variables (excluding bilirubin, which was missing in 76.5% of the presented data records) |
Soares et al. (2004) [40] | APACHE II APACHE III MPM II (0) MPM II (24) SAPS II | Zero points or normal values were inserted where data were missing [19]. There were no missing variables for physiological data. Among laboratory variables, normal values were inserted for albumin in 623 (49.6%), prothrombin time in 274 (21.8%) and bilirubin in 676 (53.8%) patients. No patient with jaundice on physical examination lacked serum bilirubin measurements |
Soares and Salluh (2006) [39] | SAPS II SAPS 3 SAPS 3 (CSA) | Zero points or normal values were assigned for missing variables [1, 12]. There were no missing data for demographic, clinical and physiologic data. Among laboratory variables, normal values were attributed only for bilirubin in 535 patients (56%). No patient with jaundice lacked bilirubin level measurements |
Soares et al. (2010) [41] | MPM III (24) SAPS II SAPS 3 SAPS 3 (CSA) | Zero points or normal values were attributed for missing variables. |
Exclusion and normal value imputation | ||
Nassar et al. (2012) [42] | APACHE IV SAPS 3 MPM III | 3.02% patients with incomplete data which prevented adequate calculation of one or more of the scores were excluded; these missing data could be pre-ICU length of stay, reason for ICU admission, chronic health variables and mechanical ventilation on first day. Missing physiologic variables, namely bilirubin, acid-base abnormalities, PaO2 or PaO2/FiO2 ratio, were considered as normal for purpose of calculations |
Model performance
Evaluation of the performance of existing models
Study | Type of adjustment and changes made | Discrimination (original scoring system) | Discrimination(after adjustment) | Calibration (original scoring system) | Calibration (after adjustment) |
---|---|---|---|---|---|
APACHE II | |||||
Khwannimit and Bhurayanontachai (2011) [51] | Recalibration (first-level customisation): customised APACHE II logit = –7.7206 + (APACHE II score × 0.2013) + new diagnostic category weight (Appendix I [51]) | 0.936 (0.925–0.947) (entire population n = 2022) | 0.936 (0.925–0.947) (validation dataset n = 1011) | C statistic χ2 = 104.2 (p = 0.001), H statistic χ2 = 113.1 (p < 0.001) | C statistic χ2 = 16.1 (p = 0.09), H statistic χ2 = 14.1 (p = 0.17) |
Eapen et al. (1997) [30] | Variable adjustment: GCS excluded | Not evaluated | 0.6068 | Not reported | Not reported |
Hashmi et al. (2016) [77] | Modelling technique adjustments: APACHE II calculated automatically by software which uses manually entered values using the logit equation = –4.063 + (APACHE II) × 0.181 | 0.823 (0.76–0.88) (manual calculation) | 0.827 (0.77–0.88) (software calculation) | χ2 = 11.76 (p = 0.16) | χ2 = 5.46 (p = 0.71) |
Nimgaonkar et al. (2004) [35] | Modelling technique adjustments: Artificial Neural Network (ANN 22) model trained on an Indian patient dataset using all 22 APACHE II variables | 0.77 | 0.87 (p < 0.002) | H statistic χ2 = 123.5 (p < 0.05) | H statistic χ2 = 22.4 (p < 0.05) |
Nimgaonkar et al. (2004) [35] | Modelling technique adjustments: Artificial Neural Network (ANN 15) model trained on an Indian patient dataset using 15 APACHE II variables | 0.77 | 0.88 (p < 0.001) ANN 15 | H statistic χ2 = 123.5 (p < 0.05) | H statistic χ2 = 27.7 (p < 0.05) |
SAPS II | |||||
Khwannimit and Bhurayanontachai (2011) [51] | Recalibration (first-level customisation): customised SAPS II logit = –10.1779 + 0.0719 (SAPS II score) + 1.4891 × ln(SAPS II score + 1) | 0.914 (0.901–0.928) (entire population n = 2022) | 0.919 (0.900–0.938) (validation dataset n = 1011) | C statistic χ2 = 124.9 (p < 0.001), H statistic χ2 = 97.5 (p < 0.001) | C statistic χ2 = 8.6 (p = 0.57), H statistic χ2 = 9.6 (p = 0.48) |
Zhao et al. (2013) [50] | Variable adjustment: 1. Underlying disease variables excluded 2. Admission type variables excluded | 0.776 (95% CI 0.750–0.802) at admission, 0.826 (95% CI 0.803–0.850) at 24 hours | Not reported: correlation was suggested between the simplified SAPS II score at each time point and outcome with OR of 1.109 (p = 0.000), regardless of the diagnosis | Not reported | Not reported |
SAPS 3 | |||||
Khwannimit and Bhurayanontachai (2011) [51] | Recalibration (first-level customisation): customised SAPS 3 logit = –33.4249 + ln(SAPS 3 score +1) × 7.8699 | 0.913 (0.899–0.924) (entire population n = 2022) | 0.917 (0.897–0.937) (validation dataset n = 1011) | C statistic χ2 = 170 (p < 0.001), H statistic χ2 = 79.9 (p < 0.001) | C statistic χ2 = 8.2 (p = 0.61), H statistic χ2 = 79.9 (p < 0.001) |
Riviello et al. (2016) [43] | MPM (0) III | Exclusion of two patients (0.5%) due to lack of discharge vital status Normal values attribution details provided in Supplementary Table 3 of the original paper. Highest proportions of missing values were for GCS (36.30%) followed by chronic renal compromise/insufficiency (7.96%) |
Study | Model | Performance |
---|---|---|
Abhinandan and Vedavathi (2013) [60] | APACHE II | Student t test |
Although APACHE II score was higher among non-survivors than survivors (23.28 vs 18.75), it was just statistically significantly with p = 0.068+ | ||
Haidri et al. (2011) [74] | APACHE II | Comparison of means between those who survived and those who died |
The mean 24 h APACHE II score of those who were discharged was 18.93 ± 7.19 and that of those who died was 22.33 ± 7.80. | ||
Mohan et al. (2015) [46] | APACHE II | 30% of patients with APACHE II score < 14 died (unadjusted relative risk = 1.00) and 68.3% with score > 14 died (relative risk = 2.6 (95% CI 1.5–2.7), p < 0.001. |
Naved et al. (2011) [34] | APACHE II (48 hours) | Chi-square test |
Significant relationship between outcome and APACHE II score (χ2 = 58.7, p = 0.001) | ||
Teoh et al. (1991) [81] | APACHE II | APACHE II scores were correlated with hospital mortality (bar graph) |
Mortality was higher with a higher APACHE II score. There were no deaths in the 0–4 APACHE II score group. In higher ranges of APACHE II score of 30 onwards there was a 100% mortality, except for APACHE II score of 45–49 for which there were no admissions within this group | ||
Turner et al. (1989) [36] | APACHE II | APACHE II scores were correlated with hospital mortality (bar graphs plotted) |
Shoukat et al. (2016) [71] | APACHE IV | The mortality increased with an increase in APACHE IV score (scores vs mortality presented as bar graph). All patients with score more than 39 did not survive |
Yamin et al. (2011) [28] | APACHE IV | Mean predicted mortality of overall patient was found to be 25.7% while observed mortality was 28.4% with SD of 0.439 and SMR = 1.09. 62.1% of the overall population show the same outcome as predicted by APACHE IV (p = 0.61) |
Celik et al. (2014) [63] | SAPS II | Student t test |
Mean SAPS II score of the patients who died (59.37 ± 16.50) was significantly higher than that of the patients who were discharged (33.70 ± 13.90) (t = 18.85, p= 0.000). | ||
Zhao et al. (2013) [50] | Simplified SAPS II | A correlation was suggested between the simplified SAPS II score at each time point and outcome with OR of 1.109 (p = 0.000), regardless of the diagnosis |
Study | Scoring system | Discrimination | Calibration | Sensitivity | Specificity | Accuracy |
---|---|---|---|---|---|---|
Khwannimit and Bhurayanontachai (2011) [51] | APACHE II (recalibrated model) | 0.936 (0.925–0.947) | C statistic χ2 = 16.1 (p = 0.09) | NR | NR | NR |
H statistic χ2 = 14.1(p < 0.17) | ||||||
Khwannimit and Bhurayanontachai (2011) [51] | APACHE II | 0.936 (0.925–0.947) | C statistic χ2 = 104.2(p < 0.001) | NR | NR | NR |
H statistic χ2 = 113.1(p < 0.001) | ||||||
Khan et al. (2015) [32] | APACHE II (48 hours) | 0.933 (0.873–0.992) | NR | 94.1% (DC > 9.5) | 86.3% (DC > 9.5) | NR |
Godinjak et al. (2016) [31] | APACHE II | 0.920 (0.87–0.97) | NR | 74.5% (DC = 27.5) | 93.4% (DC = 27.5) | NR |
Khawannimit and Geater (2007) [38] | APACHE II | 0.911 (0.891–0.93) | C statistic χ2 = 66.65 (p < 0.001) | 73.87% (95% CI = 65.23–75.66) | 92% (95% CI = 89.66–93.20) | 87% (95% CI = 85.47–89.13) |
H statistic χ2 = 66.59 (p < 0.001) | ||||||
Fadaizadeh et al. (2012) [67] | APACHE II | 0.897 (0.858–0.937) | C statistic χ2 = 3:27 (p = 0:916) | 90% (DC = 13.5) | 75% (DC 13.5) | NR |
Juneja et al. (2012) [1] | APACHE II | 0.894 (0.864–0.925) | χ2 = 7.959 (p = 0.438) | 74.8% (DC >20.5) | 84.9% (DC >20.5) | NR |
Soares et al. (2004) [40] | APACHE II | 0.888 (0.868–0.907) | H statistic χ2 = 78.181 (p < 0.001) | NR | NR | NR |
Xing et al. (2015) [52] | APACHE II | 0.863 (0.804–0.923) | χ2 = 3.486 (p = 0.900) | NR | NR | NR |
Sathe and Bapat (2014) [69] | APACHE II | 0.86 | χ2 = 12.8 (p = 0.03) | NR | NR | NR |
Hosseini and Ramazani (2015) [80] | APACHE II | 0.857 (0.788–0.925) | χ2 = 10.203 (p = 0.251) | 96.6% | 62.80% | 79.70% |
Naqvi et al. (2016) [33] | APACHE II | 0.835 | χ2 = 3.199 (p = 0.866) | NR | NR | NR |
Gilani et al. (2014) [70] | APACHE II | 0.828 | χ2 = 5.419 (p = 0.712) | 88.2% (DC = 19) | 65.5% (DC = 19) | 27.9% |
Hashmi et al. (2016) [77] | APACHE II (automatic calculation using software) | 0.827 (0.77–0.88) | χ2 = 5.46 (p = 0.71) | 55.71% | 90.21% | 78.87% |
Hashmi et al. (2016) [77] | APACHE II | 0.823 (0.76–0.88) | χ2 = 11.76 (p = 0.16) | 51.42% | 90.91% | 77.9% |
Chang et al. (2006) [64] | APACHE II | 0.82 | χ2 = 9.8 (p =0.28) | NR | NR | NR |
Nouira et al. (1998) [66] | APACHE II | 0.82 | C statistic χ2 = 25.95 (p < 0.001) | NR | NR | NR |
H statistic χ2 = 32.15 (p < 0.05) | ||||||
Liu et al. (2016) [61] | APACHE II | 0.813 ± 0.055 | NR | 89.6% (DC ≥ 15.0) | 74.8% (DC ≥ 15.0) | NR |
Ratanarat et al. (2005) [68] | APACHE II | 0.788 | NR | 80.9% (DC = 20) | 63.2% (DC = 20) | NR |
Khan et al. (2015) [32] | APACHE II | 0.785 (0.69–0.88) | NR | 94.4% (DC > 9.5) | 49% (DC > 9.5) | NR |
Nimgaonkar et al. (2004) [35] | APACHE II | 0.77 | H statistic χ2 = 123.5 (p < 0.05) | NR | NR | NR |
Faruq et al. (2013) [37] | APACHE II | 0.75 (0.67–0.82) | C statistic χ2 = 8.304 (p = 0.40) | NR | NR | NR |
Ahluwalia et al. (1999) [29] | APACHE II | 0.74 | NR | 93% | 23.6% | NR |
Chiavone and Rasslan (2005) [65] | APACHE II | 0.729 (0.63–0.83) | NR | NR | NR | NR |
Wilairatana et al. (1995) [82] | APACHE II | 0.723 | NR | 77.4% (DC = 19) | 61.1% (DC = 19) | 70.8% |
Aggarwal et al. (2006) [7] | APACHE II | 0.713 | C statistic χ2 = 119.3 (p < 0.001) H statistic χ2 = 81.1 (p < 0.001) | 48% (39.9–56.2) (DC = 25%), 20.1% (14.1–27.3) (DC = 50%), 3.2% (1.1–7.4) (DC = 75%) | 84% (79.6–88.2) (DC 25%), 96% (92.7–97.7) (DC 50%), 97.7% (98.1–100) (DC 75%) | 71.9% (67.5–76) (DC 25%), 69.9% (65.1–74.3) (DC 50%), 66.8 (62.3–71.1) (DC 75%) |
Halim et al. (2009) [75] | APACHE II | 0.694 | χ2 = 10.627 (p = 0.014) | 83% | 55.2% | 66% |
Gupta and Arora (2004) [72] | APACHE II | 0.63 | χ2 = 10.34 (p > 0.05) | NR | NR | 89.7% (DC 70%) |
Sekulic et al. (2015) [48] | APACHE II | 0.623 | χ2 = 3.05 (p =0.931) | Presented as a figure | 81.80% | NR |
Eapen et al. (1997) [30] | APACHE II (VA) | 0.6068 | NR | NR | NR | NR |
Galal et al. (2013) [49] | APACHE II | 0.6 | χ2 = 7.34, p = 0.39 | 93% (DC = 11) | 24% (DC = 11) | 55.2% |
Evran et al. (2016) [47] | APACHE II | Presented as a figure | NR | NR | NR | 81.3% |
Kiatboonsri and Charoenpan (1995) [45] | APACHE II | NR | NR | 60% (DC = 50%) | 95% (DC = 50%) | 83% (DC = 50%) |
Juneja et al. (2012) [1] | APACHE III | 0.922 (0.894–0.949) | χ2 = 3.674 (p = 0.885) | 78.6% (DC > 73) | 86% (DC > 73) | NR |
Soares et al. (2004) [40] | APACHE III | 0.915 (0.898–0.933) | H statistic χ2 = 117.206 (p < 0.001) | NR | NR | NR |
Shrestha et al. (2011) [73] | APACHE III | 0.895 (0.839–0.952) | χ 2 = 16.904 (p = 0.031) | 91% | 73.97% | 80.34% |
Gilani et al. (2014) [70] | APACHE III | 0.78 | χ2 = 8.442 (p = 0.392) | 82.3% (DC = 24) | 58% (DC = 24) | NR |
Wilairatana et al. (1995) [82] | APACHE III | 0.694 | NR | 79.8% (DC = 60) | 66% (DC = 60) | 72.4% |
Juneja et al. (2012) [1] | APACHE IV | 0.928 (0.903–0.953) | χ2 = 8.790 (p = 0.360) | 93.2% (cut-off point > 12.5) | 66.9% (cut-off point > 12.5) | NR |
Nassar et al. (2012) [42] | APACHE IV | 0.883 (0.874–0.891) | C statistic χ2 = 53.7 (p < 0.01) | NR | NR | NR |
Xing et al. (2015) [52] | APACHE IV | 0.873 (0.813–0.934) | χ2 = 3.756 (p = 0.878) | NR | NR | NR |
Hamza et al. (2009) [76] | APACHE IV | 0.845 (0.786–0.904) | H statistic χ2 = 5.123 (p = 0.744) | NR | NR | NR |
Evran et al. (2016) [47] | APACHE IV | Presented as a figure | NR | NR | NR | 79.30% |
Study | Scoring system | Discrimination | Calibration | Sensitivity | Specificity | Accuracy |
---|---|---|---|---|---|---|
Ratanarat et al. (2005) [68] | SAPS I | 0.746 | NR | 70.2% (DC = 15) | 67.1% (DC = 15) | NR |
Nair et al. (2016) [62] | SAPS I | 0.742 | NR | 44.4% (DC = 61) | 94.9% (DC = 61) | 85.42% |
Khwannimit and Bhurayanontachai (2011) [51] | SAPS II (recalibrated model) | 0.919 (0.899–9.24) | C statistic χ2 = 8.6 (p = 0.57) | NR | NR | NR |
H statistic χ2 = 9.6 (p = 0.48) | ||||||
Khwannimit and Bhurayanontachai (2011) [51] | SAPS II | 0.919 (0.899–9.24) | C statistic χ2 = 124.9 (p < 0.001) | NR | NR | NR |
H statistic χ2 = 97.5 (p < 0.001) | ||||||
Soares et al. (2004) [40] | SAPS II | 0.916 (0.899–0.933) | H statistic χ2 = 29.400 (p < 0.001) | NR | NR | NR |
Juneja et al. (2012) [1] | SAPS II | 0.899 (0.870–0.928) | χ2 = 14.097 (p = 0.079) | 83.5% (DC > 47.5) | 83.5% (DC > 47.5) | NR |
Godinjak et al. (2016) [31] | SAPS II | 0.892 (0.84–0.94) | NR | 90.2% (DC = 50.5) | 75.7% (DC =5 0.5) | NR |
Khawannimit and Geater (2007) [38] | SAPS II | 0.888 (0.867–0.909) | C statistic χ2 = 71.44 (p < 0.001) H statistic χ2 = 54.01 (p < 0.001) | 70.65% (95% CI = 65.23–75.66) | 89% (95% CI = 87.08–91.02) | 85% (95% CI = 82.75–86.70) |
Fadaizadeh et al. (2012) [67] | SAPS II | 0.887 (0.847–0.926) | C statistic χ2 = 7014 (p = 0:522) | 83% (DC = 86.5) | 77% (DC = 86.5) | NR |
Sutheechet (2009) [79] | SAPS II | 0.88 (0.85–0.91) | C statistic χ2 = 20.65 (p = 0.008) | Individual values for each risk level | Individual values for each risk level | Individual values for each risk level |
Soares and Salluh (2006) [39] | SAPS II | 0.88 (0.86–0.9) | C statistic χ2 = 32.136 (p < 0.001) | NR | NR | NR |
Khan et al. (2015) [32] | SAPS II (48 hours) | 0.871 (0.794–0.948) | NR | 70.6% (DC > 30) | 86.3% (DC > 30) | NR |
Hamza et al. (2009) [76] | SAPS II | 0.845 (0.787–0.903) | H statistic χ2 = 12.140 (p = 0.145) | NR | NR | NR |
Soares et al. (2010) [41] | SAPS II | 0.84 (0.81–0.87) | C statistic χ2 = 21.143 (p = 0.007) | NR | NR | NR |
Nouira et al. (1998) [66] | SAPS II | 0.84 | C statistic χ2 = 73.78 (p < 0.001) | NR | NR | NR |
H statistic χ2 = 76.89 (p < 0.05) | ||||||
Sathe and Bapat (2014) [69] | SAPS II | 0.83 (0.81–0.86) | χ2 = 26.6 (p = 0.001) | NR | NR | NR |
Zhao et al. (2013) [50] | SAPS II | 0.826 (0.803–0.85) | χ2 = 12.176 (p = 0.144) | 85% | 74.3% | 82.4% |
Zhao et al. (2013) [50] | SAPS II (48 hours) | 0.821 (0.795–0.848) | χ2 = 11.294 (p = 0.186) | 85% | 74.3% | 83.8% |
Aggarwal et al. (2006) [7] | SAPS II | 0.781 | C statistic χ2 = 195.6 (p < 0.001) H statistic χ2 = 159.6 (p < 0.001) | 46.1% (38.1–54.3) (DC = 25%), 27.35% (20.4–35.0) (DC = 50%), 10.4% (6.1–16.3) (DC = 75%) | 89.3% (85.2–92.5) (DC = 25%), 95.6% (92.7–97.7) (DC = 50%), 98.7% (96.6–99.6) (DC = 75%) | 74.6% (70.3–78.5) (DC = 25%), 72.3% (68.0–76.4) (DC = 50%), 68.6% (64.1–72.8) (DC = 75%) |
Gilani et al. (2014) [70] | SAPS II | 0.78 | χ2 = 8.575 (p = 0.379) | 70.5% (DC = 13) | 63% (DC = 13) | NR |
Zhao et al. (2013) [50] | SAPS II (initial) | 0.776 (0.75–0.802) | χ2 = 8.332 (p = 0.402) | 85% | 74.3% | 80% |
Naqvi et al. (2016) [33] | SAPS II | 0.75 | χ2 = 3.724 (p = 0.811) | NR | NR | NR |
Faruq et al. (2013) [37] | SAPS II | 0.74 (0.66–0.81) | C statistic χ2 = 9.040 (p = 0.34) | NR | NR | NR |
Khan et al. (2015) [32] | SAPS II | 0.718 (0.608–0.828) | NR | 70.6% (DC > 30) | 60.8% (DC > 30) | NR |
Wilairatana et al. (1995) [82] | SAPS II | 0.71 | NR | 0.742 (cut-off point = 14) | 0.6 (cut-off point = 14) | 68.40% |
Sekulic et al. (2015) [48] | SAPS II | 0.69 | χ2 = 4.41 (p = 0.732) | Presented as a figure | Presented as a figure | NR |
Galal et al. (2013) [49] | SAPS II | 0.59 | χ2 = 7.2, p = 0.3 | 53.4% (DC = 40) | 62% (DC = 40) | 57.1% |
Mohan et al. (2015) [46] | SAPS II | NR | NR | 81.1% (DC > 35) | 59.5% (DC > 35) | NR |
Xing et al. (2015) [52] | SAPS 3 | 0.948 (0.914–0.982) | χ2 = 4.987 (p = 0.759) | NR | NR | NR |
Khwannimit and Bhurayanontachai (2011) [51] | SAPS 3 (AUS) (recalibrated model) | 0.917 (0.902–0.929) | C statistic χ2 = 8.2 (p = 0.61) | NR | NR | NR |
H statistic χ2 = 79.9 (p < 0.001) | ||||||
Khwannimit and Bhurayanontachai (2011) [51] | SAPS 3 (AUS) | 0.917 (0.902–0.929) | C statistic χ2 = 170 (p < 0.001) | NR | NR | NR |
H statistic χ2 = 79.9 (p < 0.001) | ||||||
Khwannimit and Bhurayanontachai (2011) [51] | SAPS 3 | 0.914 (0.901–0.928) | C statistic χ2 = 176.3 (p < 0.001) | NR | NR | NR |
H statistic χ2 = 101.6 (p < 0.001) | ||||||
Juneja et al. (2012) [1] | SAPS 3 | 0.901 (0.871–0.932) | χ2 = 13.123 (p = 0.108) | 76.7% (DC > 56.5) | 84.7% (DC > 56.5) | NR |
Soares and Salluh (2006) [39] | SAPS 3 (CSA) | 0.87 (0.85–0.9) | C statistic χ2 = 9.132 (p = 0.33) | NR | NR | NR |
Soares and Salluh (2006) [39] | SAPS 3 | 0.87 (0.85–0.9) | C statistic χ2 = 13.637 (p = 0.092) | NR | NR | NR |
Silva Junior et al. (2010) [55] | SAPS 3 | 0.86 (0.83–0.88) | C statistic χ2 = 10.47 (p = 0.234) | 0.75 (DC = 57) | 0.86 (DC = 57) | NR |
Nassar et al. (2012) [42] | SAPS 3 | 0.855 (0.846–0.864) | C statistic χ2 = 226.6 (p < 0.01) | NR | NR | NR |
Soares et al. (2010) [41] | SAPS 3 (CSA) | 0.84 (0.81–0.87) | C statistic χ2 = 12.608 (p = 0.126) | NR | NR | NR |
Soares et al. (2010) [41] | SAPS 3 | 0.84 (0.81–0.87) | C statistic χ2 = 15.804 (p = 0.045) | NR | NR | NR |
Hernandez et al. (2014) [78] | SAPS 3 | 0.8 (0.78–0.81) | C statistic χ2 = 11.5 (p = 0.18) | NR | NR | NR |
Evran et al. (2016) [47] | SAPS 3 | Presented as a figure | NR | NR | NR | 81.3% |
Mohan et al. (2015) [46] | SAPS 3 | NR | NR | 81.1% (DC > 47) | 51.1% (DC > 47) | NR |
Study | Scoring system | Discrimination | Calibration | Sensitivity | Specificity | Accuracy |
---|---|---|---|---|---|---|
Nouira et al. (1998) [66] | MPM (24 hours) | 0.88 | C statistic χ2 = 29.59 (p < 0.001) | NR | NR | NR |
H statistic χ2 = 19.9 (p < 0.05) | ||||||
Nouira et al. (1998) [66] | MPM (initial) | 0.85 | C statistic χ2 = 36.66 (p < 0.001) | NR | NR | NR |
H statistic χ2 = 38 (p < 0.05) | ||||||
Sekulic et al. (2015) [48] | MPM II (7 days) | 1.00 | χ2 = 0.00 (p =1.000) | 100% | Presented as figure | NR |
Juneja et al. (2012) [1] | MPM II (initial) | 0.928 (0.904–0.952) | χ2 = 8.627 (p = 0.375) | 95.1% (DC > 27) | 68.2% (DC > 27) | NR |
Sutheechet (2009) [79] | MPM II (24 hours) | 0.91 (0.88–0.93) | C statistic χ2 = 14.45 (p = 0.07) | Individual values for each risk level | Individual values for each risk level | Individual values for each risk level |
Soares et al. (2004) [40] | MPM II (24 hours) | 0.909 (0.891–0.926) | H statistic χ2 = 114.713 (p < 0.001) | NR | NR | NR |
Soares et al. (2004) [40] | MPM II (initial) | 0.854 (0.83–0.878) | H statistic χ2 = 373.317 (p < 0.001) | NR | NR | NR |
Sekulic et al. (2015) [48] | MPM II (48 hours) | 0.836 | χ2 = 11.37 (p = 0.181) | Presented as figure | Presented as figure | NR |
Sekulic et al.(2015) [48] | MPM II (72 hours) | 0.817 | χ2 = 6.04 (p = 0.534) | Presented as figure | Presented as figure | NR |
Hamza et al. (2009) [76] | MPM II (initial) | 0.81 (0.738–0.882) | H statistic χ2 = 8.825 (p = 0.357) | NR | NR | NR |
Aggarwal et al. (2006) [7] | MPM II (24 hours) | 0.733 | C statistics χ2 = 73.1 (p < 0.001) H statistics χ2 = 69.7 (p < 0.001) | 48.1% (39.9–56.2) (DC = 25%), 24.0% (17.5–31.6) (DC = 50%), 14.35 (9.2–20.8) (DC = 75%) | 0.836 (0.789–0.876) (DC = 25%), 0.956 (0.927–0.977) (DC = 50%), 0.990 (0.971–0.998) (DC = 75%) | 0.715 (0.671–0.756) (DC = 25%), 0.712 (0.668–0.754) (DC = 50%), 0.701 (0.657–0.743) (DC = 75%) |
Aggarwal et al. (2006) [7] | MPM II (initial) | 0.665 | C statistics χ2 = 599.2 (p < 0.001) H statistics χ2 = 456.4 (p < 0.001) | 27.9% (0.210–0.355) (DC = 25%), 11.4% (0.069–0.174) (DC = 50%), 2.5% (0.007–0.064) (DC = 75%) | 0.920 (0.884–0.948) (DC = 25%), 0.980 (0.957–0.993) (DC = 50%), 0.993 (0.976–0.999) (DC = 75%) | 0.699 (0.655–0.741) (DC = 25%), 0.682 (0.637–0.724) (DC = 50%), 0.660 (0.615–0.703) (DC = 75%) |
Sekulic et al. (2015) [48] | MPM II (initial) | 0.654 | χ2 = 6.90 (p = 0.548) | 85.20% | Presented as figure | NR |
Sekulic et al. (2015) [48] | MPM II (24 hours) | Presented as figure | χ2 = 16.23 (p = 0.039) | Presented as figure | Presented as figure | NR |
Juneja et al. (2012) [1] | MPM III (initial) | 0.947 (0.927–0.967) | χ2 = 13.835 (p = 0.086) | 94.2% (DC >19) | 80.2% (DC > 19) | NR |
Nassar et al. (2012) [42] | MPM III (initial) | 0.84 (0.83–0.849) | C statistic χ2 = 134.2 (p < 0.01) | NR | NR | NR |
Riviello et al. (2016) [43] | MPM III (initial) | 0.72 | χ2 = 17.66 (0.024) | NR | NR | NR |
Soares et al. (2010) [41] | MPM III (24 hours) | 0.71 (0.67–0.75) | C statistic χ2 = 14.242 (p = 0.076) | NR | NR | NR |
Discriminatory ability of models
Ability of models to calibrate
New model development
Study | Ahluwalia et al. (1999) [29] | Riviello et al. (2016) [43] | Riviello et al. (2016) [43] | Nimgaonkar et al. (2004) [35] | Nimgaonkar et al. (2004) [35] |
---|---|---|---|---|---|
Model | New score | Rwanda MPM (R-MPM) | Simplified R-MPM | Artificial Neural Network (ANN 22) | Artificial Neural Network (ANN 15) |
Source | Prospective cohort | Prospective cohort | Prospective cohort | Prospective cohort | Prospective cohort |
Participants | Consecutive admissions (>13 years) to eight-bed medical ICU, India; inclusion period NR; participant age range 13–80, mean = 46 | Consecutive patients (>15 years) admitted to two ICUs in different hospitals; exclusion criteria: not specified; August 2013–October 2014; participant age range 34 years (IQR 25–47) (median) | Consecutive patients (>15 years) admitted to two ICUs in different hospitals; exclusion criteria: not specified; August 2013–October 2014; participant age range 34 years (IQR 25–47) (median) | All consecutive patients (>12 years) admitted to 17-bed medical–neurological ICU, tertiary referral hospital, India; January 1996–May 1998 | All consecutive patients (>12 years) admitted to 17-bed medical–neurological ICU, tertiary referral hospital, India; January 1996–May 1998 |
Outcomes | Hospital mortality | Hospital mortality | Hospital mortality | Hospital mortality | Hospital mortality |
Predictors | 1. pH (at admission); 2. serum albumin (at admission); 3. heart rate (at 48 hours); 4. GCS (at 48 hours); 5. bilirubin (at 48 hours) | Only the following five variables were included: 1. age; 2.confirmed or suspected infection within 24 hours of ICU admission; 3. hypotension or shock as a reason for ICU admission; 4. heart rate at ICU admission; 5. GSC at time of admission | Altered mental status on ICU admission (present vs not present) used in place of the GCS score in the R-MPM (see previous model) | 22 APACHE II variables | 15 APACHE II variables with the highest information gain (measured by calculation of entropy) |
Sample size | 79 | 427 | 427 | 2962 | 2962 |
Missing data | Not reported | Normal values attributed as in original study; two patients excluded due to lack of discharge status | Normal values attributed as in original study; two patients excluded due to lack of discharge status | Not reported | Not reported |
Model development | Based on APACHE II (Knaus et al. 1985 [10]) and 11 other clinical and laboratory parameters. Backward step method used to remove non-significant (p > 0.05) variables (of univariate analysis) | Based on the 16 MPM III (initial) and additional variables. Variables for inclusion in model selected from the univariate analyses, based on their predictive power (as determined by p < 0.05) as well as their ease of capture based on experience, the proportion of missing values in the dataset, and their clinical significance | Based on the 16 MPM III (initial) and additional variables. Variables for inclusion in model selected from the univariate analyses, based on their predictive power (as determined by p < 0.05) as well as their ease of capture based on experience, the proportion of missing values in the dataset, and their clinical significance | Artificial Neural Network trained on an Indian patient dataset using the 22 APACHE II variables | Artificial Neural Network trained on an Indian patient dataset using the 15 APACHE II variables with the highest information gain (measured by calculation of entropy) |
Model performance | Discrimination measured in terms of AUROC, sensitivity and specificity. Multivariate and univariate regression | Discrimination measured in terms of AUROC. Calibration measured as Hosmer–Lemeshow. Multivariate and univariate regression | Discrimination measured in terms of AUROC. Calibration measured as Hosmer–Lemeshow. Multivariate and univariate regression | Discrimination measured as AUROC. Calibration measured as Hosmer–Lemeshow | Discrimination measured as AUROC. Calibration measured as Hosmer–Lemeshow |
Model evaluation | Developmental dataset only, no further evaluation (compared with APACHE II at 48 hours) | Internal validation with bootstrapping (compared with MPM III (initial)) | Internal validation with bootstrapping (compared with MPM III (initial)) | Data from 1962 patients were used to train the neural network using a back-propagation algorithm. Data from the remaining 1000 patients were used for testing this model and comparing it with APACHE II | Data from 1962 patients were used to train the neural network using a back-propagation algorithm. Data from the remaining 1000 patients were used for testing this model and comparing it with APACHE II |
Results | New score ROC: 0.90, sensitivity: 98.2%, specificity: 66.6%. APACHE II (after 48 hours) ROC: 0.74, sensitivity: 92.8%, specificity: 23.6% | Rwanda MPM (R-MPM) AUROC: 0.81 (0.77–0.86), HL: χ2 = 11.94 (p = 0.154). MPM III (initial) AUROC: 0.72, HL: χ2 = 17.66 (p = 0.024) | Simplified R-MPM AUROC: 0.76, HL: χ2 = 11.46 (p = 0.177). MPM III (initial) AUROC: 0.72, HL: χ2 = 17.66 (p = 0.024) | ANN 22 AUROC: 0.87, HL H statistic: χ2 = 22.4 (p < 0.05). APACHE II AUROC: 0.77, HL H statistic: χ2 = 123.5 (p < 0.05) | ANN 15 AUROC: 0.88, HL H statistic: χ2 = 27.7 (p < 0.05). APACHE II AUROC: 0.77, HL H statistic: χ2 = 123.5 (p < 0.05) |