Development of an algorithm to aid triage decisions for intensive care unit admission: a clinical vignette and retrospective cohort study

Intensive care unit (ICU) admission triage is performed routinely in ICUs worldwide [1, 2], and patients who are refused ICU admission may have a higher risk of death [2]. This triage process is associated with several factors, including clinical characteristics of the patients [2‐4], but it is also influenced by nonclinical factors [2, 4, 5]. Moreover, it has been shown that physicians may not follow triage recommendations [3], and concern has been raised that these clinical judgments could mask prejudice or bias [1]. So, development of objective directives and guidelines applying to individual patients, along with the need for an objective triage score, has been discussed in the literature [6, 7].

The Society of Critical Care Medicine (SCCM) has established guidelines for ICU triage [8], which categorizes patients into four priority strata. Priority 1 patients usually have no therapeutic limits and are critically ill unstable patients in need of intensive treatment and monitoring that cannot be provided outside the ICU. Priority 2 patients usually have no therapeutic limits, require intensive monitoring, and may potentially need immediate intervention. Priority 3 patients are critically ill but have a reduced likelihood of recovery because of underlying disease or the nature of their acute illness; these patients may have therapeutic limitations. Priority 4 patients are generally not appropriate for ICU admission, because either (1) they have little or no anticipated benefit from ICU care based on low risk of active intervention that could not safely be administered in a non-ICU setting or (2) they are patients with terminal and irreversible illness and face imminent death. However, these guidelines have not been formally evaluated.

There is no established gold standard for appropriateness of ICU triage, so, to assess the usefulness of any tool aimed at improving this process, it is important to use more than one method of evaluation and to demonstrate the reliability and validity of the tool [9, 10]. Reliability “refers to the degree to which repeated assessments of the same patient with a triage instrument will deliver the same acuity level,” as such yielding reproducible results [9, 11]. Validity would refer “to the degree with which the measured acuity level reflects the patient’s true acuity at the time of triage” [9], or the capacity of an instrument to reflect what it is proposed to measure. Validity is further delineated into three major types: content (i.e., criteria fit with current knowledge), construct (criteria measure what they are supposed to measure), and criterion (new criteria agree with existing standard) [10, 11].

To help in triage decision making for ICU admission in a setting of ICU bed scarcity, the Hospital das Clinicas da Faculdade de Medicina de Sao Paulo (HCFMUSP) assembled a critical care team to come up with a set of defined criteria for ICU admission. This group proposed an algorithm to aid triage decision making based on the SCCM’s guidelines. The algorithm was designed to help standardize, not replace, clinical judgment in ICU triage. We sought to evaluate the reliability and validity of this algorithm.

This ICU triage algorithm has good reliability and validity, achieving construct validity by comparison of the results of the algorithm-based priorities with the reference standard, and by comparison with physicians’ judgment of appropriateness of ICU admission, in both nonscarcity and scarcity settings. Interrater reliability was substantial and was higher for the algorithm-based priorities than for physician-based priorities. The criterion (predictive) validity of this algorithm was further supported by correlation with surrogate clinical outcomes, such as ICU admission, palliative care consultation, and mortality. Moreover, when analyzing the individual components of the algorithm, it was found that each of these components contributed to the overall correlation.

There is growing consensus that clear and objective directives for ICU admission triage could benefit patient and resource allocation decision making [1, 6, 7]. Nevertheless, triage and risk stratification tools have not yet been routinely implemented in the ICU, other than in catastrophic settings [18, 19]. In part, this situation can be explained by the current absence of a validated triage tool for ICU admission, because the SCCM guidelines for ICU triage have never been formally evaluated, other than in small studies in which researchers examined specific priority strata [17, 20]. Moreover, it has been argued that existing physiological scores alone should not be used to predict outcomes [6]. Therefore, an alternative could be to use standardized subjective scores that combine objective data with the physician’s intuitive judgment of prognosis [21‐23]. Such an instrument could be used to standardize and reduce biases that could adversely influence proper evaluation and decision making for ICU admission of individual patients.

This study has shown that this algorithm-based ICU triage decision-making tool has good interrater reliability, with a κ of 0.61, outperforming the interrater reliability of physicians’ prioritization. Because there are no ICU triage scales to use as benchmarks, this algorithm’s performance was compared with emergency triage scale performance reported in the literature. The authors of a systematic review of studies evaluating the Manchester Triage Score (MTS) [24] found a wide interrater agreement range. The ranges for unweighted and weighted κ statistics in these studies were 0.31–0.76 and 0.40–0.82, respectively. Of note, most studies were of low or moderate quality. The authors of another systematic review [25] found good reliability for the Canadian Triage and Acuity Scale and the Emergency Severity Index, with κ statistics ranging from 0.7 to 0.95, but found the MTS and the Australasian Triage Scale to be less reliable (κ statistics 0.3–0.6). Farrohknia et al. [26], in analyzing the current evidence on emergency triage scales, found no high-quality study available and also found a wide interrater agreement range, with κ values varying from 0.202 to 0.87.

Although comparable to rates for other published triage tools and higher than senior physicians’ agreement rates, these agreement rates are less than perfect. These agreement rates may be due to the inherent subjectivity associated with ICU triage [1], because disagreement on appropriateness of care may occur in up to 66 % of cases labeled as receiving inappropriate care [27]. To minimize this issue, we chose well-defined closed-end questions that have been shown to correlate with outcomes and to have high interrater reliability. The exception was question 4 (estimated prognosis), which was expected to be more intuitive and less reliable. This problem was mitigated by the fact that, in our algorithm, this specific question has less influence on the final priority. Question 4 would change only the ascribed priority from 3 to 4 in the case of a patient with advanced disease or severe functional dependence that would have been ascribed an intuitive estimate of no survival.

There are also controversies in the literature regarding the interpretation of κ values, and some suggest that the usual interpretation of 0.61–0.80 as substantial agreement [16] may be too lenient [28]. However, authors of many recent papers evaluating triage [24, 26, 29‐32] and other scales such as the Glasgow Coma Scale [33] and the American Society of Anesthesiologists physical status classification [34] have used the more traditional interpretation of κ values and have shown similar agreement rates [24, 25, 34], which may help contextualize our results. It is possible that this more stringent approach was meant to be directed for more objective data, such as laboratory values [28]. Moreover, even when the more stringent approach was used, this tool’s performance was at least moderate and was probably better than the physicians’ intuitive priorities. In addition, as suggested by some authors [28], we have also reported other measures of reliability, such as percentage agreement and intraclass correlation coefficient, to help with interpretation of the results.

Validity, or the capacity of the tool to reflect the true triage priority, was evaluated. Because there is no established gold standard for prioritization or appropriateness of ICU, and in accordance with recommendations from the literature [9‐11], the validity of this ICU admission triage algorithm was assessed by using a series of methods. In the context of the clinical vignettes, algorithm-based priorities demonstrated substantial agreement with the selected reference standard (two investigators with access to patients’ records), with a κ value of 0.72. In addition, the algorithm was shown to correlate with physicians’ judgment of appropriateness of ICU admission, especially in an ICU scarcity setting, demonstrating a linear relationship between prioritization and the proportion of scenarios that would be admitted to the last ICU bed. These results support the construct validity of the algorithm.

For further validation, this triage algorithm was compared with actual clinical outcomes in a subset of patients with requests for urgent ICU admission. The algorithm correlated with hospital mortality, palliative care consultation, and ICU admission, which supports its criterion (predictive) validity. However, although statistically significant, the difference in ICU admission rates between higher and lower priorities could be expected to be larger. Importantly, this observed difference is a reflection of current practice because the algorithm had no influence on ICU admission rates. Moreover, there is a perceived scarcity of ICU beds in our hospital, which is corroborated by the 30 % ICU refusal rate, so it is possible that this observed difference in admission rates may be due to a high rate of “inappropriate” admissions. Of note, despite the good correlation with outcomes in the retrospective cohort, the algorithm could not be properly evaluated regarding the association with benefit of ICU admission, defined as the difference in outcome with and without critical care [35].

This study has several strengths. It is one of the few to evaluate a triage tool for ICU admission and is, to our knowledge, the first to evaluate a triage algorithm for ICU admission in a resource-limited, noncatastrophic setting. Despite its single-center nature, in this study we evaluated a population of patients in a large academic tertiary hospital with 10 independent ICUs with different patient admission policies. Acknowledging the difficulty of establishing gold standards to ICU admission, we evaluated different surrogates covering different aspects of the process.

The study has several limitations, however. In the validation cohort, a short period of time was evaluated, and, due to the retrospective algorithm-based priority input, it was not possible to adjust for potential confounders and to ascertain all possible biases. Another important limitation is that in this study we evaluated the correlation of priority levels with outcomes, but the study was not designed to predict the impact of ICU admission on mortality in each priority stratum.

In addition, the clinical vignette methodology used, despite being based on real patients, may be less robust than real encounters with patients. Vignettes offer significant advantages, such as quantification of physician performance, ease of use, and low cost [29], and, as such, researchers in several studies have used case vignettes to study clinicians’ attitudes in both triage [29, 36‐38] and nontriage settings [39‐41]. Nevertheless, the vignette methodology differs from real patients in important ways because it summarizes and standardizes important clinical information and excludes the effect of external factors that could impact the decision-making process, such as time pressure and cognitive load [29]. It could be argued, for instance, that these vignettes’ characteristics could falsely increase the agreement rate in a subjective setting such as ICU triage. However, this is not consensual in the literature, because studies in which researchers have used case vignettes to evaluate decisions to withdraw support in critically ill patients [42] and patients with sepsis [43] and triage for ICU admission of typical ICU cases [44] have demonstrated wide disagreement in individual responses. Because the algorithm improved interrater agreement in the vignette methodology, it is possible to hypothesize that the algorithm would systematize the triage process and that this improvement could be even greater in the unstandardized setting of real patient encounters. Nonetheless, in this study we did not evaluate physicians’ performance with real patients, so we cannot be certain about the impact of the chosen methodology regarding the agreement rate.

Therefore, notwithstanding the good characteristics of this tool, this ICU admission triage algorithm may not be suitable for clinical practice yet, because, despite its correlations with outcomes, it was not properly evaluated for prediction of a differential benefit in each ICU priority stratum that would justify ICU admission of one stratum over the others. Further prospective studies with additional collected variables should be implemented to address the algorithm’s limitations.

An algorithm-based prioritization system for ICU admission derived from the SCCM priority categories demonstrated good reliability and validity, with substantial interrater reliability and correlation with physician’s assessments and with the established reference standard. Algorithm-based priorities were also associated with clinical outcomes such as hospital mortality, palliative care consultation, and ICU admission, with lower priorities correlating with worse outcomes. However, at this stage, this algorithm could not be properly evaluated to predict the differential benefit of critical care for each priority stratum and may still not be ready for implementation. Further prospective studies that address these limitations are warranted.