Prognostic accuracy of the Hamilton Early Warning Score (HEWS) and the National Early Warning Score 2 (NEWS2) among hospitalized patients assessed by a rapid response team

Patients admitted to the hospital ward are at risk of short-term deterioration. Approximately 3–9% of hospitalized patients will demonstrate subjective or objective signs of clinical or biochemical deterioration during their hospitalization [1]. To improve quality of care among admitted patients, many hospitals have implemented rapid response teams (RRTs), which serve to identify and respond to deteriorating patients outside of the intensive care unit (ICU) [2]. While it remains unclear whether RRTs are associated with improved hospital mortality, the existing evidence does support their use in reducing cardiac arrests, facilitating ICU admission, and engaging patients and caregivers in discussions related to goals of care and end of life [3‐7].

Early Warning Scores (EWSs), which incorporate objective measures of clinical status such as vital signs with clearly defined thresholds, serve to rapidly identify patients at risk of future deterioration (including death and ICU admission) [8]. A wide number of EWSs exist, and these scores have been found to have varying degrees of accuracy in predicting cardiac arrest and death [9, 10]. These tools may be utilized to increase monitoring, escalate care, or activate the RRT. RRTs may use these scores for the purposes of prognosis, in order to guide how aggressive to be with resuscitation, as well as disposition. Perhaps, the most well-known EWS, with the best prognostic characteristics, is the National Early Warning Score (NEWS), created by the Royal College of Physicians, London [11]. In 2017, the Royal College of Physicians updated the NEWS score (NEWS2), in order to improve precision (as based upon larger validation databases that included more varied patient types) [12] and to improve awareness of potential sepsis. A NEWS2 ≥ 5 represents the key threshold for urgent response.

The Hamilton Early Warning Score (HEWS) is a novel EWS that has been successfully pilot-tested in the emergency department (ED) setting at two hospitals [13, 14], and is being used in several centers in Canada. A HEWS ≥ 5 has been described as the threshold for concern of deterioration. Many similarities exist between the HEWS and the NEWS2 (Additional file 2: Figure S1), though little is known regarding how these two EWS compare. In particular, the HEWS and NEWS2 were designed to improve the early detection of all hospitalized patients at risk for deterioration, but particularly among those with suspected infection. Early identification, appropriate management, and disposition of patients with sepsis and septic shock are crucial to improving survival in this population [15]. Previous work investigating the use of the systemic inflammatory response syndrome (SIRS) criteria [16] and the quick Sequential Organ Failure Assessment (qSOFA) criteria [17, 18] in patients with suspected infection assessed by the RRT found both these tools to be unsatisfactory in predicting in-hospital mortality [19]. Given the variable accuracy of the existing EWSs in the RRT population, particularly among patients with suspected infection, new tools for risk stratification are required. The HEWS and NEWS2 are potential early warning tools that may be used in this setting; however, their prognostic accuracy among RRT patients is unknown. Therefore, we sought to compare the prognostic accuracy of the HEWS and the NEWS2 among patients assessed by the RRT, and in particular those patients with suspected infection.

Ethics approval for this study was obtained from The Ottawa Health Science Network Research Ethics Board.

During the study period, the RRT was activated for 6132 discrete adult patients. Of these, 109 were excluded due to missing outcome data. An additional 532 patients (8.7%) did not have sufficient available data to calculate both the HEWS and NEWS2 scores and were therefore excluded (Additional file 3: Figure S2). This left 5491 patients for analysis. Of these patients, 1708 (31.1%) had suspected infection. Baseline characteristics of all patients are depicted in Table 1. Mean age was 67.4 years (SD, 16.3). The majority of these patients (3876, 70.6%) were admitted to hospital from home. The most common comorbidities noted were diabetes mellitus (45.1%), hypertension (33.7%), and arrhythmia (23.1%). Of note, 19.8% of patients had limits on the aggressiveness of their care which did not allow for cardiopulmonary resuscitation or ICU admission. A total of 1837 patients (33.4%) died during hospitalization. RRT call characteristics are displayed in Table 2. The majority of activations took place during daytime hours (64.3%). The most common reasons for RRT activation included respiratory distress (25.4%), altered level of consciousness (18.4%), and tachycardia or arrhythmia (18.1%).

Comparison of the prognostic accuracy of the HEWS and NEWS2 for prediction of in-hospital mortality among all RRT patients is displayed in Table 3. A HEWS ≥ 3 was found among 3181 patients (57.9%). A HEWS ≥ 3 had a sensitivity of 95.9% (95% CI 94.9–96.7) and a specificity of 44.6% (94.9–96.7). The NNE was 2.15 (95% CI 2.12–2.18). A HEWS ≥ 5 was found among 2578 patients (46.9%). Sensitivity was 75.9% (95% CI 73.9–77.9) and specificity was 67.6% (95% CI 66.1–69.1). The NNE was 1.85 (95% CI 1.81–1.89). In comparison, 3416 patients (62.2%) had a NEWS2 ≥ 5. The sensitivity of NEWS2 ≥ 5 was 84.5% (95% CI 82.8–86.2) and specificity was 49.0 (47.4–50.7). The NNE was 2.20 (2.16–2.25). The rTPR of NEWS2 to HEWS was 1.11 (95% CI 0.98–1.24, P = 0.09), while the rFPR was 1.57 (95% CI 1.46–1.69, P < 0.01). Prognostic accuracy of HEWS and NEWS2 for prediction of ICU admission among RRT patients is displayed in Additional file 4: Table S2. Sensitivity of HEWS ≥ 3 for prediction of ICU admission was 91.8% (95% CI 90.2–93.1), as compared to 83.4% (95% CI 81.4–85.3) for NEWS2. Specificity of HEWS ≥ 3 was 63.3% (95% CI 61.6–65.1), as compared to 64.5% (95% CI 62.7–66.2) for NEWS2.

Table 4 describes the prognostic accuracy of HEWS and NEWS2 among the cohort of patients with suspected infection. A HEWS ≥ 5 was found among 799 patients (46.8%). Sensitivity of HEWS ≥ 5 was 83.2% (95% CI 79.8–86.1), and specificity was 71.4% (68.7–74.1). The NNE was 1.69. Conversely, 1075 patients with suspected infection (62.9%) had a NEWS2 ≥ 5. Sensitivity was 87.7% (95% CI 84.7–90.3), and specificity was 49.1% (95% CI 46.2–52.1). The NNE was 2.19 (95% CI 2.11–2.27). Among patients with suspected infection, the rTPR of NEWS2 to HEWS was 1.05 (95% CI 0.94–1.17, P = 0.38) and the rFPR was 1.78 (95% CI 1.66–1.89, P < 0.01).

AUROC curves of HEWS and NEWS2 for prediction of in-hospital mortality among the entire RRT cohort and those only with suspected infection are displayed in Fig. 1. Among the entire RRT cohort (Fig. 1a), HEWS had an AUROC of 0.76 (95% CI 0.75–0.77) and NEWS2 had an AUROC of 0.72 (95% CI 0.71–0.74). For patients with suspected infection only (Fig. 1b), HEWS had an AUROC of 0.79 (95% CI 0.77–0.81), while NEWS2 had an AUROC of 0.75 (95% CI 0.73–0.78).

We evaluated the prognostic accuracy of the HEWS and the NEWS2 among hospitalized patients with acute deterioration, requiring RRT assessment. We found that both HEWS and NEWS2 accurately predicted in-hospital mortality in this population and particularly among patients with suspected infection. At the critical threshold (≥ 5), the NEWS2 had comparable sensitivity to HEWS (as evidenced by rTPR), but lower specificity (as evidenced by rFPR). This was true among the entire cohort, as well as those patients with suspected infection. Taken together, this work provides novel evidence comparing the prognostic accuracy of the NEWS2 and HEWS scores in a population of hospitalized patients at high risk for deterioration.

RRTs serve an important role in the hospital Early Warning System. While RRT intervention can assist with end-of-life care and reduce the rate of unexpected death, it also functions to aid in the care of hospitalized patients outside of the ICU, in order to optimize their management and disposition. To that end, providers often have to prognosticate for risk of future deterioration in order to guide RRT resources. Among hospitalized patients, risk of deterioration can often be predicted through vital signs [28]. However, despite this, there can be delay to RRT activation, and a patient may deteriorate further. Furthermore, certain populations of patients (such as the elderly) may deteriorate prior to objective changes in vital signs, which may contribute to delay to RRT activation [29]. Delays in RRT activation are common and also often reflect variation in time of day, personnel on duty, and patient volume [30, 31]. Prolonged delays are associated with increased mortality in this population [32]. For this reason, EWSs with mandatory calling criteria have an important role in the identification of deteriorating patients and the activation of the RRT.

Incorporating EWSs into electronic ward and health databases provides a promising method for the immediate detection of patients who may deteriorate [33]. However, at present, the ideal utility of an EWS is unknown. It may seem optimal for an EWS to maximize sensitivity, and thereby reduce the number of false negatives. However, in practice, an EWS that is highly sensitive but poorly specific may result in unnecessary workload, alarm fatigue, and an inability of RRT or ICU capacity to meet demand [8]. For this reason, experts have endorsed the utilization of the NNE metric, as it serves to quantify the trade-off between an EWS that is poorly sensitive and thus misses many deteriorating patients, and an EWS that is poorly specific, resulting in frequent activation and alarm fatigue [27].

In this study, we found that the NEWS2 score had comparable sensitivity to HEWS (84.5% vs. 75.9%, rTPR 1.11). However, the specificity of NEWS2 was poorer (49.0%), as compared to HEWS (67.6%), with a rFPR of 1.57. When comparing the NNE, HEWS was superior to NEWS2 (1.85 vs. 2.20). Until this point, there has been little external validation of the NEWS2 score [12]. Overall, we found that the specificity of HEWS (as evidence by rFPR) and its lower NNE may suggest that it may be the preferable EWS in reducing alarm fatigue.

Deterioration among patients admitted to the hospital wards with suspected infection is especially common [15]. Such patients are at risk of developing sepsis and septic shock, unplanned ICU admission, and death [15]. Therefore, significant effort has been directed at the early identification of deterioration in this population. Traditionally, the SIRS criteria have been used for this purpose [34, 35]. Our group has previously shown that, while the SIRS criteria has demonstrated high sensitivity among RRT patients with suspected infection, this has come at the cost of poor specificity [19]. In contrast, while qSOFA [17, 18] has been associated with high specificity for mortality among RRT patients with suspected infection, its overall sensitivity is poor [19, 36]. Churpek et al. evaluated the prognostic accuracy of qSOFA against other EWSs, such as the original NEWS and the Modified Early Warning Score (MEWS), and found that qSOFA was inferior to these existing EWSs in predicting in-hospital mortality [37]. We found that the HEWS score had superior sensitivity and NNE than qSOFA, when applied to our population of RRT patients with suspected infection. This suggests a possible benefit of utilizing the HEWS score as compared to qSOFA among RRT patients with suspected infection, as the higher sensitivity of HEWS may function as a better prompt for escalating treatment.

This study has several strengths, including a large sample size, comparisons among patients with and without suspected infection, and comprehensive data on many patient and RRT variables. Furthermore, our study provides one of the first external applications of both the HEWS and NEWS2. However, there are several limitations that hinder the generalizability of our data. Most importantly, our study does not address the question related to EWS accuracy in identification of patients prior to deterioration, as we only included patients who had experienced RRT activation. We still sought to investigate the prognostic accuracy of these tools for in-hospital mortality and ICU admission, as it provides evidence for the use of HEWS and NEWS2 in risk-stratification among RRT-activated patients. Future prospective work should focus on HEWS and NEWS2 accuracy in identifying patients at risk of deterioration prior to RRT activation. Second, while our database included consecutive patients, NEWS2 and HEWS scores were gathered on the basis of the patient’s first RRT call. Therefore, it is possible that patients may have initially had a lower NEWS2 or HEWS score on initial assessment, but then subsequently deteriorated and died in-hospital on a later date. Third, while we utilized the NNE as a metric of EWS effectiveness, it is important to note that the NNE is derived from the positive predictive value, and therefore is influenced by the prevalence of the studied outcome. Thus, these NNE values may not be generalizable to populations with lower mortality rates. Additionally, 8.7% of patients in our original cohort were excluded due to insufficient data, though the mortality of this population did not differ from our included patients. Finally, while our data were gathered from different centers, they exist within the same health network and city and therefore may be susceptible to local variation in ICU admission and treatment practices.

We found that that the HEWS and NEWS2 had similar overall prognostic accuracy for prediction of in-hospital mortality among RRT patients, and particularly those with suspected infection. While a NEWS2 score ≥ 5 had comparable sensitivity for mortality as a HEWS score ≥ 5, this came at the cost of much lower specificity and a higher overall NNE and therefore may contribute to alarm fatigue. Knowledge related to the prognostic performance of these two EWSs will allow RRT clinicians to properly contextualize their use in the care of patients with acute deterioration.