Comparison of different sepsis scoring systems and pathways: qSOFA, SIRS, Shapiro criteria and CEC SEPSIS KILLS pathway in bacteraemic and non-bacteraemic patients presenting to the emergency department

Sepsis a heterogeneous syndrome broadly defined as life-threatening organ dysfunction caused by a deregulated host response to infection [1]. It remains a leading cause of mortality in patients at extremities of age, the immunosuppressed and individuals with multiple comorbidities, while higher mortality rates have been observed in individuals with bacteraemia associated with sepsis than those without [2‐4]. Survival benefit has been linked to prompt identification of sepsis and septic shock and early administration of antimicrobials, leading to the development of global initiatives including the Surviving Sepsis campaign (developed by the Society of Critical Care Medicine and the European Society of Intensive Care) which provides management plans based on these principles [5, 6]. Locally the Australian Commission on Safety and Quality in Healthcare are in the consultation stage of developing a national sepsis program.

In 1992, the American College of Chest Physicians (ACCP) and the Society of Critical Care Medicine (SCCM) introduced a definition for the Systemic Inflammatory Response Syndrome (SIRS), defining the early response of the host to a nonspecific insult, which may be infectious. Review of this criteria found it to have poor sensitivity in critically ill patients, and failed to distinguish between a host inflammatory response due to infective or and non-infective causes, limiting its use in identifying those at risk of sepsis [7, 8]. In 2008, Shapiro et al. developed a clinical prediction rule to stratify patients at risk of developing bacteraemia presenting to the Emergency Department (ED), in an attempt to rationalise blood culture procurement whilst identifying those at high risk of bacteraemia [9]. The criteria had subsequently been found to have reasonable sensitivity but poor specificity [10]. In 2016, SEPSIS-3 introduced the quick Sequential [Sepsis Related] Organ Failure score (qSOFA) to identify patients at high risk of mortality from sepsis presenting to the ED, due to its greater prognostic accuracy for in-hospital mortality [11]. Based on the simplicity of the score, the Third International Consensus Definitions for Sepsis recommended qSOFA as a prompt, accurate means of identifying patients who were likely to be septic. However, like other scoring systems mentioned above this score had low sensitivity identifying patients at risk of sepsis [12].

On a local level, in 2011 NSW Clinical Excellence Commission (CEC) introduced a state-wide SEPSIS KILLS pathway in all EDs. This pathway incorporated bundles of care that included taking blood cultures, antibiotic administration within an hour of triage and fluid resuscitation. This program aimed to reduce preventable harm to patients through improving the recognition and management of sepsis. For early visual recognition in ED, SEPSIS kills icon was created with an aim for its use by the ED triage nurse. The CEC pathway was based on the SEPSIS-2 definition of sepsis, which involved infection in the presence of 2 systemic inflammatory response syndrome (SIRS) criteria, substituting local ‘Between the Flags’ parameters [13]. Overall mortality appeared to decrease with the implementation of SEPSIS KILLS in NSW hospitals [14].

Considering the high mortality associated with bacteraemia in the septic patient, the primary outcome of this study was to determine the sensitivity and specificity of various sepsis criteria in the existing literature within a cohort of confirmed bacteraemic patients, when compared to an age matched non bacteraemic population.

The strength of agreement between the observers was considered very good (K = 0.983, 95% CI 0.969–0.998).

Overall in this study we found that the scoring systems and pathways assessed performed poorly, with low to moderate sensitivity and overall poor specificity. The positive predictive values were suboptimal in identifying bacteraemic patients presenting to the ED.

This study demonstrated that the modified Shapiro criterion was the most sensitive method of detecting true bacteraemia. However, our data suggests that this method is not sensitive enough as a stand-alone tool to predict bacteraemia, with almost 8% of patients whose bacteraemia would be missed using this scoring system alone. Our calculated sensitivity is lower than previous data, however has similar specificity. [10, 15, 16] Larger studies are therefore required to demonstrated reproducibility in these findings. Practical application of Shapiro in real-world settings should also be considered. Four of the nine minor criteria are obtained from haematological or biochemical blood analysis, meaning that the calculation of the score is not available at initial point of care. Waiting for these results may lead to delays in taking blood cultures and administering antibiotics in septic bacteraemic patients or result in blood cultures being taken after antibiotic administration, significantly reducing their yield. Prospective assessment of the score in real-time, with outcomes including delay to antibiotics and taking blood cultures should be examined.

The CEC SEPSIS KILLS pathway demonstrated the most balanced sensitivity and specificity in detecting bacteraemia with moderate accuracy. In initial development and application of the pathway, there was an emphasis on between the flags (BTF) vital sign criteria and serum lactate level only in the assessment of a patient with possible sepsis. However in recent years there has been the addition of other criteria including clinician clinical concern of sepsis. As the pathway now incorporates both vital signs and clinical assessment by a senior physician in determining the presence or absence of sepsis, it raises the possibility of inter-hospital variability of the tool due to variability in clinical experience. Another single centre NSW based study demonstrated low sensitivity (45.1%) and moderate specificity (78.9%) when clinical impression was not incorporated [17]. However other studies have shown an overestimation in bacteraemia prediction when based on clinical judgement alone [18]. Our study did not analyse whether the individual vital sign criteria was responsible for the activation of the SEPSIS KILLS pathway, or individual senior clinician suspicion, therefore we feel that a multisite assessment into this may be warranted to determine the performance of the tool with vital sign criteria alone. If the vital sign criteria alone were associated with reasonable sensitivity and specificity, we could feel more confident in this tool being used by clinicians of all skill levels.

While CEC SEPSIS KILLS criteria performed reasonably overall, the associated SEPSIS KILLS FirstNet tool was rarely applied in either group, with an activation rate of 14% in our patients with bacteraemia. We speculate that poor uptake by clinical staff was potentially due to inadequate of knowledge of the existence of the tool, the lack of clinical criteria provided in the CEC package in the recommendation for its application, inexperience and insufficient education of staff activating the tool and under-recognition of sepsis due to heterogeneity of presenting symptoms and its evolving pathophysiology. It is also clear that sepsis is an evolving process, so a single point of assessment at a variable point in the pathophysiological process is unlikely to be clinically reliable. We found that bacteraemic patients were correctly identified more often than non bacteraemic patients, which overall is of questionable significance given the overall poor rates of use of the tool. Identification of patients at risk of sepsis with the icon also did not lead to earlier administration of antibiotics. We recommend further evaluation of the uptake of the SEPSIS KILLS in other NSW EDs and include assessment of the rates of its use and clinical impact, including whether time to antibiotic therapy and fluid resuscitation were optimised. Targeted education programs could then be developed to inform frontline ED staff about the recognition of sepsis using these evaluations. The small numbers in our study suggest that antibiotic time did not improve with activation of the SEPSIS KILLS, and identification and activation of the pathway was not associated with improved mortality.

The SIRS criteria have underpinned the principles of detection of bacteraemia in early sepsis recognition pathways that include CEC SEPSIS KILLS. Previous evaluations have demonstrated the tool to be sensitive but poorly specific in the detection of septic patients including those with bacteraemia [19]. However in our population SIRS has demonstrated moderate sensitivity and poor specificity, performing slightly better than in other contemporary studies in this setting [7, 8]. This has become an increasingly recognised phenomenon, which again highlights the diversity of underlying pathophysiology underlying patients with bacteraemia, and can range from gross biochemical and clinical disturbance to minor changes in physiology depending on the time of presentation, underlying comorbidities and pathogen characteristics. SIRS also historically underperformed against more recently developed scoring systems, including qSOFA which has led to its removal from the current SEPSIS-3 definition [20]. Lastly, we found that the criteria could not identify patients at risk of death. This study therefore confirms its lack of value as a clinical prediction tool in sepsis and bacteraemia.

qSOFA was designed as a tool designed to assess severity of illness in sepsis, and initial comparative literature indicated its superiority over its predecessor SIRS in the prediction of mortality and ICU admission [21]. It has however been evaluated as a clinical prediction tool for sepsis and bacteraemia with poor results due to modest sensitivity and specificity [19]. This study confirmed poor sensitivity in clinical diagnosis of sepsis, which we concede is probably driven by its design as an illness severity score rather than a clinical decision making score. The tool was able to predict mortality when it was activated, however it identified few patients in our cohort with bacteraemia and therefore overall was not useful in predicting mortality in this group overall. Similar conclusions have been met, highlighting the inability of qSOFA in identifying mortality risk or likelihood of requiring ICU admission in patients who had severe sepsis, suggesting the need for re-evaluation of whether this tool is clinically useful [12].

In this cohort of patients with sepsis presenting to the emergency department, our study demonstrated also that bacteraemia was associated with a statistically significant increase in hospital length of stay, duration of antibiotics and sepsis-related mortality, when compared to septic patients without bacteraemia. Intensive care unit admission rate and duration was similar in both groups. Contemporary data has demonstrated conflicting evidence when comparing these two cohorts, with some evidence demonstrating higher 30-day mortality and higher and long-term mortality in bacteraemia, while other studies demonstrating no significant difference in mortality or organ dysfunction [22‐24]. Considering the heterogeneity of mortality data, we suggest that this should be a focus of further research.

Overall our findings suggest bacteraemia remains clinically important, with higher rates of mortality. We have found that current scoring systems and pathways are not accurate enough to be used as the only tool for diagnosis or mortality prediction. The heterogeneity of presentation of bacteraemia due to underlying physiological differences, comorbid status and variability in the pathophysiology of different pathogens and infective processes likely account for the difficulty in developing a standardised tool for the detection of the bacteraemic and septic process. Further research is encouraged to confirm our findings.