Background
Sepsis, the syndrome of dysregulated inflammation that occurs with severe infection, is associated with high morbidity, mortality, and cost [
1,
2]. The devastating toll of sepsis on society has prompted national performance improvement initiatives and governmental mandates for sepsis care and reporting, including a recent quality measure issued by the Centers for Medicare and Medicaid Services (CMS) [
3‐
5]. However, reliably identifying cases of sepsis and septic shock is complicated because there is no gold standard diagnostic test [
6]. The diagnosis therefore requires clinicians to interpret a constellation of nonspecific physiological and laboratory abnormalities among patients with suspected or definite infection [
7,
8]. To make the diagnosis of severe sepsis, clinicians must decide whether a patient has an infection, whether acute organ dysfunction is present, and whether acute organ dysfunction (when present) is attributable to infection. These determinations can be subjective and it is thus highly conceivable that thoughtful clinicians might differ substantially in their judgments.
Variability in how clinicians diagnose sepsis has important implications for clinical care, epidemiologic and clinical studies, public health surveillance, pay-for-performance initiatives, and quality improvement programs. Our aim in the present study was to evaluate whether and to what degree intensivists agree in how they diagnose sepsis. To do so, we distributed case vignettes of common scenarios of patients with suspected infection and organ dysfunction to a sample of intensivists. We hypothesized that there would be significant variability in sepsis diagnoses, and that this variability would exist independent of physicians’ degree of confidence in their ability to apply the traditional consensus definitions of sepsis.
Discussion
In this survey of 94 physicians, who primarily were attending intensivists at academic institutions, we found poor agreement in diagnosing sepsis, severe sepsis, or septic shock when respondents were presented with short clinical case vignettes. For purposes of quality monitoring, it is more meaningful to determine whether patients had severe sepsis/septic shock. However, when we examined responses dichotomized in this way, agreement was no better. In addition, when the analysis was limited to physicians who were strongly confident in their ability to describe and apply the traditional international consensus definitions of sepsis, agreement remained poor. Importantly, these fictional vignettes were generally felt to be very realistic and representative of common clinical scenarios.
To our knowledge, this is the first study to examine variability in diagnosing sepsis by presenting identical cases to a group of intensivists. In an international qualitative survey of over 1000 physicians (including 529 intensivists) performed in 2000 by telephone interview, researchers found that less than 20 % of respondents gave a consistent definition of sepsis, with many physicians having the misperception that fever or hypotension must be present to diagnose sepsis [
14]. However, since that survey was done, there have been substantial advances in sepsis awareness due to international initiatives such as the Surviving Sepsis Campaign, the dissemination of evidence-based management guidelines for sepsis, the publication of many high-profile clinical studies, and the recent introduction of national mandates for sepsis care and public reporting [
15‐
17]. Our findings suggest that, even with the increased awareness and focus on sepsis in recent years, there is still a significant amount of variability in diagnosing sepsis among critical care physicians—the specialists who are generally felt to have the most expertise in caring for patients with sepsis.
Subjectivity in diagnosing sepsis is to be expected early in a patient’s clinical course, when symptoms are undifferentiated and diagnostic test results are still pending. However, in our study, we used a case vignette format in which the patient’s entire clinical course was presented. We nonetheless found substantial variability in how sepsis diagnoses were assigned. Although we did not explicitly test respondents’ knowledge of the sepsis definitions, in our analysis of free text explanations we found that variability was generally due to differences in interpreting whether infection or organ dysfunction was present or if organ dysfunction was attributable to infection, rather than to a lack of knowledge about the meaning of the sepsis definitions. Prior studies have suggested that adult and pediatric clinicians often disagree about sepsis diagnoses when compared with rigorous application of international consensus definitions; however, the researchers in these studies presumed that the consensus definitions themselves can be consistently applied [
18,
19]. Interestingly, even when dichotomizing responses into any sepsis category (sepsis, severe sepsis, or septic shock) or not, there was still substantial disagreement in our study, indicating that simply deciding whether a patient has an infection can be highly variable, even in retrospect. This is important when considering the new consensus clinical definitions for sepsis recently released by the Society of Critical Care Medicine (SCCM) and the European Society of Intensive Care Medicine (ESICM) [
20]. Although the terminology and criteria for organ dysfunction are being updated, this new definition still relies on clinicians’ judgement of whether infection is present, as well as whether organ dysfunction is attributable to infection.
Our findings have important implications for epidemiologic studies, public health surveillance, and quality reporting. Currently, the method proposed by the Centers for Medicare and Medicaid Services (CMS) for identifying cases of severe sepsis for reporting of sepsis bundle adherence is based on International Classification of Diseases, Tenth Revision, codes for sepsis, followed by chart review [
4]. However, relying on diagnoses and codes is problematic when it comes to identifying sepsis cases and outcomes, as our study demonstrates that there is wide variability between clinicians in how they diagnose sepsis. This complicates current initiatives to benchmark hospitals on their care of patients with sepsis, since there is no common standard, it seems, for what constitutes a “septic” patient. Using claims data for longitudinal surveillance of sepsis trends is also complicated by the fact that the ways in which clinicians and hospitals diagnose and code for sepsis are changing over time, likely in response to rising awareness of sepsis and changing reimbursement incentives [
3,
21‐
23]. Prior studies have shown that incorporating quality metrics and potential financial penalties for conditions where there is substantial room for subjectivity in diagnosis, such as ventilator-associated pneumonia, can lead to misleading decreases in disease incidence that better reflect stricter application of subjective diagnostic criteria rather than true reductions in the number of cases [
24]. One alternative approach to surveillance that has recently been proposed is to use an objective surveillance definition that relies on electronically ascertainable clinical markers of presumed infection (such as blood cultures and antibiotics) and concurrent organ dysfunction (such as vasopressors, mechanical ventilation, and standardized changes in baseline laboratory values) rather than subjective and variable diagnoses and claim codes [
23]. This approach will increase objectivity and reproducibility, although it does not solve the problem of knowing with certainty whether a patient is infected and whether concurrent organ dysfunction is attributable to infection.
While our study suggests substantial interobserver variability in diagnosing sepsis, it is important to note that several research studies have shown reasonable to good agreement (with κ statistics in the 0.6–0.8 range) among physicians using chart reviews as a gold standard for identifying severe sepsis [
23,
25,
26]. However, the raters in these studies were formally trained using shared sets of cases, used standardized abstraction tools, and deliberately attempted to reconcile discrepant results. Intensive training, standardized abstraction tools, and formal reconciliation conferences are not part of routine clinical or coding practices, and hence the lower levels of agreement we observed in the present study may be more representative of real-world practice than the high levels of agreement reported in research studies. In addition, agreement about whether sepsis is present is likely to be higher in “sentinel” cases. For example, one prior study showed that greater severity of illness, ICU admission, bacteremia, elevated lactate, and shock were associated with greater consistency in the diagnosis of severe sepsis [
27]. However, clear-cut cases with severe illness and unambiguous infection—similar to the control case we used in our study—represent only a small subset of sepsis cases.
Our study has important limitations. First, the response rate of our survey was relatively low, and we were unable to compare the characteristics of physicians who were contacted but did not respond. However, if anything, we would expect that physicians who completed the survey might have a greater degree of interest in (and knowledge about) sepsis than nonrespondents. It is thus even possible that this could have biased our results to overestimate agreement. Second, our survey was heavily weighted toward academic physicians in the northeastern United States, limiting the generalizability of our findings. Third, it is possible that overall agreement would be better in a large, random sample of actual patients with suspected infection. However, respondents in the present study generally felt that the study vignettes were both realistic and representative of actual patients, underscoring the fact that ambiguous cases of sepsis are likely fairly common. Fourth, our survey was conducted before the release of the new SCCM/ESICM consensus definition of sepsis, which may have performance characteristics in terms of interobserver variability that are different from those of the prior sepsis definition set. However, the new definition uses the same framework of seeking patients with acute organ dysfunction attributable to infection, and hence subjectivity in assigning sepsis diagnoses will likely persist. Furthermore, it may take time for these new definitions to gain full acceptance in the medical community, while the traditional definitions will still be used for the foreseeable future as part of the CMS quality metrics.
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
CR designed the study and survey, disseminated the survey, analyzed the data, and drafted the manuscript. MK contributed to the study and survey design, data interpretation, and critical revision of the manuscript for intellectual content. SK, AFS, RLD, GL, AFM, and BTK contributed to the study and survey design, survey dissemination, and critical revision of the manuscript for intellectual content. All authors read and approved the final manuscript.