Modified cardiovascular SOFA score in sepsis: development and internal and external validation

Sepsis is defined as life-threatening organ dysfunction caused by dysregulated host responses to infection [1]. Worldwide, sepsis has a high incidence, morbidity, and mortality and represents a major public health problem [2, 3]. Given this background, the WHO has announced sepsis as a global health priority [4].

The Sequential Organ Failure Assessment (SOFA) score was developed in 1996 [5], and this score is now extensively used in critically ill patients. Moreover, the development of a new sepsis definition, which adopts SOFA score as a main diagnostic tool, has broadened the score’s application [1]. However, the cardiovascular SOFA score has critical limitations. When first developed, the guideline recommended the use of dopamine as the first-line vasopressor in septic shock [6, 7]; but, in 2008, this first-line vasopressor recommendation was changed to norepinephrine. This use of norepinephrine has become standard management [8].

Sepsis-3 defines septic shock as a subset of sepsis with circulatory dysfunction and cellular metabolic abnormality which can be estimated by hyperlactatemia [1]. Because an elevated lactate level is reflective of tissue hypoxia caused by insufficient tissue oxygen delivery and impaired aerobic respiration, lactate is an essential biomarker in sepsis [9].

Considering the importance of the SOFA score, we propose that the SOFA score be modified to reflect the current clinical practice patterns for vasopressor use and the diagnostic importance of lactate level. Our proposed modified SOFA scoring system is based on data from multiple cohorts. We developed and internally and externally validated our modified SOFA scoring system, and we compared this system with the original SOFA scoring system in terms of predictive validity.

In this study, we demonstrated that the modified CV SOFA score reflecting the current sepsis guidelines could be more useful both in prognostication for sepsis and detection of sepsis at risk. These current guidelines include the use of norepinephrine as vasopressor of choice and the use of lactate level as an important tissue perfusion biomarker.

The SOFA score was created by the Working Group of the European Society of Intensive Care Medicine. The SOFA score aimed to describe as quantitatively and objectively as possible the degree of organ dysfunction/failure in sepsis patients [5]. Recently, the SOFA score has been advocated and adopted as means of identifying sepsis among patients with suspected infection in 2016 [10]. The new definition is important in research, performance monitoring, and accreditation [23]. However, the CV SOFA score has a critical issue in terms of the use of vasopressors. The SOFA score was introduced in 1996 when dopamine was the drug of choice as vasopressor in sepsis [6, 7]. Thereafter, dopamine was used in the CV SOFA score. However, in 2008, norepinephrine replaced dopamine as the first-line vasopressor in sepsis [8]. This changed clinical practice, but the change was not accounted for in the CV SOFA score. Reflecting this, in our 3 cohorts, there were few cases with CV SOFA score of 2, which is defined as use of dopamine less than 5 µg/kg/min or any dose of dobutamine, and this is consistent with recent studies [24, 25]. Even when equivalent dose of norepinephrine has been used to overcome this, original CV SOFA score performance is not good. Therefore, modification of the CV SOFA scoring system is urgently needed and provides the motivation behind this study.

Our modified SOFA score model showed significantly improved mortality-discriminant power than the original SOFA score in the suspected infection, sepsis, and septic shock cohorts. In previous studies, the mortality rate of each SOFA score did not show incremental tendency [26‐30]. In this study, the same findings were detected in all three independent cohorts. However, the newly-developed modified SOFA score showed a more incremental tendency. In addition, the modified SOFA model can detect more patients at risk of septic mortality than the original SOFA score. Moreover, in the suspected infection cohort, the modified score showed high agreement with the current SOFA score (Cohen’s kappa, 0.916), implying that this modified SOFA could have clinical applications.

We decided that lactate level should be included in the modified CV SOFA score with the presence of pre-existing hypotension and the use/dosage of vasopressor in the original CV SOFA score. Lactate has been extensively investigated as a good biomarker for tissue perfusion, and lactate level is widely used in sepsis [31]. In the Sepsis-3 definition study, lactate level was identified for testing in cohort studies by the Delphi consensus, and lactate level was included in the definition of septic shock [1]. Lactate level was also proposed as a screening tool for sepsis or septic shock, but this level was not included in the final quick SOFA. The group extensively investigated the usefulness of lactate level and found that 1 added point to qSOFA score for elevated serum lactate level 2.0 mmol/L or more significantly increased predictive validity of qSOFA [10]. However, the group designated lactate level’s inclusion in the quick SOFA as an “area of further inquiry”. The group proposed that lactate levels could be used for patients with borderline qSOFA values or could substitute for individual qSOFA variables in healthy systems in which lactate levels are reliably measured at low cost and in a timely manner. Interestingly, the group did not investigate the value that lactate addition could have with SOFA score. We used the various cut-off levels of lactate used in previous investigations [10] in our derivation and validation models. We ultimately decided on the cut-off level as 2, which has been used in the new septic shock definition, to be included in the modified CV SOFA score [1, 19].

We determined multiple cut-off points of vasopressor dose referring to both “a priori” and “data-driven” optimal values [32, 33]. We incorporated these into the model and decided on two cut-off points regarding incidence and mortality rate according to the CV/total SOFA score, discrimination, and calibration. We could not be confident that these cut-off values are consistently valid in other cohorts, leaving generalizability concerns. We did not include the use and dose of arginine vasopressin as independent scoring variables. Given that vasopressin and its analogs are commonly used in clinical practice for the management of sepsis [8], the modified CV SOFA score could be more accurate if their use were included. However, the limited score of 0 to 4 on the CV SOFA becomes too complex when too many variables are added. Instead, a conversion table for vasopressor doses might be used [34].

In the modified CV SOFA score, the use of NE was included from the score of 1, rather than the score of 2 in the original CV SOFA score. Recently, the beneficial effect of early use of NE in septic shock has been investigated [35] and has led to the early use of this vasopressor in current clinical practice [36]. Therefore, we decided that a modified CV score of 1 should include the use of small doses of NE.

We tried to modify cardiovascular SOFA with blood pressure and various cut-offs of vasopressors, but their performances were not better than those with the original cardiovascular SOFA model. The AUROC of the vasopressor only model were 0.64 at best, which as included in Table 3. Therefore, we included lactate in the modified SOFA model and found that the discriminative performances were significantly improved (range 0.648–0.716) than that of the original CV SOFA (range 0.557–0.638) and that of vasopressor only model (range 0.610–0.640). We, therefore, decided to include lactate value in the model since lactate has been investigated as significant mortality-associated factor, independently with blood pressure or vasopressor use [12, 37, 38]. Lactate was also included in the Sepsis-3 definition [1].

The performance of the modified total SOFA score could be considered modestly increased in clinical aspect. However, the modified CV SOFA performance seems to be significant in the clinical aspect. SOFA score has 6 sub-categories and a change of one category might have a modest increase in total SOFA scores.

The discriminatory power of the SOFA in our three cohorts was similar to that in previous studies [10, 26, 39], implying the reliability of our cohorts.

We modified the SOFA score to detect sepsis. Even though the SOFA score is used to detect sepsis, it is not limited to septic patients, and this inherent limitation of the SOFA score should be considered in further study.

With this study, we could not propose the global use of our modified CV SOFA score, but this study offers a good starting point for SOFA score modification. Modification is necessary to reflect current guidelines regarding the clinical use of vasopressors and diagnostic use of lactate levels in sepsis patients.

There are some limitations to this study. First, all cohorts were derived from emergency departments, and validation with ICU data is required. Second, the characteristics of the three cohorts are different. The sepsis cohorts were collected in accordance with either the Sepsis-2 or Sepsis-3 definition depending on the period. The septic shock cohort was collected in accordance with the sepsis-3 definition, and the suspected infection cohort included patients in whom antibiotics and blood culture were administered. However, this could be a study strength because the newly-modified SOFA score could be applied to differently defined cohorts, meaning more generalizability. Third, these cohorts are all from a single country and all from university-based hospitals. Multinational and multi-level center validation is necessary. Fourth, the purpose of the three cohorts used in this study was not to develop the new CV SOFA score. Fifth, we did not develop an entirely new scoring system that is usually performed according to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) recommendations [40]. The SOFA score-based definition of sepsis has been widely adopted, so entirely changing the CV SOFA scoring system would not be useful at this time. Therefore, we intended to change the CV SOFA scoring system as minimally as possible. Supporting this, the agreement between the original SOFA score and the modified SOFA score was excellent. Sixth, we postulated a baseline SOFA score of 0, but in clinical practice, this is an inevitable limitation. Seventh, the cohort we used in derivation and internal validation is from a single center. However, we tested 16 candidate models in two other large cohorts (external validation cohorts); and model 3 performed better than the other models in terms of incidence, mortality rate, discrimination, and calibration (Additional file 1: Figs. S8-10). Eighth, we used the initial lactate level in the modified CV model. Even though lactate is widely used in sepsis, there are some controversies about the role of spot lactate (initial lactate level) in sepsis. We could not find any review or meta-analysis study about the role of the initial level of lactate in sepsis, which could be conclusive on the utility of lactate. Also, the changes in lactate levels over time are relatively slow, so the patients still have a lactate level above the normal range even after they were resuscitated [41]. This concept could be a major obstacle to include lactate in the modified SOFA score. This needs further evaluation with larger and multi-national cohorts. Ninth, we developed and tested modify SOFA model with mortality as a primary outcome. Even though we did not perform this study to propose modified prognostic scoring systems but modify the SOFA score as a tool to detect sepsis, we tactically used mortality as a primary outcome to investigate modified models following the method of developing the Sepsis-3 definition. Lastly, there were some missing data in all three cohorts. However, the missing data rate was low in most cases, and the results of complete analysis among patients without missing data were nearly identical, implying minimal effects of missing data on the primary analysis (Additional file 1: Fig. S7).