Validity of administrative data in recording sepsis: a systematic review

In this review, we identified and summarized the published literature evaluating and validating ICD-9 and ICD-10 codes used to identify sepsis in administrative databases. We identified 12 studies that met all eligibility criteria for this systematic review and found large variations in terms of the scope of ICD codes used and the estimates of validity among studies. All studies validated inpatient data, and the majority of the studies showed that ICD codes defining a diagnosis of sepsis in administrative data are highly specific but lack Sn. In 10 of the 12 studies, Sn was low (<53%), even in cases of altering study characteristics [20]. A reasonable conclusion is that sepsis is largely undercoded in administrative data using ICD-9 or ICD-10 coded case definitions, regardless of study characteristics. However, the high Sp and NPV do mean that few false-positives would be present in such a dataset.

The heterogeneity seen among the studies in coding accuracy, especially with respect to Sn and PPV, may be due to multiple factors, including the number of codes used, the version of ICD used, the sample population, the reference standard comparison used and the type of administrative data. For instance Gedeborg et al. [20] applied the same ICD-9 and ICD-10 coding algorithms to different patient populations, including ICU patients with community-acquired sepsis and infectious disease department patients, and tested these against two different reference standard definitions (sepsis clinical trial patients and patients from an ICU-specific coded database). They showed the data accuracy to have large variations that were dependent on the patient population being studied and reference standard used. Not surprisingly, limiting the sample population to one in which an infectious disease service was consulted during the patient stay actually decreased the Sn by 28.5% while only increasing the Sp by 1.9%. It has also been reported that severe sepsis is poorly documented outside the ICU, although in one study sepsis was commonly found on non-ICU medical wards [31], suggesting that the accuracy of diagnostic codes may be substantially impacted, depending on the population selected or the criteria used to define the population.

Validity is also dependent on diagnostic coding field location (primary or secondary or all). Cevasco et al. [19] examined a population-based inpatient database but restricted the sepsis diagnostic code to a secondary coding field position in two separate populations, resulting in lower PPV values (43% for Veterans Affairs patients and 51% for community hospital patients). Grijalva et al. [21] restricted the population to a highly specific patient sample (rheumatoid arthritis patients) and examined only five ICD-9-CM codes; however, they allowed the coding field position to be either primary or secondary, which resulted in a PPV of 80%. Gedeborg et al. [20] performed multiple comparisons using primary or both primary and secondary code field positions. They reported consistently high Sn estimates when both the primary and secondary coding field positions were included. The primary coding field is normally designated for the condition that contributed the most to a patient’s length of stay or was the main reason for admission (depending on country). Thus, sicker patients presenting with severe sepsis or septic shock are more likely to be captured using the primary diagnosis alone. A further limitation of severity level coding is reflected in the organ dysfunction codes used to identify severe sepsis, as these diagnostic codes would most likely be recorded in the secondary code field positions. In none of the studies were any particular organ dysfunction codes validated or the coding field positions examined.

The variation in diagnosing sepsis alone translates to variable recording of the diagnosis in the medical record. O’Malley et al. [32] described the patient trajectory from admission to discharge and the process of recording the admitting diagnosis to the assignment of an ICD code post-discharge. A suggested error when a physician records a diagnosis in the medical record is based on the variance across terms and language used to describe the disease and/or reporting of an infection without concomitant reporting of systemic inflammation or associated organ dysfunction. Peoze et al. [33] examined how a physician’s awareness and attitude towards the diagnosis of sepsis impacted the recording of sepsis. They reported that 46% of the time in the case of sepsis, the cause of death was incorrectly recorded as due to another disease. Assunção et al. [34] found that sepsis was most frequently misdiagnosed, up to 66.5% of the time, as infection without clinical and laboratory signs of inflammatory response. Therefore, low case capture of sepsis may also be due to the capacity of practicing physicians to recognize and report clinical cases of systemic inflammatory response syndrome, sepsis, severe sepsis and septic shock in the medical record. No study examined the expertise of the coders or the impact of physician documentation on the selected codes.

The results of this systematic review should raise a question about whether reliable research on sepsis can be performed using administrative data. On the basis of the findings of our review, hospital discharge abstract data alone are an insufficient source for researchers to examine sepsis incidence accurately or for surveillance. However, administrative data and ICD coding algorithms could still be used to examine risk factors for the development of sepsis or outcomes. In these studies, a high Sp with a reduced Sn may suffice to minimize the number of false-positive cases, with a caveat being a limitation that these studies may include a subset of more easily defined and/or recognized cases or a more severe form of sepsis.

The complexity which makes up the clinical entity of sepsis has led to a significant effort over the past 20-plus years to standardize clinical and laboratory diagnostic criteria and definitions [10,30,35,36]. Although designed primarily for clinical use, these definitions have led to practical applications for other research, including health care quality and utilization improvement initiatives and surveillance. Particularly for surveillance, one of the purposes is to monitor disease prevalence over a span of years and forecast future trends. Thus, the trend is related to the stability of the data validity in the observation period, regardless of the level of validity. That said, administrative data are still an invaluable resource to monitor sepsis, although it does not capture the same amount of clinical detail that an Electronic Medical Record (EMR) does. Other advantages, such as wide geographical coverage, a population-based capture of nearly every contact with the health care system and the overall cost-effectiveness [6], make administrative data a lucrative source of health information. Administrative data cannot replicate the complex myriad of the clinical criteria comprising sepsis; therefore, translating this clinical definition into coded data and evaluating the validity of the coding of sepsis in administrative data are crucial.

Although a desired definition with Sn and Sp of 100% would be ideal, modifying and optimizing the data definition to capture sepsis as accurately as possible, with Sn falling above 75%, similar to that of other hospital-acquired infections internationally [37] and for non-communicable diseases such as hypertension [38] and diabetes [39], should be the ultimate goal. Improving the quality of administrative health data and increasing the case capture and validity of sepsis could be accomplished through a number of simple strategies, such as (1) improved physician documentation, including documenting sepsis in the front pages of the chart to get the attention of coders; (2) having a specialized coding procedure for ICU patients, perhaps including specific training of health care coders to improve familiarity with the case mix of patients and conditions that are more prevalent in the ICU to increase Sn and case capture; and (3) for those countries in which a limited number of diagnostic coding fields exist, there should be at least eight coding fields for diagnosis to capture conditions such as sepsis [40]. These strategies can be used in combination with data linkage to other data sources such as laboratory, pharmacy or microbiology data and the EMRs, and with clinical factors such as heart rate, respiratory rate, body temperature, white blood cell count and markers of organ dysfunction, to try to incorporate the key characteristics of sepsis defined and listed in the ACCP/SCCM definitions [30]. Both improving the definition of sepsis and making it comparable across national and international jurisdictions is of the utmost importance to continue improving the understanding of how quality of sepsis care is impacting the incidence and outcomes of the disease.

There are limitations to this systematic review. The search strategy was limited to only studies published in English, and a grey literature search was not conducted. The target of the study was ICD codes used for sepsis specifically. Because sepsis itself is difficult to diagnose and has a range of clinical presentations, there is a possibility that validation studies examining only these other conditions and not sepsis specifically may have been missed. Publication bias in validation studies may also be a concern, as authors may report only better-performing case definitions and may not publish less well-performing case definitions with very low diagnostic accuracy. However, our systematic review included studies with very low values for case definitions, and therefore there is little concern that publication bias has occurred.