Development and validation of an automated emergency department-based syndromic surveillance system to enhance public health surveillance in Yukon: a lower-resourced and remote setting

The Territorial Epidemiologist collaborated with health information analysts at Yukon’s Whitehorse General Hospital (WGH), the only hospital in Whitehorse and Yukon’s primary hospital to access records from the newly implemented real-time emergency department electronic records collection tool: Meditech ED-Tracker system. This system comprises of an electronic medical records database that captures the following information at the time of patient assessment (1) demographic characteristics and date of visit (2) clinical notes (CN) (e.g., free-text describing a brief history of the stated complaint, the recorded temperature (in °C) at triage) (3) the chief complaint (CC) containing the Canadian Emergency Department Information System (CEDIS) code [40, 41] and (4) the physician discharge diagnosis (DD) which is a free-text field providing the diagnosis at patient discharge from the ED. Collaboration between the territorial epidemiologist and WGH stakeholders (the custodians of the data) and the completion of a privacy impact assessment (PIA) led to the establishment of a daily (1 day lag) secure file transfer of ED visit data to the Office of the Chief Medical Officer of Health (OCMOH) in November 2019. Historical data of all ED visits from January 1, 2018 onwards were also made available for use. However, data between January 1, 2018 and September 30, 2018 were deemed inconsistent (e.g., incomplete data and/or missing fields) and therefore excluded from further use. National Ambulatory Care Reporting System (NACRS) records were used as a reference in our validation process for the inclusion of International Classification of Disease, 10th edition (ICD-10) codes; NACRS data were thus extracted for the time period of October 1, 2018 to April 30, 2019 and merged with ED-Tracker data. Observations without a match between the two datasets were dropped. All analyses were performed using Stata LP 15.1 (Texas, USA).

Syndromes of interest were identified via consensus between Yukon’s Communicable Disease Control program (YCDC) manager, the office of the Chief Medical Officer of Health (OCMOH), and the territorial epidemiologist. A review of existing syndromic surveillance platforms (e.g., NC-Detect, ESSENCE-II) [42, 43] along with current literature were used to inform the terms used in the initial case definitions for each syndrome [3, 13, 18, 21, 22, 24‐29, 31, 32, 34, 44‐50]. These sources enabled us to create an initial list of key terms that have been used for monitoring of similar syndromes of interest. Terms were then further refined for local context by review from stakeholders at YCDC and the OCMOH. For example, in clinical triage notes, the nurse attendant may capture inebriation or alcohol abuse by using the short-hand term ‘EtOH’; several examples such as this were included in our NLP algorithms to ensure adequate capture of these types of locally adapted data collection terms. In addition to key terms, CEDIS codes related to each syndrome were also identified and included in case definitions. Additionally, a syndrome related to COVID-19 was developed following recommendations published by the World Health Organization (WHO) [51] and local consensus with the OCMOH. This case-finding definition was further developed through extensive internal review, a review of WHO’s COVID-19 weekly Situation Reports [52], and recommendations made available online by the Public Health Agency of Canada [53].

For each syndrome, individual terms and codes that were identified through literature review or expert consensus were organized under more broadly defined concepts that were then used to determine whether a patient record met the case definition for the syndrome of interest (see Supplement A1). Using these key terms and concepts we developed natural language processing algorithms in Stata and applied these to query ED-Tracker records from October 1, 2018 to April 30, 2019 as this dataset coincided with available NACRS data that could be used to for downstream validation purposes. Algorithms were constructed using regular expressions (i.e., the “regexm” function in Stata) to detect full or partial strings from the list of key terms and concepts in either the CN, CC, or DD fields. In general, algorithms were applied for each syndrome to query data fields using a forward-inclusion strategy, starting with DD, then querying the CC and CN fields to determine whether or not a case record met the case definition for any of the syndromes of interest (see Table 1).

Records with an exact match between ED-Tracker and NACRS datasets (i.e., visits occurring between October 1st, 2018 and April 30th, 2019) were used to validate the ED-SyS. Note: algorithms for detecting gastrointestinal and respiratory syndromes resulted in what was pragmatically considered too many records for manual review with current staff resources (e.g., > 600 records per syndrome), therefore an abbreviated dataset containing records from November 1, 2018 to January 31, 2019 was used instead for validating these two syndrome groups. Following the example set by [24], records flagged for meeting syndromic case definitions were assigned one of three validation classifications: likely related to the disease of interest (true-positive) = “0”; unclear/conflicting information (false positive) = “1”; does not match case definition/not related (false positive) = “2” (Table 1). These assignments were carried out first by reviewing available ICD-10 codes; any flagged case with an ICD-10 code equivalent to the those outlined in [54] was assigned a zero (“0”). This step was accomplished using the US-CDC’s database of equivalent code translation [55]. Remaining records were validated manually by two epidemiologists reviewing the ICD-10 code, DD, CC and CN fields, typically in that order. After an initial review, a random sample of 10 cases was taken from the “0”, “1”, and “2” sub-groups for each syndrome (30 cases total per syndrome) and reviewed by both epidemiologists to ensure agreement and consistency among classification. Discrepancies in scoring were discussed, and corrections were retroactively applied to the validated datasets so that consensus for all scores was achieved.

Following validation, we manually reviewed the validated records from each syndrome to identify: (1) misspellings or shorthand among the free-text fields that were not considered during the development of the initial algorithms; (2) terms and CEDIS codes that could be used to further refine the inclusion or exclusion of true- and false-positives, respectively; and (3) adjustments to the algorithm structure to improve case classification.

Each syndrome included in the ED-SyS was evaluated by measuring the positive predictive value (PPV) using the validated case records. The PPV was defined as the total number of true-positive records (classification = “0”) over the total number of records (classification = “0”, “1”, or “2”), as a percent. To evaluate the contribution of each data field on the overall performance of the ED-SyS, we measured the PPV from each component individually as well as in combination (e.g., CN; CC; DD; CN + CC; CN + CC + DD). Finally, to evaluate which terms were most influential in identifying each syndrome, the frequency of terms present among true-positive case records were measured and visually assessed using word clouds (NB: select terms for the construction of the word clouds were double-counted as some concept terms exist in both CN and DD algorithms).

An overview of the Yukon ED-SyS is presented in Fig. 1. In Yukon, the general Whitehorse population can access emergency and primary care services through the WGH-ED, with primary care services also accessible through community clinics. These are both access points where symptomatic individuals with infectious diseases of interest present for clinical assessment. Positive results from the WGH hospital Laboratory serve as the territory’s laboratory-based surveillance system for reportable infectious diseases. The ED-SyS described here complements the laboratory component of the system, providing a source for pre-diagnosis data in Yukon sent electronically to the OCMOH daily for analysis. Once an extract of ED-SyS data is received at the OCMOH, the Territorial Epidemiologist runs the NLP syndrome algorithm and inputs the syndrome aggregate counts into the US CDC’s Early Aberration Reporting System (EARS-X) to detect any statistically significant increases in counts that may warrant further investigation. This adaptive control chart detection tool uses three algorithms (C1, C2, and C2) based on a positive 1-sided CUSUM calculation to detect aberrant data. The C1 algorithm generates an alert if counts are 3 standard deviations above the moving 7-day baseline, C2 incorporates an additional 2-day window between the observed count and baseline, and C3 uses the sum of C2 and the previous 2 days [11, 56]. EARS-X was selected for use as it required minimal historical baselines, minimal training for use, and was readily accessible. Alerts generated are then reviewed by the Territorial Epidemiologist for any evidence of clustering by age, geography, gender, chief complaint, and discharge diagnosis. Evidence of clustering are forwarded along to YCDC for clinical assessment resulting in one of the following: (1) an investigation, (2) continued monitoring, or (3) no action.

In total, seven syndromes of interest were selected for daily monitoring: gastrointestinal illness (GI), influenza-like-illness (ILI), mumps, neurological infections (Neuro), rash, respiratory illnesses (Resp), and COVID-19. For each syndrome, diseases/conditions of interest were identified, and case definitions were developed based on algorithms that query information from the WGH ED-Tracker database (see Table 1). A complete list of terms used to query each syndrome is available in supplemental material (Supplement A1).

Results from our validation are presented in Table 2. Among our initial case definitions, DD consistently returned the highest proportion of true positive cases (PPV: 51.3 to 100%), compared to CN (PPV: 22.8 to 86.1%) and CC (PPV: 0 to 35.0%) when used individually. In general, the CN field produced the most sensitive results, flagging the highest number of visits for all syndromes, while CC and DD fields provided a lower number of hits with higher specificity. These trends were maintained after adjustment to the initial case definitions, with the largest improvements observed in the GI syndrome (e.g., PPV for DD improved from 51.3 to 86.9%).

Combining multiple levels of data from ED-Tracker produced an average of 6.7-fold more hits to each syndrome than querying each component individually (Table 2). The largest example of this was observed when querying for visits related to the Resp Syndrome; using CC alongside CN increased hits from our original case definitions over 33-fold. Using our final case definitions for the combined fields produced the largest improvements to the GI Syndrome, with the PPV for the CN/CC combination increasing to 44.5% from 22.3% and the CN/CC/DD combination increasing to 78.8% from 48.8%. Changes observed among the other syndromes were minimal, save for the PPV of detecting the Mumps Syndrome, which increased from 50.9 to 94.1%, although the total number of observed hits decreased from 57 to 17.

After an initial review of the ED-SyS performance using our validation dataset, several adjustments were made to the terms and algorithms used in our ED-SyS to improve the system’s performance within the local context. In general, three areas were useful in redefining ED-SyS queries to provide more accurate results in the Yukon setting: (1) Misspellings or shorthand among the free-text fields that were not considered during the development of the initial algorithms. For example, the acronym “LWBS” in the DD field was often used to indicate a patient had “left without being seen”, and additional acronyms were needed to describe the diarrhea, nausea, and vomit concepts within the GI syndrome case definition including “N&V” (nausea and vomiting), “N, V, D” (nausea, vomiting, diarrhea), and “V/D” (vomiting/diarrhea). (2) Terms and CEDIS code that could be used to further refine the inclusion or exclusion of true- and false-positives, respectively. For example, additional negating terms were identified for several syndromes; the inclusion of “no”, “denies” and “–” were therefore added to indicate negation within the diarrhea, nausea, vomit, fever, and cough concepts. Several CEDIS codes were consistently present among true- and false-positive results for each syndrome; we used these codes to help provide additional specificity to our inclusion and exclusion criteria based on local context (Table 3). For example, a high proportion of visits flagged by our GI case definition presented with abdominal pain (CEDIS 251), but were validated as unclear/conflicting information (n = 91/571, classification = “1”) or unrelated (n = 365/571, classification = “2”). For this reason, we elected to remove CEDIS 251 from the final GI syndrome case definition in addition to several related terms that were present in the original syndromic case definition at the CN and DD level (e.g., “ascites”, “LUQ”, “LLQ”, “RLQ”, “RUQ”, “diverticulitis”, “appendicitis”, “abdominal bloating”, and “flatus”) that were also associated with false-positive records. (3) Adjustments to the algorithm logic that could improve the application of our algorithms. For example, the original algorithm for detecting the Neuro syndrome queried the CN and CC fields only if the information from the DD field was blank. We modified this process to query CN and CC when the DD field contained at least one of the terms “febrile seizure,” “headache nyd,” “malaise nyd”, or “sepsis” without mention of the term “mening”. This improved both the sensitivity and specificity of the algorithm, increasing the number of hits from six to 19, while concomitantly increasing PPV from 83.3 to 89.5%.

After completing the above adjustments to improve case-finding, we visualized the most influential terms among the true-positive case records for each syndrome (Fig. 2). Each syndrome appeared to have several terms that were essential to identifying the syndrome of interest. In three syndromes (Rash, Resp, and ILI) CEDIS codes appeared among the most frequent terms.