Syndromic Surveillance of Mental and Substance Use Disorders: A Validation Study Using Emergency Department Chief Complaints

There is increasing recognition that public health surveillance activities should include surveillance of mental and substance use disorders (1,2). Such disorders—for example, depression, anxiety, and alcohol and drug abuse—affect approximately half of all Americans at some point during their lifetime (3) and as many as 25% of adults during any given year (4). They also account for growing portions of the U.S. population’s total burden of disability (5,6) and economic costs (7,8). Ongoing surveillance of mental and substance use disorders is key to assessing community mental and behavioral health needs and to monitoring long-term trends in the disorders’ occurrence (2). Such surveillance is also important for documenting shorter-term trends, given that the occurrence of and care seeking for mental and substance use disorders can fluctuate rapidly in response to community stressors, such as a natural disaster (9,10), economic downturn (11,12), and terrorist attack (13,14), and to changes in health care practices (15).

Although most current efforts to conduct surveillance of mental and substance use disorders take place at the national or state level (1), such surveillance may be most useful at the local level. County and city agencies are typically responsible for administering and evaluating mental health services and for responding to psychiatric crisis situations. However, surveillance of mental and substance use disorders at local levels presents several challenges. First, although conducting population-representative health surveys is the most rigorous approach to ascertaining burden of prevalent mental health problems (16,17), the resources required for such surveys are prohibitive for most local jurisdictions. Second, it can be difficult to establish case definitions of mental and substance use disorders that are usable in surveillance contexts because of evolving psychiatric definitions and interpractitioner variability in diagnosis (2). Third, data on mental and substance use disorders at the local level are rarely available on a timely basis. For example, frequency counts of patients obtaining care from a county department of behavioral health may be reported just once per year, with little opportunity to monitor or respond to shorter-term fluctuations (18).

Syndromic surveillance of presentations to emergency departments (EDs) that are related to mental and substance use disorders may be one promising approach to addressing these challenges. Syndromic surveillance is a widely used, though underrecognized, functionality of many local public health agencies in the United States. In their most typical structure, syndromic surveillance systems utilize specialized software that automatically collects, processes, and transmits prediagnostic patient data (chief complaints and basic demographic characteristics) in real time from local EDs to a city or county public health department (19,20). A crucial aspect of this process is the automatic classification of each ED patient’s chief complaint into one or more syndromes—such as respiratory, neurological, rash, or injury—based on keyword searches of the free-text chief complaints. Such data have primarily been used to identify emerging outbreaks of infectious disease, and no research has used syndromic surveillance system data to track mental and substance use disorders. However, it has been proposed that syndromic surveillance system data might also provide a useful indicator of clinically relevant occurrences or exacerbations of mental and substance use disorders (2).

To address this question, this study made use of data from Fresno County, California, to investigate whether chief complaints and their associated syndrome classifications can be used to validly and reliably ascertain ED visits for mental and substance use disorders. In 2008, the Fresno County Department of Public Health (Fresno DPH) launched a syndromic surveillance program that collects visit data from local emergency departments that are recorded prior to diagnosis. The visit data include chief complaints that are then automatically classified into syndromes. In January 2012, this data stream began incorporating ICD-9-CM (21) discharge diagnoses of ED patients who were evaluated by a clinician. This change provided the opportunity to assess the validity of using chief complaints as the basis of syndrome classification of mental and substance use disorders by using the associated discharge diagnosis as the criterion standard. ICD-9-CM diagnosis patterns and overall ED utilization among patients who received a syndrome classification related to mental and substance use disorders were also examined.

Methods

Data Source

Data for the study were obtained from the Fresno DPH syndromic surveillance system. This system utilizes the EpiCenter software package developed by Health Monitoring Systems (HMS). EpiCenter electronically captures ED visit data elements collected during routine administrative and patient care activities at linked hospital facilities and securely transmits those data to HMS in real time for preprocessing. This preprocessing includes automatic classification of patient chief complaints into one or more syndrome classifications, developed by HMS. The preprocessed data are then transmitted to Fresno DPH for further analysis and investigation.

The analytic sample for this study comprised all ED visits by Fresno residents to participating facilities from January 1 to December 31, 2013. During this period, two of the county’s four acute care hospitals participated in the syndromic surveillance system; these two facilities together accounted for approximately 170,000 ED visits, or 60% of annual countywide visits. The larger of the facilities is the region’s largest health care provider as well as the sole level 1 trauma center and safety-net hospital in the county.

ED visit data used for this study included patients’ age and sex, visit disposition, chief complaint at registration, up to seven automated syndrome classifications, and up to 10 patient discharge diagnoses. Discharge diagnoses typically become available in the syndromic surveillance software system at the time of patient discharge, but this process can be delayed or obstructed by billing procedures, electronic transmission errors, or other problems. Therefore, not all ED visits in the study database had associated discharge diagnoses.

The Fresno syndromic surveillance system captures encrypted unique patient identifiers that allow patient tracking between various EDs within Fresno County. Because the primary goal was to examine overall ED use, in the main analyses all ED visits were counted as independent. However, the analysis also examined ED utilization patterns for the subset of unique patients who made at least one visit for a mental disorder during the study period. Duplicate visits for the same date, time, and unique patient identifier were deleted from the analytic sample.

Measures

Syndrome classification.

ED visits were defined as having a mental health/substance abuse (MHSA) syndrome classification if the automated syndrome classification of “mental disorder” was present in any of the seven syndrome positions. EpiCenter’s “mental disorder” syndrome classification uses keywords found in the free-text chief complaints describing the patient’s main symptoms. Relevant keywords include “sad,” “anxiety,” “depression,” “chemical dependency,” and “attempted suicide”; also included are common misspellings (such as “anxeity” and “anxity”) and abbreviations (“chem dp”). A sample of the chief complaints classified under the MHSA syndrome category, along with their primary syndrome classifications, is shown in Table 1.

Discharge diagnosis.

Following standard practice in emergency medicine research (22,23), ED visits were defined as having an MHSA discharge diagnosis if an ICD-9-CM diagnosis of 290.0–316.x was present in the principal or the first two secondary discharge diagnosis positions.

Visit variables.

Patient age was examined as a continuous variable. Visit disposition was categorized as discharged home, left against medical advice/without being seen by a clinician, transferred to psychiatric/rehabilitation facility, transferred to other facility, transferred to law enforcement, or other. Approximately one in five ED visits (20.8%) were missing disposition data; on the recommendation of the EpiCenter developers and Fresno DPH staff, these were assumed to reflect patients who left the ED without being seen.

Statistical Analyses

Descriptive statistics were calculated to characterize the ED visits in the entire study sample. Statistical calculations for the study’s central analyses used an analytic sample restricted to ED visits with a valid discharge diagnosis. Kappa statistics were used to assess overall agreement between MHSA syndrome classifications and MHSA discharge diagnoses. Sensitivity, specificity, positive predictive value, and negative predictive value of the MHSA syndrome classification measure were also calculated, with the MHSA discharge diagnoses used as the reference standard. All analyses were conducted by using Stata, version 14.1 (StataCorp LP). The institutional review boards of the University of California–Merced and the Fresno DPH approved this study.

Results

During the 12-month study period, there were a total of 146,315 ED visits (mean±SD=12,193±461.7 visits per month) to the participating Fresno facilities. Of these visits, 11,941 (8.2%) had an MHSA syndrome classification. Nearly three-quarters of all ED visits during the study period (N=104,513; 71.4%) had a valid discharge diagnosis. Most (95.7%) visits without a valid discharge diagnosis had a disposition indicating the patient had left without being seen. Of the visits with a valid discharge diagnosis, 9,669 (9.3%) had an MHSA discharge diagnosis. Figure 1 shows the monthly percentage of all ED visits with an MHSA syndrome classification and ED visits with an MHSA discharge diagnosis during the study period.

Descriptive statistics for the overall sample of ED visits, for ED visits with an MHSA syndrome classification, and for ED visits with an MHSA discharge diagnosis are shown in Table 2. Patient and visit characteristics for the subsamples of visits with an MHSA syndrome classification and with an MHSA discharge diagnosis (hereafter referred to jointly as “MHSA-related visits”) were very similar. Compared with other patients, those with MHSA-related visits were significantly more likely to be male (p<.001) and were significantly younger (p<.05), according to chi-square tests. The visit dispositions of patients with MHSA-related visits also differed. Whereas 18.1% of patients with an MHSA syndrome classification and 17.4% of patients with an MHSA discharge diagnosis were transferred upon discharge from the ED to a psychiatric or rehabilitation facility, less than 1% of other patients were transferred to such facilities (odds ratio=37.7 and 24.1, respectively; p<.001 for both).

Concordance statistics between MHSA syndrome classifications and MHSA discharge diagnoses for the 104,513 visits with a valid discharge diagnosis are presented in Table 3. Overall agreement between the MHSA syndrome classification and discharge diagnoses was very high (κ=.92, 95% confidence interval [CI]=.91–.92); there were only 1,509 cases in which syndrome classification and discharge diagnosis were discordant. All visits receiving an MHSA diagnosis (N=9,668) also received an MHSA syndrome classification (sensitivity=100%; CI=99.9%−100.0%). Of the 94,845 visits receiving a diagnosis unrelated to an MHSA, 93,336 received a non-MHSA syndrome classification (specificity=98.4%; CI=98.3%−98.5%). The positive predictive value of the MHSA syndrome classification, indicating the proportion of syndrome-classified patients who ultimately received an MHSA discharge diagnosis from a clinician, was 86.5% (CI=85.9%−87.1%). The syndrome classification’s negative predictive value was 100% (CI=99.9%−100.0%), indicating that all patients with other kinds of chief complaints ultimately received a discharge diagnosis unrelated to an MHSA.

The most common ICD-9-CM principal diagnosis codes among ED visits with an MHSA syndrome classification were 305.x (nondependent drug or alcohol abuse, 13.9% of visits), 300 (anxiety states, 13.8%), 298.9 (psychosis, 8.4%), and 296.2 (major depressive episode, 5.1%). The most common ICD-9-CM code for secondary and tertiary diagnoses among these visits was V62.84 (suicidal ideation).

Data on the 84,526 unique patients in the sample were used to examine ED utilization patterns among Fresno patients who presented with at least one MHSA-related visit in 2013. Patients who ever presented with an MHSA syndrome classification (N=7,469) or an MHSA discharge diagnosis (N=6,324) made significantly more ED visits in 2013 compared with other patients. The mean number of total ED visits for patients with a visit with an MHSA syndrome classification (3.38±4.99) and patients with a visit with an MHSA discharge diagnosis (3.53±5.23) were significantly higher than for patients with no MHSA-related visit (1.57±1.56) (t=–70.85 and –70.46, respectively; df=84,524 and p<.001 for both).

Discussion

To our knowledge, this study is the first to assess the utility of emergency department–based syndromic surveillance data for public health surveillance of mental and substance use disorders. The findings reveal that the chief complaints associated with ED visits can be utilized to reliably and validly ascertain the incidence of patient presentations for mental and substance use disorders. Notably, there was near-perfect agreement between the MHSA syndrome classification and the group of ICD-9-CM discharge diagnoses indicating psychiatric or substance use disorders.

Demonstrating high concordance between syndrome classifications based on chief complaint and ICD discharge diagnoses, as do these analyses, is an important step in public mental health surveillance efforts. Discharge diagnoses are frequently not available in syndromic surveillance system data. Some software systems do not currently collect this information; in those that do, diagnoses may be missing for a large subset of ED visits, for example, those that involve patients who leave without being seen or those that involve delays in billing and transmission of the final diagnosis or patient admissions from EDs to inpatient facilities with different electronic health record systems. Therefore, diagnosis codes cannot be relied upon for comprehensive, real-time monitoring of MHSA-related (or other) ED visits. Syndrome-classified chief complaints—which are universally and promptly collected by ED-based syndromic surveillance systems—thus offer a valuable alternative.

The study findings also suggest high levels of health care burden and morbidity among patients whose visit was associated with an MHSA syndrome classification. These patients were far more likely than other patients to be transferred to a psychiatric, rehabilitation, or inpatient facility (overall transfer rates of 27% versus 6%), and they were more frequent utilizers of the Fresno County ED system. This study’s findings lend support to the argument that patients with visits receiving an MHSA syndrome classification constitute a population with genuinely impaired functioning and service needs, not simply elevated levels of distress. However, future research should correlate trends detected by syndromic surveillance of MHSA-related visits with additional claims or behavioral health systems data to better understand service utilization among these high-acuity cases.

This study had several limitations. First, data quality depended on participating providers capturing maximally complete and valid patient information. As with most syndromic surveillance data sets, discharge diagnoses were not available for all ED visits, which could have affected calculations of the agreement between visits with MHSA syndrome classification and discharge diagnoses if there were substantial differences in the presenting complaints of patients who did and did not receive a diagnosis (24). However, of the visits with a missing discharge diagnosis, only 1.8% had an MHSA syndrome classification. Given high levels of agreement between the MHSA syndrome classification and discharge diagnoses, any bias in the results is likely to be small in magnitude (25). Second, although the algorithm used in this study to determine MHSA syndrome classification has already been commercially developed and is immediately applicable to other public health agencies using the EpiCenter system, it cannot be assumed that the results are generalizable to other systems or geographic areas (26).

Third, the EpiCenter “mental disorder” syndrome classification does not distinguish between mental disorder–related (such as psychosis) and substance-related (such as drug overdose) ED presentations; instead, both types of presentation can be included in this single category. The development of new syndrome classification algorithms that more precisely differentiate between mental and substance use disorder chief complaints would thus be of significant value. Any new algorithms, however, would need to be informed by local context. For example, California ED triage staff use the chief complaint phrase “5150” to denote patients under an involuntary psychiatric hold, but this phrase is relevant only in California.

The findings of this study have implications for public mental health surveillance efforts and behavioral health care delivery. Although syndromic surveillance data on MHSA-related ED presentations cannot be used to assess overall prevalence or new cases of disorder, our results suggest that such data can reliably indicate acute disorder occurrences or exacerbations. Prior research shows that the rate of such presentations responds precipitously to natural disasters (27) and changes in the local mental health system, among other things (28). Mental health data from syndromic surveillance systems—the only real-time, automated indicator of service utilization currently available—may thus be particularly useful for behavioral health service coordination and delivery in emergency situations (for example, incidents of mass violence), where timely cooperation between behavioral health, public health, and law enforcement agencies is vital.

Such data could also serve as a useful basis for ongoing communication and planning between local public health and behavioral health agencies, which frequently serve the same populations but which may not routinely share resources or information (29). For example, syndromic surveillance data could be feasibly used for tracking ED boarding times and disposition outcomes for patients with psychiatric emergencies, targeting substance abuse intervention and outreach efforts, and evaluating the effectiveness of locally implemented mental health–related programs. All of these prospective uses have direct implications for client-level service delivery. Moreover, because of the highly automated nature of syndromic surveillance systems, health agencies typically do not need additional staff in order to effectively make use of syndromic data. However, training of local emergency department staff to ensure the accuracy of the original data transmissions will improve the utility of these systems for public health practice.

Conclusions

This study demonstrates the potential for algorithms based on chief complaints to validly detect ED presentations for disorders related to mental health and substance abuse. Wider adoption of such algorithms into syndromic surveillance systems, and implementation of a standardized software system across agencies, could be a valuable tool for both local and national public mental health surveillance efforts.