Public health surveillance in the U.S. Department of Veterans Affairs: evaluation of the Praedico surveillance system

Simplicity refers to structure and ease of system operation. Praedico is a complex system but with a user-friendly interface that does not require knowledge of a programming language. Instead, users select the data source or domain(s) to query and specify search criteria using dropdowns and wildcard expressions (Fig. 2). This provides time savings for routine querying and allows for less experienced users to perform queries without the need to write query expressions. Queries can be saved, refined, and executed on demand. This allows data needed on a routine basis to be retrieved quickly and consistently. An imbedded advanced query tool and data join feature allows more experienced users to create custom syntax for complicated queries and perform intersecting time series analyses (Fig. 3). This is important for complex epidemiologic questions, special studies and data requests, as well as some PHPO epidemiological investigations which need to link data from different clinical domains.

Data analysis capabilities are built into the system, including time series, pie and bar graphs, table generation, percentage calculations, anomaly detection, data stratification, and data joins for combining multiple queries. (Fig. 4). Data can also be exported into other applications such as Excel, R, SAS®, Tableau®, Python, and others. Data visualization includes geospatial mapping (Fig. 5). Data discovery, graphing and visualization tools enables users to get a better sense of the distribution of a dataset and provides opportunities to assess trends or monitor a disease over time or geography.

Flexibility reflects the way a system adapts to changing information needs, technology, or operating conditions. Highly flexible systems adapt with relatively little time, personnel or additional funds and can be integrated with other systems. Praedico is data source agnostic and scalable. It can be adjusted to pull data from multiple sources including HL7, flat files, VistA, SQL databases and big data stacks (such as HDFS) and has the capability to harvest data from commercial EHR systems (e.g. Cerner Millennium® and Epic®), as well as unstructured sources (e.g. social media and news outlets). Data can be directly imported into the user interface and exported from any of the three system layers. Praedico has demonstrated flexibility by successfully adding new data domains from different sources over time, including the VHA’s telephone triage (DSHI TriageXpert™) system. Because Telephone triage data reside in a separate database and are not available in VistA or CDW, Praedico is the only way for PHPO epidemiologists to access this important data source. Triage data captures information on a range of illnesses and severity, occurs before patients access other health care resources or seek in-person care, and thus is important data for early event detection and situational awareness. This is particularly important for augmenting VHA’s influenza and other disease surveillance [29].

Since its launch in 2015, Praedico has been adapted to meet new PHPO influenza surveillance needs by supplying inpatient, outpatient, telephone triage, ILI and vaccine data for biweekly influenza surveillance reports and studies [29]. In 2019, Praedico began producing an automated, weekly report of influenza-related hospitalizations, intensive care admissions and deaths that eliminates much of the manual case review needed previously. During the emergence of Zika virus, Praedico was leveraged to track Zika laboratory results via a weekly data pull for on-going surveillance across VHA and to monitor hospitalizations, neurologic complications and deaths following Zika virus infection [35, 36]. More recently during the COVID-19 pandemic, Praedico data was used to develop a SARS-CoV-2 machine learning prediction model combining vital sign values, laboratory, and imaging results [39] and supplied data on COVID-19 hospitalizations and complications among Veterans with SARS-CoV-2 compared to hospitalized patients with influenza [37]. Praedico was also used for an internal evaluation of COVID-19 testing practices because the performing laboratory location (including send-out tests) could be quickly added to Praedico laboratory domain [40]. Although the system has demonstrated a good deal of flexibility, incorporation of additional data domains or major system modifications may incur additional costs. For example, current work to add additional immunization data required a contract modification and supplemental funding.

Praedico is flexible in terms of hardware and software requirements. It’s based on a scalable modular architecture that can run on commodity hardware. The Cloudera infrastructure supporting Praedico can serve a broader team’s raw data needs using Cloudera components (e.g. Pig, Hive and Apache Phoenix). These tools provide a platform for experimenting with the viability of incorporating new data domains inside Praedico’s data rubric and the ability to independently run and verify queries generated by the application.

Praedico was designed to allow various IT platforms within VHA to communicate with one another to create a central pool of data for surveillance. Praedico’s data-ingestion services seamlessly connects and consumes data from VHA’s heterogeneous databases. Praedico’s data lake repository architecture is designed to work on structured, semi-structured and unstructured data. As a result, it is ideally suited to work with and adapt to any data model, including newer standards such as Observational Medical Outcomes Partnership (OMOP) Common Data Model and Fast Healthcare Interoperability Resources (FHIR). For potential data sharing, Praedico could tap into other systems and directly integrate data sources to allow for simple exchange of data, rather than requiring agencies to adopt the same or similar systems.

Stability focuses on reliability and availability of the surveillance system. Overall, we found system stability very high, particularly on the application side. Praedico runs on clusters of commodity hardware for distributed processing, load balancing, high availability, and failover support. There are rare occasions where a data domain load will fail and need to be re-extracted from VistA or issues loading data from a specific VHA facility. To monitor reliability, users can navigate to the “System Status” section of Praedico to review data load status for each data domain, including daily incremental load status and any known data issues. These can include site-specific issues requiring re-extraction or reload. When issues are resolved a notation is added. For any large data extraction or availability issues, a banner will appear at the top of the application alerting users to data availability issues. A review of system reliability found that across all domains, approximately 0.01% of files fail and cannot be re-extracting in time to be available to users on the day that it should be. A review of system uptime from May 2018-February 2021 found that Praedico was available 99.5% of the time during regular business hours, with all downtimes traced back to infrastructure problems (Table 2).

Stability is also reflected in resilience to system changes. In 2015, Praedico seamlessly adopted the ICD-10-CM and ICD-10-PCS coding systems, while maintaining ICD-9-CM system for historical records. The user interface allows users to query ICD diagnoses, procedures or syndromes using either or both coding systems. It has built-in crosswalk functionality with general equivalence mapping between ICD-9-CM and ICD-10-CM that can be used to append or replace codes from one system to the other (Fig. 2).

Timeliness considers the speed between steps in the system and how quickly data is made available. Praedico is currently configured to extract and process VHA clinical data once daily based on the current license parameters, which means that users can typically query data as recently as the day prior to the current date. System alerts for pre-defined syndromes, designated reportable diseases and user-created custom alerts also run once daily. In comparison, VHA data is sent hourly to CDC BioSense and is loaded and available in NSSP ESSENCE within 4-5 hours. Data in CDW production is updated once daily. Increasing the frequency of data pulls across all data domains in Praedico requires a higher network bandwidth than currently supported in VHA. However, the application allows pulling in of targeted domains more frequently, should users require it.

Near-real-time data upload allows users to identify actionable data in a timely manner for appropriate response or mitigation efforts. For all domains, data is available in Praedico within 1 day of the triggering event. Generally, this is when an encounter is created or updated. Because outpatient data are not available for extraction until at least one diagnosis or procedure code has been assigned there can be a minor lag in data availability. For inpatient domain, some admission records are not captured in daily incremental runs due to the way record sequence numbers are assigned. This is mitigated with a full historical extract of inpatient files each weekend, but can lead to a lag of a week or more for some inpatient records.

To quantitatively assess speed of the system, a series of 5 diagnosis and syndromic queries were compared against analogous queries in NSSP ESSENCE. To account for possible variations in network speed, the queries were run during three distinct time periods. The time to execute and download query results was measured (Table 3). Run times were faster in Praedico for all queries, regardless of the time of day performed. The time to download was similar, although slightly faster overall in NSSP ESSENCE.

Representativeness refers to how well a system accurately describes occurrence of health events over time, and distribution in the population. VHA serves approximately 9 million enrollees out of over 14 million who are currently eligible to receive care from VHA. Enrollees are predominately older males and there is little to no pediatric data. However, enrollment of younger Veterans is increasing, and surveys have found that 62% of the over 1.9 million Veterans from Operation Enduring Freedom, Operation Iraqi Freedom and Operation New Dawn (OEF/OIF/OND) have obtained healthcare from VHA [45]. Praedico captures data for individuals who receive care within the VHA, and limited data for Veterans who received care outside the VHA system. Praedico is generally representative of the Veteran population receiving care in VHA however, VHA surveillance data may not be generalizable to the broader U.S. population. Nevertheless, for some surveillance data (such as influenza surveillance) we found that VHA data are highly correlated with national CDC surveillance data [29].

The system pulls outpatient, inpatient, bed section, pharmacy orders, Bar-Coded Medication Administration (BCMA), clinical chemistry laboratory (Lab-Chem), clinical microbiology laboratory (Lab-Micro), surgery, immunization, and procedure data as direct VistA extracts. For most domains, data are available from January 1, 2010 to present (full Lab-Chem data are available from January 2015 onward, with test types limited from January 2010-December 2014, and immunization table data from 2019 onward). Cost constraints have precluded ingestion of additional historical data for Praedico at this time. There is also a data gap in Telephone Triage data from November 2016-June 2017 due to a migration from regional to a single centralized server. Data in CDW are available back to 1999, which is an advantage when additional historical data are needed.

Only limited non-VHA care data (including DoD and community care) are available in Praedico. Although the data in Praedico encompass most critical public health surveillance domains, Praedico does not currently pull in other important clinical datasets and domains such as radiology, vital signs, problem lists, critical care management data (e.g. Picis or CareVue), pathology, transfusion, or provider progress notes. Although CDW contains some of these domains, it’s also missing data in key domains that are present in Praedico. Notably, in the fall of 2015, a safety communication was issued reporting an association of Nontuberculous Mycobacterium with heater-cooler devices (used to warm or cool blood and organs during surgeries) [46]. VHA PHPO epidemiologists collaborated with VA National Center for Patient Safety to identify potential infections among Veterans exposed to these devices. Praedico was used because Mycobacterial laboratory data are incomplete in CDW [41]. Praedico has subsequently been used for other Mycobacterial investigations and Tuberculosis reviews [42‐44]. A summary of the availability of key data domains across the various platforms is presented in Table 4.

Data quality measures the completeness and validity of the data in a system. A strength of Praedico is that data are extracted directly from the VistA source. There are multiple extractors with failover and redundancy capabilities and monitoring to check that extractors are operating correctly. Praedico’s extract, transform, load (ETL) module performs the suite of data quality checks needed for ensuring data fidelity to its source. For completeness, each extracted file has a header and trailer that are checked during processing. If these are missing, the file is deleted and re-extracted. At each stage of ETL, the input versus output record count must match, otherwise an alert is generated. There is also manual sampling and a statistical trend analysis to monitor the expected range of records and deviations are investigated. For validity, the ETL pipeline verifies that all expected fields are present in the correct order and that the data type for each field is correct. Additional checks include the extraction engine keeping track of the total records extracted from VistA in each extract file which is further tallied in the ETL pipeline during processing. Assessments of Praedico data quality, including completeness and validity were performed via a record-level comparison to CDW and raw VistA data extracts by data domain in 2019 (Table 5) and re-assessed in 2020.

Data quality was in general high. Regarding data completeness, the review identified missing records in the Lab-Chem domain (~ 1% missing records) and Pharmacy Outpatient-Refills (~ 38% missing records, which required a reload of historical data from 2010 to 2019 for all facilities). Users also reported data completeness issues with missing test name field in some Lab-Micro records prior to 2013 and there is incomplete data from the Spokane VA Medical Center which was the first VHA facility to migrate from VistA/CPRS to Cerner EHR for which Praedico data access has not been established. The 2020 re-assessment found improvement in most domains, except for BCMA where approximately 3.3% of CDW records were missing in Praedico. Some deficiencies were also identified with data validity where the content of Praedico records did not match CDW. These discrepancies are generally related to records that are altered, updated, or deleted after initial transmission. Praedico identifies modified records using an incremental lookback window. The window varies by domain and must be balanced with time needed for daily extraction of new records. Praedico has expanded its lookback window for most domains (previously 5000 records per domain) and added a longer window to run every weekend (Table 6). This improved but did not completely resolve identified data completeness and validity issues. CDW, on the other hand, has access to VistA journal, an audit trail of all changes made to VistA files, which Praedico does not currently have permission to read. Data quality issues are discussed further in the following two sections.

Sensitivity is best described as the proportion of cases of a disease that the system identifies as well as the ability of the system to detect outbreaks or monitor changes in cases over time. To perform an evaluation of sensitivity, the ability of Praedico to correctly identify influenza hospitalizations was assessed. Influenza diagnosis codes (ICD-10-CM: J09-J11) were queried in Praedico for the 2017-18 and again for the 2019-20 influenza seasons and compared to equivalent queries developed for CDW. EHR reviews were performed to determine why cases were missed. Praedico correctly identified 10,424 of the 10,426 hospitalizations found in CDW during 2017-18 and 7018 of 7079 hospitalizations during 2019-20 (sensitivity = 0.999 and 0.991, respectively). Two hospitalizations missed in 2017-18 were modified records where the influenza diagnosis code was not initially present when the record was extracted by Praedico. Seven missing records from 2019-20 were patients hospitalized outside the VHA system. The remaining missing records in 2019-20 were determined to have an “out of order” record identifier in the VistA file which prevented Praedico’s VistA extract from identifying them.

An analysis to evaluate the sensitivity of Praedico data for the Lab-Chem domain was performed for January 2018 - March 2019. Laboratory results are frequently utilized to identify disease cases or to confirm a diagnosis. During this time there were over 750 million Lab-Chem records, with a total of 265,972 (0.04%) missing records in Praedico. A more detailed analysis of the test result value field was performed for 14 different VHA facilities (4 facilities during January - December 2016, 5 facilities during January - December 2017, and 5 facilities during January 2018 - February 2019). The number of incorrect test results in Praedico ranged from 3 to 669 for the 1.6 million to 12 million tests performed for the facilities and time periods analyzed. Sensitivity was > 0.999 for all analyses performed. Again, incorrect test values occurred because records were subsequently modified but not updated in Praedico. This issue and approaches to minimize its impact were described in the Data Quality section.

For outbreak detection, a comparison of Praedico with the retired VA ESSENCE was performed. A validation set combining 17 million DoD and 25 million VHA records was used for the initial evaluation [28] and a second analysis used 513 million VHA patient records was performed [47]. For the influenza-like-illness (ILI) syndrome, both applications displayed the same number of ILI cases. However, for the time period June 2014 – May 2015, only 62% of Praedico syndromic alerts directly correlated with VA ESSENCE, suggesting that Praedico identified some ILI outbreaks/clusters that were not detected by ESSENCE and that the alerts in Praedico were not simply a subset of VA ESSENCE alerts. Praedico also demonstrated higher seasonal sensitivity (with increased ILI alerting in December and January and more likely representing actual influenza activity), adjusting for seasonality using historical and seasonal information, whereas VA ESSENCE alerts were more uniformly distributed throughout the year [47]. Sensitivity of the Praedico system is additionally enhanced through the access to DSHI TriageXpert™ telephone triage data which captures mild illnesses among individuals that may never seek in-person care and thus represents illness data that is not available via CDW or other active VHA surveillance systems.

PVP describes the proportion of disease cases in a system that are true cases. Here, we again evaluated the ability of Praedico to correctly identify influenza-coded hospitalizations in VHA as well as false positives in the laboratory domains. As described above, influenza hospitalizations were queried during two influenza seasons and compared to results from CDW. A total of 4 out of 10,428 hospitalizations in 2017-18 and 1 out of 7079 in 2019-20 identified in Praedico were not present in CDW (PVP > 0.999 for both seasons). Record review against raw VistA files determined these represented false positives because the influenza code was deleted from the record during a subsequent modification, but not updated in Praedico.

The Lab-Chem analysis for the 14 VHA facilities described above found false positives ranging from 32 to 25,450 for the 1.6 million to 12 million lab tests performed per site. PVP was > 0.98 for all analyses performed. For the Lab-Micro domain during January 1, 2010 – April 10, 2019 we identified 21,714 false positives out of 20,923,588 tests performed (PVP = 0.9989). In these domains, false positives were records deleted from VistA but not from Praedico. A further broad review of all records in 7 major Praedico data domains found that false positives ranged from a low of 0.1% (BCMA, Lab-Chem, Lab-Micro, Outpatient Pharmacy and Outpatient Pharmacy- Refills) to a high of 1% (Inpatient).

False positive alerts are a problem that plague many surveillance systems. The ability of the BDaP data extraction component of Praedico to scrub data following a set of business intelligence rules minimizes the number of extraneous alerts. False alerting results in lost time and resources and creates the impression among users that most alerts are irrelevant. To assess alerting, Praedico was compared to VA ESSENCE and found to be somewhat more accurate with fewer false positive alerts [47]. This translated to less alerting fatigue for users and lessened the need for frequent and intensive manual review of surveillance data.

Acceptability refers to the willingness of persons and organizations to participate in the surveillance system. Veterans do not need to enroll or opt-in to having their clinical data made available to Praedico [48]. Because the data are used for public health operational activities, VHA Office of Research Oversight considers analysis of these data as operational and not research in VHA [49]. Features of Praedico that contribute to acceptability among users of the system are simplicity, speed, flexibility, and other key attributes described earlier, such as alerting, automation, analytic and visualization capabilities, and a straightforward user interface. One additional element of Praedico that increases acceptability is the user workspace. Here users can run and save queries that span multiple datasets, set alerts, request and share reports, and more. Saved queries can be executed, modified, duplicated, deleted, or shared. Workspaces are customizable, creating a dynamic dashboard that organizes queries in user-defined groupings.

In general, however, acceptability is a limitation as Praedico is only utilized by a small number of PHPO epidemiologists. However, the system has relevant applications for a variety of health system personnel including epidemiologists, informaticians, clinicians, researchers, infection preventionists, antimicrobial stewards and other data analysts. Spin-offs of Praedico using similar technology have been developed by Bitscopic Inc. and some are currently in use in VHA including PraediGene (laboratory workflow, financial and DNA analysis tool), PraedAlert (clinical surveillance and antimicrobial stewardship tool), PraediCare (analytic and reporting system for high acuity clinical areas) and PraediTrial (clinical trial recruitment/enrollment tool). Praedico is a major expenditure for VHA’s PHPO group and only available when VHA has an active license for its use. Costs must be budgeted carefully as any major modification requires contract modification and supplemental funding. Also, separate IT funding must be secured for system maintenance and infrastructure needs. Ultimately, cost may be the primary factor working against acceptability, especially in a climate of restricted program budgets. Examples of other factors and limitations users have reported which affect acceptability are fewer years of historical data and limited number of data domains available, constraints on the number of records that can be downloaded and exported, missing tests names for Lab-Micro data prior to 2013, missing test types for Lab-Chem data prior to 2015, missing Emergency and Urgent Care chief complaint data, and not being able to filter numeric values for laboratory test results. Furthermore, adapting existing CDW queries to run in Praedico may not be straightforward or possible, which leads some users to continue using CDW in situations where they already have a working query.