Comparison of health information exchange data with self-report in measuring cancer screening

Evidence-based screening tests are important for early detection of cancer to reduce the likelihood of a more advanced stage at diagnosis when cancer may be less treatable [1]. Information about the receipt of cancer screening tests is widely available from patient data stored in electronic health records (EHR), a systematic collection of patient health information in a digital format [2]. Another alternative source of information about cancer screening is from patients’ self-report, an arguably more cost-effective source [3]. EHRs may benefit both clinical and epidemiologic research, patient care, as well as performance measurement. However, structured EHR data inevitably contains incomplete or inaccurate information. Furthermore, EHRs typically are deployed in a single health care practice, hospital, or system. Health information exchange (HIE), which involves the electronic exchange of clinical and administrative information across a variety of health care organizations, provides a more population-based approach to the measurement of cancer screening as it contains EHR data from multiple health care settings or systems [4, 5].

To our knowledge, no prior statewide studies of cancer screening has compared HIE data with patient self-report. Also, previous literature comparing self-report and EHR data showed varying levels of agreement based on factors such as type of clinical condition, type of clinical procedures performed, or data collection methods [6‐10]. Given the varying results of previous validation studies and the implications for population-based measurement in the use of HIE, we assessed the concordance between self-reported cancer screening by survey among Indiana residents and the individuals’ corresponding information from the statewide Indiana Health Information Exchange (IHIE). We focused on the cancer screening phase of the cancer care continuum for colorectal, cervical, and breast cancers. We hypothesized that the value of information gained from different data sources on cancer screening, might vary according to the types of screening tests.

From January 2018 through February 2018, Indiana residents who had been seen at least once in the previous year at Indiana University Health (IU Health) participated in a cross-sectional, mail-based survey known as the Hoosier Health Survey. A statewide integrated health system, IU Health operates 16 hospitals in Indiana and 178 clinics that provide outpatient care; it is the largest health system in the state, had 115,690 admissions in 2021 with a leading 31.3% market share in its primary service area (PSA) in central Indiana [11]. The purpose of this survey was to better understand the cancer control needs of the community served by the Indiana University Cancer Center [12]. The population catchment area of IU Cancer Center as defined for the National Cancer Institute includes the entire state of Indiana. Following HIPAA authorization from respondents to access their electronic health records, cancer screening information obtained through the survey were matched with the participant’s longitudinal EHR data by referencing each patient’s first and last name, birthdate, and place of residence. The electronic information was obtained from the Indiana Network for Patient Care (INPC), which is the clinical data repository for IHIE, a community-wide HIE operating in central Indiana, with the support of the Regenstrief Institute. The INPC consists of clinical observations across five major hospital systems, public health departments (both state and county), and Indiana Medicaid [13, 14]. The study was approved by the IUPUI Institutional Review Board.

Descriptive statistics, adjusted for sampling weights, were performed of individual socio-demographic characteristics that are patient reported, including age, gender, race, educational level, marital status, insurance status, income, home ownership, employment status, rurality based on RUCA codes, and self-reported health status.

For the questions on receipt of cancer screening and time since last screening, as a first step, we conducted bivariate analysis on participants’ responses to the survey and the corresponding information in HIE data using Chi-square tests to check for any significant differences between the two information sources. As a second step, we evaluated the sensitivity, specificity (only for receipt of cancer screening), and concordance as validity measures between the two measures of screening information. For the questions on time since last screening, we only considered those participants whose HIE data as well as self-report indicated receipt of screening. All the analyses were adjusted for sampling weights.

To assess under-reporting, we estimated sensitivity (proportion of patients who self-reported having a screening test done among those with the test documented in their EMR). To assess over-reporting, we estimated specificity (proportion of patients who self-reported not having a screening test done among those without the test documented in their EMR). Finally, we used the Gwet’s agreement coefficient (Gwet’s AC) to measure the concordance of screening information obtained from HIE data and survey self-report. The Gwet's agreement coefficient (Gwet's AC), a measure of correlation, is defined as the conditional probability that two randomly chosen observational measurements will agree, assuming no agreement by chance. The agreement coefficients were calculated using Gwet's new chance-corrected inter-rater agreement coefficients weighted ordinally, extending all existing agreement coefficients to include multiple raters, multiple rating categories, any measurement level, and multiple ratings per subject.

As an additional analysis, we showed comparison between three different agreement measures—Gwet’s Agreement Coefficient, Fleiss Kappa and Intraclass Correlation Coefficient (ICC) (see Appendix 3) but we chose Gwet’s AC as the final measure of concordance for this study over the alternative measures¹ for various statistical concerns [19‐26]. The Gwet's AC is interpreted according to Landis and Koch's guidelines [27, 28].

The analyses were performed in Stata (Stata 16.1, StataCorp LLC, College Station, TX).

In our study we focused on the cancer screening phase and evaluated the concordance between HIE data and self-reported responses of surveyed Indiana residents seen in a statewide healthcare system for receipt of screenings and the time since receipt of the last screening test. For screening receipt, results indicated the highest level of agreement for mammogram and lowest level of agreement for HPV test. For screening timing, the highest level of agreement between the two data sources was for mammogram timing and the lowest level of agreement for FIT test timing. Additionally, HIE data provided relatively more information about FIT and HPV tests, which are both laboratory-based screening tests. Self-reported data provided more information about colonoscopy, Pap test, and mammography, all of which are medical screening procedures.

In the screening phase of the cancer care continuum, one of the earliest prior studies assessed the concordance between self-report and non-electronic medical record documentation among Kaiser Permanente Medical Care Program participants. Data were collected on the reason and timing for Pap tests, mammograms, clinical breast exams, fecal occult blood tests (FIT tests), digital rectal examinations, and sigmoidoscopies. Researchers found that self-reported response and non-electronic medical record documentation generally agreed more for procedures involving a test report (mammogram, Pap test, fecal occult blood test, and sigmoidoscopy) than a physician's note (clinical breast examination and digital rectal examination) [6]. The results of our study are similar to their findings, especially for mammogram timing and receipt of FIT test where we found the most agreement between HIE data and self-report. A relatively recent study conducted among patients in 25 New Jersey Primary Care Practices who participated in the SCOPE program (supporting colorectal cancer outcomes through participatory engagement) indicated evidence of agreement for cancer screening ranging from 61% for Pap and PSA test to 83% for colorectal endoscopy. In this study, self-reports had a higher rate than non-electronic medical records [7]. Both of these studies compared patients' self-reports against non-electronic medical records. Paper-based documentation may not provide accurate information on screening histories if information is entered incompletely; further, non-electronic medical records also suffer from disorganization, non-integration with other electronic systems, and lack of backups and security issues [29, 30]. Hence, our study has the advantage of using EHRs from HIE over non-electronic health records which is more consistent with current medical practice. Moreover, EHRs arguably provide higher quality data with more accessible, accurate, complete and up-to-date patient records with built-in privacy and security features [31, 32].

In the treatment phase of the cancer care continuum, an academic hospital cancer registry study among breast cancer survivors from 2004–2009 evaluated concordance first between electronic query and manual review used to extract EHR data, and second between survivors' self-reports and the extracted EHR data on post-treatment mammography. Electronic query identified more mammograms post-treatment than manual review, with high concordance between the two methods (0.90). Fewer days since mammogram were associated with better concordance between self-reporting and EHR data. In conclusion, Tiro et al. encouraged the use of self-report as a screening tool among cancer survivors for surveillance care delivery [10]. The advantage of our approach over prior EHR-based studies is the fact that they were studies performed within a single health care setting. On the other hand, EHRs in a state-based HIE, as explored in this study, use clinical data from among patient populations aggregated across multiple health care organizations [33]. In addition to offering a complete, accurate, and holistic view of patient records, HIEs reduce duplication of information on procedures or tests, improving the usefulness of patient health records. HIE also improves the accessibility of medical data across multiple clinical settings, thereby improving the capacity to use population data for public health purposes and other quality improvement activities across consortia of health care organizations [34‐36]. Hence, using the Indiana Network of Patient Care (INPC), encompassing a community wide HIE, enables the measurement of population-based cancer screening behavior at a population level with greater efficiency and completeness.

Some study limitations must be considered when interpreting these results. First, our survey response rate was relatively low, at 12%, despite using established methods for survey research. Due to our expectation of a low response rate based upon current survey experience [37‐39], enough surveys were delivered, with follow-up postcard reminders and a second copy of the survey to have meaningful population-based estimates. This was to ensure a relatively large absolute number of surveys among the target population. Other data collection methods, such as in-person interviews, might have improved our participation rate but would not have had the same reach as the mailed survey. Nonetheless, we received completed surveys from every surveyed county [40], and respondents and non-respondents did not differ significantly across available sociodemographic characteristics [41]. Second, we selected only those residents who had been involved with a single health system in Indiana, although IU Health is Indiana's largest integrated health system serving approximately 1 million individuals community wide https://​iuhealth.​org. Nonetheless, the results of our study should be interpreted as a sample from a statewide health system with at least some access to healthcare, as opposed to a population-based state sample. Overall, the access to healthcare by participants increased the likelihood of cancer screening receipt occurring. Finally, some HIEs have greater challenges to data sharing because of state-level variation in patient consent policies for sharing of health data [42]. Specifically, opt-in policies that require providers to consent each patient to sharing information with HIE programs increase administrative costs that make HIE more burdensome. Thus, all researchers will not have uniform access to HIE in their communities.

Different data sources yielded different information value about the receipt of cancer screening, depending on the type of cancer screening. The HIE data, for example, provided relatively more information about FIT and HPV tests, both laboratory tests, than about colonoscopy, Pap tests, or mammograms, all procedures. To choose the ideal data collection method, studies that use a single data source should consider the type of cancer screening test. Both HIE and self-reports provided unique information about cancer screening. The most robust measurement approach involves collecting both HIE and self-reported screening information. When there are disagreements between the data sources, a practical approach may be to consider most positives measures of cancer screening tests as true positives, in order to overcome the risks of false negatives posed by HIE (missing data) and self-report (recall bias). If one source of data is used over another, it will likely create biases in prediction, for example algorithms based upon different sources of cancer screening data (HIE vs. self-reported) will very likely provide different predictions about cancer mortality, and furthermore, these predictions will vary by different race/ethnicity groups. Moreover, the optimal data source may vary depending on the outcome of interest being measured, whether it is clinical decision making, performance measurement, or population surveillance. Future research opportunities include looking at concordance between self-report and EHR data over time, concentrating on vulnerable populations. Data should also be considered from different EHR systems such as single vendor EHRs, as well as EHRs controlled by patients known as personal health records, to draw comparisons with corresponding patient self-report regarding cancer screening.