On the effect of electronic patient portal on primary care utilization and appointment adherence

The study was conducted at UF Health, a medical network associated with the University of Florida (UF). The UF Health network includes two academic hospitals and several other hospitals and facilities in North Central Florida. In 2011, UF Health started offering “MyUFHealth,” also known as MyChart® by Epic®. MyUFHealth is an electronic patient portal that provides patients a secure and convenient way to access portions of their medical records (e.g., released test results, after visit summary), communication with the clinical service providers using secure messaging, request prescription refills, and management of outpatient appointments. MyUFHealth is available to patients who are seen in the UF Health network at Gainesville or Jacksonville hospitals and physician outpatient practices. MyUFHealth pediatric proxy for children under 18 years old is also available and can be established in the UF Health Physicians clinics. Proxy access allows a parent (or guardian) to log into their personal MyUFHealth account, and then connect to the MyUFHealth account of their child. Therefore, children under 18 years old can also be portal users in this study.

The data protocol used for the retrospective observational study was approved by the UF Institutional Review Board. The study period was from July 1st, 2013 to June 30th, 2016, i.e., fiscal years 2013–2015. We denote January – March as Q1, April – June as Q2, July – September as Q3, and October – December as Q4. For instance, Y13Q3 stands for the third quarter of the year 2013. The study population consisted of 46,544 UF Health patients who had at least one visit to UF Health Physicians clinics during the study period. Among these 46,544 patients, more than 95% of them came from North Central Florida (based on zip-code categorization), and more than 70% of them were from Alachua County, where UF Health is the major care provider. To classify patients, we only denote patients who adopted portal during the course of the study, and kept the active status until the end of the study as users, which were 9049 patients out of the whole population. Five thousand four hundred twelve patients adopted portal before the study period, and were nominated as consistent-users. Additionally, there were 715 temporary-users, who became inactive by the end of the study. Meanwhile, 31,368 patients never registered for patient portal and were denoted as non-users. The patient classification can be found in Fig. 1.

We further define new patients as who have not received any professional services, i.e., Evaluation and Management (E/M) service or other face-to-face service (e.g., surgical procedure) from the physician or physician group practice (family medicine or primary care) within the previous 3 years in the UF Health network. This definition is in consistent with the Current Procedural Terminology (CPT) [31]. Because patients were more likely to receive the portal activation code during or after an office visit, there was a resulting mechanical increase in the office visit rate for portal users at the quarter of adoption and its surrounding quarters (trends similar to the figure presented in [7] were witnessed). To eliminate the nested effect, we limited our focus to new patients, for whom we can identify the time they recorded a new diagnosis, allowing some assessment of the natural disease process. We used the visit type categorization “New” in EMR to pin down patients’ first-time UF Health encounters. Notably, patients typically adopted portal at the time of their new visits or right after their new visits. To ensure a before-after portal adoption contrast, we only focused on new patients who had their new visits and portal adoption (users only) after Y13Q3. We also excluded patients who were consistent-users and temporary-users. This inclusion criteria led to 15,659 non-users and 5494 users identified.

We were able to obtain information from the EMR and administrative databases on patients’ age, gender, race/ethnicity, marital status, insurance status, and ambulatory care utilization (telephone encounters, and office visits to PCP and specialty care clinics within the care network). We also recorded patients’ active problem number (APN). The active problem list in EMR records a patient’s major health conditions, including both chronic conditions and ongoing conditions that are resolvable but are important for physicians’ to make clinical decisions. The active problem list is typically reviewed and updated if needed at each patient encounter, and is used as a proxy for an individual’s disease burden in this study. Monthly portal access events and clinical service usage of individual patient were generally not frequent, so data were consolidated by quarter. Admittedly, the data set only includes patient ambulatory care utilization within the UF Health network. However, UF Health is the leading care provider in the region under investigation. Additionally, more than 95% of patients in this study were insured. The insurers typically request patients to select their primary care physician (e.g., Blue Cross Blue Shield, Commercial and Managed Care), or seek care with participating providers (e.g., Medicare and Medicaid). These suggest that primary care services rendered to patients outside of the UF Health network would be very limited. Therefore, the primary care utilization within UF Health is used as an indicator of the overall primary care utilization of patients.

As an observational study, the initial patient characteristics of users and non-users were dissimilar, and we first calculated propensity scores by estimating the probability of becoming a user using logistic regression. Confounders that potentially affect both outcomes (primary care service utilization) and exposure to the intervention (becoming a portal user) include time-invariant variables (age, gender, race, marital status, insurance status), and time-varying variables (APN and PCP office visit arrived, no-show, and cancellation).

As an observational study, both events -- becoming a new patient of UF Health and adopting patient portal -- happened throughout the study period, rather than at a fixed time. A challenge for matching emerges: if matched at the beginning of the study period, individual patients’ time-varying characteristics could diverge dramatically before the treatment started. Therefore, a baseline for each patient is defined for this study as the time (quarter) right before adopting portal, or before having the new visit, whichever was earlier. For instance, if a patient had his/her new visit in Y13Q4, and adopted portal in Y14Q1, then, the baseline is chosen as Y13Q3. The baseline definition is illustrated in Fig. 2.

To match patients, the baseline APN was discretized into three categories, characterized by `0′, `1–4′, and `5 and more’ active medical problems. Summation of PCP office visits (including new and other office visit types, if any) arrived at baseline were categorized as `0′, `1′, and `2 and more’, and that of no-show, and cancellation were dichotomized into `0′, and `one and more’. Discretization helps eliminate outliers and adjust the variance structure of count data.

The matching was performed using propensity scores with the nearest neighbor selection criteria. To ensure the quality of matching, descriptive statistics of the confounding variables were calculated and tested using Chi-square tests (see Table 1). In Fig. 3, we illustrate a comparison of a subpopulation of portal users and matched non-users. The subpopulation included users who had their new visits during Y14Q2 and their portal adoption during Y14Q2 - Y14Q3, and non-users who had their new visits during Y14Q2 with characteristics matched with users at baseline Y14Q1. Notably, patients typically received the portal activation code during or after an office visit, and the office visit rate at the quarter of enrollment and its surrounding quarters were significantly higher than average due to the nature of disease process. Additionally, although matched at the baseline, users’ and non-users’ active problem numbers can diverge significantly. To elicit the effect of patient portal, the time-dynamic feature of disease burden needs to be captured, which motivates a causal inference study accounting for such time-varying confounders.

In our study, we recorded both the time when patient i had their new PCP office visit (time of clinic enrollment: NewIndex_i) and the treatment time (time of portal activation AdoptIndex_i). NewIndex_i and AdoptIndex_i for each patient i can be different. Both disease effect and treatment effect are time varying, and we are interested in the effect of treatment over time. Thus, we proposed a model using a panel-DID framework accounting for heterogeneous enrollment and treatment times. All the DID analyses in this study were conducted using users and matched non-users.

To address the confounding effects, discretized active problem number is incorporated as a time-varying covariate APN_it. Other confounders are represented by X_i, an ensemble of the time-invariant control covariates (age, gender, race, marital status, and insurance status) for the i^th patient. To accommodate the special variance structure of count responses, a generalized linear model with Poisson distribution adjusted for over-dispersion is proposed:

In model (1), g(·) is a log-link function, Y_it is the outcome variable (number of office visits arrived for the i^th patient at time t), and E[Y_it] represents the expectation of the corresponding outcome. User_i is a dummy variable to indicate whether the i^th patient belongs to the treatment group or not. Let index t denote the time the measurements (responses and covariates) were recorded, t = 0, 1, …, T. We let t = 0 represent the beginning of study period (Y13Q3), and T = 11 represent Y16Q2. Let Time_iτ(t) be a dummy variable to adjust for fixed effect of time λ_τ, and Time_iτ(t) = 1 if and only if τ = t, t = 0, …, T.

Patients could have their new PCP office visits during any t such that T ≥ t > 0. To incorporate the disease process which affects both users and non-users, a dummy variable D_ik(t) is introduced to represent if the absolute time t is k quarters post becoming new patients, and D_ik(t) = 1 if and only if t − NewIndex_i = k. The coefficient θ_k is the disease effect at time lag-k, k = 0, …, T − 1. To investigate the role time-lag plays in treatment (portal adoption) effects, a dummy variable P_ik(t) is introduced, and P_ik(t) = 1 if and only if t − AdoptIndex_i = k and User_i = 1. Consequently, β_k is the lag-k treatment effect, i.e., the difference-in-differences between the users and non-users at the k^th quarter post portal adoption. Specifically, β₀ is the treatment effect at the time of adoption. Figure 4 is presented to illustrate the design of the DID analysis.

Using the proposed model, the time-varying portal effect on care utilization can be analyzed. In addition, we evaluated the prevalence of appointment no-shows and cancellations between the user and non-user groups. To this end, Y_it is replaced by the count of PCP appointment no-shows for patient i at quarter t and we fitted Model (1). With this modification, another model was run with the same structure but the response Z_it is the count of PCP office visit appointments made that were not no-show.

No-show rate is defined as the number of appointment no-shows divided by the number of appointments scheduled for a given duration. At patient-level, we denote the number of appointment no-shows for patient i at the k^th quarter post adoption as Y_ik(t), and denote the number of appointments made that were not no-show for as Z_ik(t). P_ik(t) denotes whether time t is the k^th quarter post adoption or not, and X_i reflects a patient’s individual time-invariant attributes. For users at the k^th quarter post intervention, we are interested in the patient-level no-show rate denoted as

For brevity purposes, we omit the quarter index k and introduce \( {r}_i^1 \) and \( {r}_i^0 \) to represent a user’s no-show rate and a counterfactual case (if that patient is a non-user):

Then, the no-show rate ratio of patient i comparing before and after portal adoption is denoted as \( R{R}_i=\frac{r_i^1}{r_i^0}. \) Before moving forward, we introduce another set of variables: \( {\rho}_i^1=\frac{E\Big[{Y}_i\left|{P}_i=1,{X}_i\right]}{E\Big[{Z}_i\left|{P}_i=1,{X}_i\right]}, \) \( {\rho}_i^0=\frac{E\Big[{Y}_i\left|{P}_i=0,{X}_i\right]}{E\Big[{Z}_i\left|{P}_i=0,{X}_i\right]}, \) and \( {\Omega}_i=\frac{\rho_i^1}{\rho_i^0}. \)

\( {\rho}_i^1 \) and \( {\rho}_i^0 \) can be roughly interpreted as the average number of no-shows that will occur to successfully complete one appointment. We denote θ and δ as the coefficients that are independent of P_i and X_i; then, based on models (1) and (2), \( {\rho}_i^1={\exp}^{\theta -\delta +\left({\beta}_k-{b}_k\right)+\left(\eta -\omega \right){X}_i} \), \( {\rho}_i^0={\exp}^{\theta -\delta +\left(\eta -\omega \right){X}_i} \), and \( {\Omega}_i={\exp}^{\beta_k-{b}_k} \). It can be easily proved that RR_i < 1iff. Ω_i < 1. If the no-show rate is reduced after portal adoption, then Ω_i < 1, and vice versa. Therefore, testing the hypothesis H₀: RR_i < 1 is equivalent to testing the hypothesis H₀^’: Ω_i < 1. Specifically, β_k is the difference-in-differences of PCP appointment no-shows and b_k is difference-in-differences of PCP appointments made that were not no-show between the users and non-users at the k^th quarter post adoption. If \( {\exp}^{\beta_k-{b}_k}<1 \), then, it can be concluded that accessing portal is associated with a reduced no-show rate. The differences of coefficients of treatment effect between the two models, i.e., β_k − b_k, k = 0, …, T − 1, can be compared and formally tested. Similarly, Y_ik(t) can be the number of PCP appointment cancellations for patient i at quarter k post intervention to investigate the ratio of cancellation rate in the user group to that in the non-user group.

The characteristics of the unmatched and matched cohorts are shown in Table 1. When employing propensity scoring to match non-users to portal users, we noticed that individuals who enrolled in patient portal were mostly middle aged (between 31 and 64 years) and more likely to be female. The proportion of teenage patients was greater in the non-user cohort, possibly due to impacts from the proxy issue. Moreover, patients whose marital status was recorded as “Married” were more likely to adopt patient portal. Medicare and Medicaid patients, and patients with race self-reported as Black or African American were less likely to be portal adopters. These observations are consistent with previous studies on social disparities in enrollment and use of patient portals [32‐37]. Portal adoption was also associated with the baseline number of active medical problems on the problem list (p < 0.05).

Using the panel-DID models, we compared the utilization of PCP office visits between portal users and non-users. The RR of users to non-users attributable to treatment effect at the time of adoption was 1.33 (95% CI 1.27–1.39; P < 0.001). This boosting effect declined immediately after adoption (see Table 2). Specifically, the count of PCP office visits was similar between users and non-users in the first four quarters after portal adoption, and that of users was significantly lower than non-users in the seventh (RR =0.82; 95% CI, 0.73–0.91; p < 0.001) and the eighth (RR = 0.80; 95% CI, 0.70–0.90; p < 0.001) quarters post adoption. Insignificant differences for the ninth and tenth quarters post adoption, possibly due to insufficient sample size (only about 10% of users defined in the study adopted portal at the second quarter (Y13Q4) and had a history of ten quarters post adoption) were observed. The trend of RR attributable to lag-k treatment effect can be seen in Table 2. The observation is not consistent with those in [7] and other analyses which employed the same methodologies without accounting for the natural process of disease confounder.

Additionally, we examined the relationship between portal usage and appointment adherence. In particular, we chose our proxies for PCP appointment no-show and cancellation rates as outcomes (see Table 2). The no-show rate proxies of the user group were significantly lower than that of non-users. RRs were between 0.60 and 0.83 for eight out of eleven quarters, and for the remaining three quarters, the differences were not significant (detailed p-values can be found in Table 2). The differences in cancellation rate proxies were not significant (with p > 0.05 for all cases). Overall, patient appointment adherence was improved after portal adoption.

Robustness is the degree to which a model performs effectively even if some of its variables are modified. First, we analyzed robustness using an alternative measure of portal usage, where we define users as new patients who exhibited at least two portal logins (one for initial activation and then a subsequent login) during Y13Q4 – Y16Q2. Those who did not have at least two logins were considered as non-users even if the patient qualifies the inclusion criteria for “user” defined previously. This modification guarantees that the alternative definition is close to the original one but with minor changes, and would eliminate the coding errors when patients’ portal status was mischaracterized as “Activated.” The corresponding results were similar to the original model with respect to RR estimates and significance, which confirmed the robustness of the underlying model. Furthermore, we adjusted the inclusion criteria by including new patients who were consistent-users (who adopted portal before or right at the beginning of the study period). We identified 744 patients and assumed for those consistent-users that their adoption happened at the first quarter (AdoptIndex_i = 0). Adding consistent-users to the user group produced minor discrepancies between the two sets of panel-DID models concerning the absolute values and significance of coefficient estimations, but the trends of treatment effects in both sets of models were consistent. The trends of RR attributable to lag-k treatment effect for PCP office visit arrived with different inclusion criteria, (a) users with more than one portal login, and (b) including users and consistent-users are exhibited in Fig. 5.

Portal users were found to have fewer PCP office visits, adjusted by disease burden, compared to non-users in about one and a half year and thereafter post portal adoption. It suggests that the convenience brought by patient portal for supporting better provider-patient interaction might reduce patient in-person visit in a longer time-frame rather than immediately. This might possibly be explained as patients need time to adapt to portal functionalities, and patient portals influence patient health behavior gradually. In addition, it has been shown that patients with a high propensity to “no-show” for appointments will have worse clinical and acute care utilization outcomes compared to patients with a lower propensity [40]. We found that the no-show rates of portal users were lower compared to non-users, which suggests that portal access offers the promise to improve patient outcome. Moreover, missed appointments are known to critically impair clinic operational efficiency. In this study, the number of no-shows was estimated to be 0.14–0.20 per enrolled patient per year. Considering a PCP with a panel size of 1300 patients [41], the current no-show rate is equivalent to wasting an average of 182–260 appointment slots per provider. An average of 20% reduction, if it can be achieved through portal adoption, entails around forty less no-shows annually, or one less no-show weekly. Although not a significant improvement of provider productivity, the impact could be more pronounced by considering rescheduling no-show patients and other associated operating costs. On the other hand, it should be noted that portal access incurred an average of four messages sent from patients to their providers annually during the study period. If all 1300 panel patients turn into portal users, then a weekly average of 113 messages is expected for a single provider. Medical resources need to be allocated and physician appointment capacity needs to be adjusted to accommodate the demand shift introduced by increased patient portal adoption. Future work can be directed at investigating whether decreased PCP visits are associated with increased portal logins or messaging, and whether increased APN is associated with more portal logins or messaging. Additionally, payment structures that accommodate technologically-mediated interactions between providers and patients (e.g., text messaging, email, virtual visit, etc.) are instrumental in eliminating the adoption barrier of providers and should also be explored.

There are several limitations to our study. First, whether portal access affects health outcomes beyond care utilization needs to be systematically evaluated, and the business value and economic impact of portal usage need to be quantified. For the study design, the follow-up is limited to 10 quarters post adoption, and a longer observation horizon is beneficial to evaluate the long-run effect of portal adoption. Moreover, in addition to the overall portal effect, the usage of different portal features, such as appointment scheduling, lab results review, medication refill, and the messaging function, should be investigated. An analysis that incorporates the type and frequency of portal access by users can be informative. The user population was highly heterogeneous, and the impact of different portal features on subgroups can be diverse. Therefore, a subgroup analysis utilizing the detailed portal usage information should be carried out to investigate different portal features’ impact on patients. Lastly, the causal inference associating greater health concerns (e.g., number of chronic health conditions) and care utilization (e.g., number of office visits) cannot be established using the data collected for this study. Future research is needed to explore the causal factors of why patients seek and subsequently use portal services.