Patient-specific computer-based decision support in primary healthcare—a randomized trial

The setting was the primary healthcare center of Sipoo, which was selected from regular users of the Mediatri EPR system. The center comprises 48 HCPs: 15 physicians, 24 nurses and 9 other HCPs (physiotherapists, ward nurses, a psychologist), described in detail elsewhere [24]. The HCPs used the EPR system during outpatient consultations as well as on an inpatient ward typical of Finnish primary care.

We used a parallel randomized controlled trial design, with patient identification (ID) numbers in the EPR system as the unit of randomization. We made use of the Finnish Personal Identity Code (PIC), by which each individual can be specifically identified [25], to produce anonymized study IDs based on the PICs. All patients who were listed as undergoing occupational healthcare were excluded for legal reasons (Figure 1).

The study started in July 2009 and ended in October 2010.

We developed a unique method for population-based outcome data gathering from the EPR archive, the Repeated Study Virtual Health Check (RSVHC). During an RSVHC, the EPR archive sent to the EBMeDS service structured patient data (diagnoses, medications, and laboratory results) on the base study population (request), and the service generated all reminders triggered by these data and returned them (answer). The RSVHC was planned to be performed weekly at night. Actually, one to five RSVHCs were performed per month. The requests and answers of each RSVHC were stored automatically in a log file located in the EPR server, to be exported to the study register and analyzed at the end of the study period.

The reminders were written by the EBMeDS editorial team using medical evidence embedded in the sets of Duodecim guidelines and other sources, and linked with evidence-based decision support (DS) rules [26]. In April 2009, the 107 available DS rules were piloted in Sipoo. Subsequently, the chief medical officer of the health center decided which of the DS rules should be implemented. These totalled 96.

Here is an example of an implemented DS rule that conforms to if–then logic: ‘Metformin is the first choice oral hypoglycaemic agent in type 2 diabetes’ (DS rule 16).

The DS rule is implemented if the diagnosis in the EPR is type 2 diabetes. First, the rule checks whether the medication list on the EPR contains metformin. If it does not, the rule then checks for the plasma/serum creatinine value from the EPR laboratory results. If the glomerular filtration rate (GFR) is in the normal range, then reminder one, ‘Type 2 diabetes – start metformin’ is shown on the screen. If the GFR is <60 ml/min, then reminder two, ‘Type 2 diabetes—start metformin, note GFR’ is shown. If the GFR is missing or out of date, then reminder three, ‘Type 2 diabetes—check renal function and start metformin’ is shown [26].

See Additional file 1 for more examples of DS rules and reminders targeting diabetes patients, a large patient group in primary care. HCPs can easily check background information and evidence behind each reminder by clicking the reminder and opening the references.

This was a register-based study using the EPR data without any direct contact with patients.

The first step in the data collection comprised 52 RSVHCs carried out between July 2009 and October 2010 in the base population. At the end of the study, the population was 17,541 (total number of patient IDs in the study register). Data in the RSVHCs were structured patient-specific information (diagnoses, medications, laboratory results), and the triggered patient-specific reminders at each time point were stored in the study register.

In the second step, using the study ID, the earliest date was defined when the patient’s EPR had been opened during the study (start of individual follow-up). All EBMeDS procedures (requests and answers) which actually took place during the study were stored in monthly log files on the EPR server. One log file (April 2010) was missed because of technical problems. A baseline date was determined individually for each study patient as the date of the first opening of the EPR during the study period, and the patient-specific follow-up started from this date.

The third step involved linking the information from steps one and two. These final data comprised patient-specific information from the individual baseline date to all RSVHC points where the patient was followed up.

The swine flu epidemic, with the ensuing universal vaccination procedures, occurred between September 2009 and February 2010. We excluded from the monthly log files patients who received only the swine flu vaccination during a short visit (5 to 10 minutes) with a nurse. According to the nurses [27], nothing else was checked, including triggered reminders.

The intervention consisted of the patient-specific reminders being shown to the HCP on opening and using the EPR. A concrete example, the reminders for one diabetes patient, has been published previously [27]. Short versions of the triggered reminders, for example, ‘Type 2 diabetes—time for nephropathy screening’ , were shown automatically on screen. The full version of the reminders could be seen when the HCP hovered the cursor over the reminder, for example, ‘This patient has type 2 diabetes and no screening for microalbuminuria has been carried out during the last year. Annual screening for microalbuminuria is recommended in type 2 diabetes’. See Table 1 for examples of the reminders according to ICD-10 diagnosis groups. All study reminders are available in Additional files 2, 3 and 4.

There are four different EBMeDS decision support service functions: reminders, guideline links, a clinical virtual health check, and drug alerts (Table 2). Reminders and drug alerts are triggered automatically, but the guideline links and clinical virtual health check functions need active querying. Here, we focused only on the automatic reminder function. The interface of the integrated systems (EPR and EBMeDS) was designed by the EPR system vendor.

The number of reminders selected for the study was 154, based on 73 DS rules. After piloting, we had to exclude 23 of the original 96 DS rules (not triggered, n = 10, or not calculable, n = 13). In practice, 14 additional DS rules failed to function as planned (missing laboratory codes or unexpected changes in the EPR system after updates). After excluding a further 38 reminders based on these 14 DS rules, the analyzable final maximum number of different reminders was 116 (Figure 2).

The primary composite outcome measure was the change in the numbers of all reminders triggered in the target population over 12 months of individual follow-up. As secondary outcome measures, we explored the changes also after three and six months of follow-up.

We planned to include all of the Sipoo patients’ EPRs in the study, and estimated that at least 50% of the population would contact the health center during one year, based on available data on visits to primary healthcare centers in Finland, and local statistics [28]. This translated into an approximation of 10,500 participants in the final study sample. The accumulation of the study participants (n = 13,588) is shown in Figure 3.

A single ratio procedure randomized the base population of the health center at the beginning of the study to an intervention and a control group of the same size without any other criteria. The procedure was done once per individual by a computer using a mathematical formula based on the PIC of each patient in the EPR system, and assigning each patient a unique study ID number. The forthcoming patients (for example, new inhabitants of Sipoo) were randomized according to the same procedure. The procedure was performed by a person outside the study group who also retained the key of the formula linking the PIC to the study ID number.

The randomization was masked from the patients, the HCPs, and the study group. However, when the HCP was shown the patient-specific reminders on screen, he or she knew that the patient belonged to the intervention group. The study group first opened randomization after the data collection period.

Baseline characteristics of the intervention and control group with ancillary analysis for triggered reminders were performed using means and standard deviations or frequencies and proportions (Table 3).

To investigate the effect of the intervention on patient care, the outcome variable was the number of triggered reminders in each RSVHC. Because the data were right-skewed and the variance was greater than the mean, the negative binomial model provided a better fit to the data and accounted for over-dispersion better than a Poisson regression model [29]. We used negative binomial regression to model the number of triggered reminders at 12, 6, and 3 months follow-up times (Table 4, models 1 to 3). The negative binomial model included a variable (group) to indicate the difference between groups at baseline and a variable (time) to indicate the changes in the number of triggered reminders over time. The difference in change in the number of triggered reminders across the intervention between the two groups was tested using an interaction term between group and time. The exponent of the coefficient of the interaction term is the incidence rate ratio (IRR), i.e., an estimate of the relative difference in percentage change in the number of triggered reminders in the intervention group, compared with the control group. We also added to the models some potential confounding variables, such as age, gender, number of diagnoses, and number of medications.

To account for the within-participant correlation between repeated measures, we used Generalized Estimation Equations (GEE) by using the STATA software package (version 12.0 for Windows). Liang and Zeger proposed the GEE approach in 1986 to deal with impractical probability distribution in handling correlated responses [30]. We used different correlation structures (exchangeable, first-order autoregressive and unstructured) to account for the correlation within each unit. All models were evaluated in terms of how well they fitted the data using the quasi-likelihood under the independence model information criterion (QIC) for model selection [31]. The model with the lowest QIC was selected as the final model. Robust standard errors were used for all GEE-fitted models. IRRs were presented with 95% confidence intervals (95% CI) and p-values. We defined <0.05 risk of error as the significance p-value.

There are a number of possible explanations for the results. First, the setting was one health center with 15 physicians making clinical decisions on diagnosis, medication, and laboratory tests. Because each physician took care of patients in the intervention group as well as in the control group, contamination was possible in so far as the physicians could have learned to treat the control group patients according to the reminders for the intervention group patients. This possible learning effect can be presumed, decreasing the trial effect. Therefore, the present results are a conservative estimate, and in future trials, a cluster randomization of several study sites or a randomization of HCPs would be preferable.

Second, the set of study reminders was chosen by the study group and may have addressed the HCPs’ needs insufficiently. Tailoring the guidance to HCPs’ needs has been indicated as a key issue for successful implementation [5, 7]. Local HCPs, not only the Chief Medical Officer, may have to be involved for an adequate understanding of their needs as the starting point for developing and implementing reminders, as had been indicated previously [34]. Further, competence-based individual tailoring could be helpful.

Some of the most common and important reminders, including those warning of high LDL cholesterol, had to be excluded from the analysis because the codes for laboratory tests had changed over time (which the research group was not aware of), and only the old codes were interpreted by the rules, resulting in false reminders based on old (and not the most recent) test results. This may have resulted in mistrust of the reminders among the HCPs, and poor compliance with all reminders.

Many patients were seen by nurses rather than physicians, but the actions suggested by the reminders (like ordering tests or medications) could only be taken by physicians. The nurses may not always have consulted physicians after they saw the reminders, resulting in no action.

According to a meta-regression of 162 randomized trials [35], the odds of success of computer-based decision support are greater for systems that require HCPs to provide reasons when overriding advice than for systems that do not. The odds of success are also better for systems that provide advice concurrently to patients and HCPs. The intervention system possessed neither of these features.

There was a delay in implementing the EBMeDS service due to technical problems. This necessitated retraining the HCPs, which for practical reasons was delayed and took place in February 2010, eight months after the introduction of the service [24]. This delay from the introduction of the service to the training of the HCPs in its use may partly explain the results.

The reliable performance of the data-gathering methods gives confidence that they can be applied successfully in future studies and combined with a more extensive use of decision support, for example the monitoring and auditing care of such large patient groups as those with type 2 diabetes.

We managed to randomize all patients of the health center as planned to two comparable study groups, indicating a high validity of the results [36‐38]. However, the results may not be generalizable to other primary care environments, where the EBMeDS service could be more vigorously implemented. Another set of physicians in another health center could use the reminders much more or much less than the present 15 physicians, and therefore the results could be totally different. Also, the integration of the Mediatri EPR and the EBMeDS service was unique. That reminders were triggered by only below a third of the participants may be related to recording issues, or to the timing of the successive accessing of the EPR. Further exploration is warranted.

The EBMeDS service, developed using the best available evidence [9, 10], aimed to offer recommendations to HCPs’ workflow across several conditions in primary care practice. During the trial, patient-specific reminders were triggered systematically but had only limited effects on patient care (our secondary outcome measure after six months’ individual follow-up time). Our results reaffirm previous evidence [39‐41] that implementation of computer-based decision support is problematic. HCPs seem to accept the service in principle [27, 42], but in practice they may neglect using the reminders for many practical reasons.

The intervention itself is complex, as reminders have different purposes in accordance with decision support rules (Table 1 and Additional files). Some provide advice on diagnosis or medication or laboratory test decisions, for example ‘Atrial fibrillation—start warfarin?’ (DS rule 457), and some are reminders to follow patient measures, for example ‘Hypertension—time to check blood pressure?’ (DS rule 578). The expected time interval between consultations and changes in the patient record differs across the reminders: some changes take place quickly, even in one visit, for example, as the HCP records a new diagnosis or prescribes medication. By contrast, some changes need more time, at least until the next visit or laboratory test. In the latter case, the time interval between the triggering of the reminder and the measurement of the outcome was too long for reminders to show differences between the intervention and control groups. Groups of reminders and individual reminders should also be evaluated separately, probably after careful tailoring of the time to outcome, in order to determine the types of reminder that have an effect and when this should be measured.

Environmental issues included functional changes in the health center, such as turnover of staff, and the coincidence of the swine flu epidemic with the trial, which could have influenced the study. Moreover, the physicians decided that ICD-10 diagnosis classification would be used systematically from spring 2009. The use of the classification was made a mandatory function for recording patient data in each encounter. This might be a key reason for changes in data entry during the trial. However, in a randomized design both groups would have been affected similarly. The reason for a patient visit may not have had any bearing on the triggered reminders, resulting in HCPs ignoring them. In fact, our feasibility study [27] indicates that missing or outdated patient data on, for example, medication, resulted in needless or wrong reminders. These had to be checked in the EPR, which took time.

We can speculate on at least three specific issues. First, our hypothesis was based on an optimistic estimation of the potential consequences of triggered reminders. We assumed that if HCPs received patient-specific reminders they would act on them, and that this automatically would decrease the number of future reminders. We did not recognize that other factors in patient care—above all, changes in patient data recording—could have an opposite effect: for example, recording a new diagnosis or medication would trigger new reminders instead of decreasing the number of reminders triggered. This confirms previous findings [43, 44].

Second, our choice of the primary outcome measure and its timing was based only on our understanding, without any actual evidence. Despite extensive research during the planning of the trial, we could not find a study to help us with this. Choosing the primary outcome measures is not easy [36].

Third, the analysis consisted of three models (Indiv_MO12, Indiv_MO6, Indiv_MO3) based on the different follow-up periods of the participants. In order to be followed up for 12 months and be included in the measurement of the primary outcome, a patient had to have a first contact by the end of October 2009. The starting date could be as late as April 2010 for inclusion in the six-month follow-up period and July for inclusion in the three-month period. Most notably, the groups of individuals with 3, 6, or 12 months’ follow-up differed in the numbers of diagnoses and medications at baseline (Table 3). The higher numbers of diagnoses, medications and triggered reminders in the 12-month group indicate more chronically ill patients in this group than in the other groups. The effects of the marked improvement in diagnosis coding during the study are difficult to assess, because the influence of reminders triggered for the intervention group may differ from that in the control group. Some of the reminders indicated to the HCP that diagnoses had not been coded. This may have resulted in more comprehensive coding of some diagnoses in the intervention group, which further may have resulted in more triggered reminders than in the control group, because many reminders could be triggered only if a specific diagnosis was present. Adjusting for the number of all diagnoses cannot fully remove this confounding factor.