Computerized clinical decision support systems for primary preventive care: A decision-maker-researcher partnership systematic review of effects on process of care and patient outcomes

This systematic review addressed two questions: Do CCDSSs improve process of care or patient outcomes for PPC, and what are the costs, safety, and provider satisfaction with CCDSS for PPC?

The review team included a partnership between McMaster University's Health Information Research Unit (HIRU), the senior administration of Hamilton Health Sciences (a large Canadian academic health sciences centre) and Local Health Integration Network (the regional health authority that includes Hamilton), and clinical service chiefs at local hospitals. Decision-maker partners were included in discussions about data extraction for, and interpretation of, factors that might affect implementation. The decision-maker-researcher partnership hypothesized positive effects of CCDSSs in both process of care and patient outcomes regarding PPC, methodological improvement in testing of CCDSSs over time, cost savings, and improved safety and provider satisfaction with CCDSS use.

We previously described our search methods up to 2004 [6] and for this update [13]. Briefly, for the latest update we used a comprehensive search strategy to retrieve potentially relevant RCTs from MEDLINE, EMBASE, Ovid's Evidence-Based Medicine Reviews, and the Inspec bibliographic database from 1 January 2004 to 8 December 2008; a further update was conducted to 6 January 2010. We performed duplicate screening of eligible RCTs and independent data-extraction using piloted forms that were constructed with our decision-maker partners; a third reviewer resolved disagreements. Inter-reviewer agreement on study eligibility was measured using the unweighted Cohen's kappa (κ), and was excellent (κ = 0.93; 95% confidence interval [CI], 0.91 to 0.94) over all applications. Study authors confirmed extracted data for 88% (36/41) of the studies included in the PPC review.

We included RCTs (including cluster RCTs) published in any language that compared the effects of care with a CCDSS for PPC, used by healthcare providers, with care without a CCDSS. Outcomes included processes of care and patient outcomes. We only considered RCTs because this method minimizes the risk of biased allocation, and there has been increased publication of RCTs since our 2005 review [6].

For PPC interventions, patients had to be free from the illness to be prevented (e.g., a specific strain of influenza) but could be seen in any setting, including acute healthcare. CCDSSs that provided only computer-aided instruction, performed actions unrelated to clinical decision making (e.g., CCDSSs for diagnostic performance against a gold standard), or evaluated CCDSS users' knowledge or performance in clinical simulations were excluded.

We excluded studies where PPC interventions were merged with a complex set of other interventions (e.g., chronic disease management) and those that did not focus on PPC (e.g., screening of medical errors). We did however include one study that evaluated a CCDSS for influenza vaccination in asthmatic patients because it provided evidence about the independent effects of the intervention on vaccination rates [1].

Independent reviewers extracted key data in duplicate, including study methods, CCDSS and population characteristics, possible sources of bias, and outcomes. Primary authors of each study were asked to review the extracted data for their study and offer comments on the extracted data.

Details of our quality assessment of included RCTs are published elsewhere [13]. RCTs were scored for methodological quality on a 10-point scale (an extension of the Jadad scale [14]) with scores ranging from 0 for the lowest study quality to 10 for the highest quality.

Researchers and decision-makers selected outcomes that were relevant to PPC from each study before evaluating intervention effects. We used RCTs as the unit of analysis to assess CCDSS effectiveness. A process of care outcome represents the delivered quality of care, while a patient outcome represents the directly measured health status of the patient. We used a dichotomous measure of effect and defined a CCDSS as effective (positive) when there was a significant (p< 0.05) improvement in the endpoint specified as main or primary by the authors or, if no primary endpoint was specified, the endpoint used to estimate study power, or, failing that, ≥50% of multiple pre-specified endpoints. When no clear pre-specified endpoints existed, we considered a CCDSS effective if it improved ≥50% of all reported outcomes. Studies that included ≥1 CCDSS treatment arm were considered effective if any of the treatment CCDSS arms was evaluated as effective. These criteria are more specific than in our 2005 review [6], and the effect assignment was adjusted for some studies from that review.

We used descriptive summary measures for data including proportions for categorical variables and means (± standard deviations) for continuous variables. When reporting results from individual studies, we cited the measures of association and p-values as reported in the studies. We considered methodological rigor and scientific quality of the included trials to analyze data and formulate conclusions. We did not pool data or compare studies using effect sizes because of study heterogeneity in populations, settings, interventions, and outcomes. A sensitivity analysis was conducted to assess the possibility of biased results in studies with a mismatch between the unit of allocation (e.g., clinicians) and the unit of analysis (e.g., individual patients without adjustment for clustering). Success rates comparing studies with matched and mismatched analyses were compared using chi-square for comparisons. No differences in reported success were found for either process of care outcomes (Fisher's exact test, 2P = 1.0) or patient outcomes (Fisher's exact test, 2P = 1.0). Accordingly, results have been reported without distinction for mismatch.

Additional file 2 Table S2 shows that 20/41 (49%) CCDSSs were integrated with an electronic medical record [1, 17, 25, 27, 29, 31, 32, 34‐36, 39, 41‐46, 49, 50, 52, 56, 59] including at least five also integrated with a computerized order entry system [1, 32, 42, 49, 56] and 21/41 (51%) were stand-alone computer systems [15, 16, 18‐22, 24, 26, 28, 30, 33, 37, 38, 40, 47, 48, 51, 53‐55, 57, 58]. The data entry method varied across systems, with a non-practitioner decision-maker entering data on 29/39 (74%) studies [1, 15, 17, 21, 23‐25, 27, 29, 31, 32, 34‐55, 59] and automatic entry through electronic health records in 15/39 (38%) cases [1, 17, 27, 29, 31, 34‐36, 41, 42, 46, 49, 50, 56, 59]. In all but one study [26], physicians used all PPC CCDSSs, either solely or shared with other healthcare providers including trainees [1, 25, 28, 29, 39, 41, 42, 46‐48, 52], advanced practice nurses [1, 17‐19, 30, 50, 59], physician assistants [18, 19, 33], and social workers [26]. No single study completely described the CCDSSs interface. Delivery methods for CCDSS recommendations varied: 17/40 studies (43%) reported use of a desktop or laptop computer [1, 26‐32, 34‐36, 42, 49, 50, 56‐59]; 10/40 (25%) used existing non-prescribing staff [17, 28, 33, 40, 41, 43‐46, 52, 53, 59]; 8/40 (20%) used research project staff [15, 20‐22, 24, 38, 39, 47]; and the remaining studies used other methods, including personal digital assistants [18, 19] and paper reports [50]. CCDSSs were pilot tested in 15/33 studies (45%), providers received training on the CCDSS in 23/35 trials (66%), and the CCDSS provided suggestions at the time of care in 36/41 studies (88%). Investigators also developed the CCDSS in 28/35 studies (80%).

Twenty-nine of 41 trials (71%) were conducted in the USA [1, 16‐26, 28, 29, 32, 33, 36, 39‐45, 47‐50, 52‐55, 59], 5/41 (12%) in the UK [30, 34, 37, 51, 57, 58], 3/41 (7%) in Canada [15, 38, 46], and 1/41 (2%) each in Italy [31], New Zealand [35], Spain [27], and The Netherlands [56]. Forty-four percent (18/41) of trials were published after the year 2001 including 14/41 (34%) published after the year 2005. Eighty percent (33/41) of trials reported a public funding source [1, 15‐24, 28‐30, 33‐35, 37, 39‐47, 49‐59], 7% (3/41) a private source [27, 36, 48], 2% (1/41) both public and private [38], and 10% (4/41) did not report these data [25, 26, 31, 32]. Twenty-two trials (54%) took place mainly in primary care settings [1, 18‐20, 22, 23, 27, 30, 31, 33‐38, 40, 49‐51, 53‐58] while 19 trials (46%) were undertaken in a combination of hospitals, specialist clinics, and primary care, or in academic centres [15‐17, 21, 22, 24‐26, 28, 29, 32, 36, 39, 41‐48, 52, 59]. In all but one [1] of the 41 trials, the patients were adults or elderly.

Many CCDSS interventions for PPC were tested in the included studies. Twenty-two (54%) studies evaluated multifaceted interventions with ≥3 preventive care components [15, 18‐23, 28, 30, 34, 35, 37, 39‐41, 46, 47, 49‐51, 53‐55, 57, 58], including educational sessions on preventive interventions and the CCDSS, supply of materials to clinicians and/or to patients, assessments of patient and clinician attitude towards health conditions and/or the CCDSS, audit and feedback of clinician performance, academic detailing, telephone reminder to patients, elimination of out-of-pocket expenses to patients, and use of local clinician leaders. Eleven (27%) trials assessed two components [1, 16, 24, 27, 31, 33, 36, 38, 42, 48, 52], and the remaining eight (21%) assessed the effectiveness of a CCDSS with one component, typically a reminder (e.g., printed, audio, or visual) [17, 25, 26, 29, 32, 43‐45, 56, 59].

Table 1 (see Additional file 4 Table S4 for detailed information) shows that all trials assessed the effects of CCDSSs on processes of care. Twenty-five of 40 (61%) studies showed an improved process of care using our dichotomous measure; three of those studies also included CCDSS treatment arms that did not improve process of care [26, 41, 56]. Four of 14 (29%) studies showed improved patient outcomes. Only 13 (32%) studies reported both process of care and patient outcomes.

CCDSSs improved the screening or referral of patients with breast, cervical, ovarian, colorectal, and prostate cancers in 5/10 (50%) trials [20, 22, 30, 39, 40]. Emery et al.[30] showed improved rate of appropriate referrals to regional genetics clinics by primary care clinicians regarding familial cancers (i.e., breast, ovarian, and colorectal cancers). Conversely, Burack et al. demonstrated no effects for reminders for mammography screening [21] and screening mammography and pap smears tests in primary care [24]. Only three studies assessed patient outcomes, and none demonstrated effects [30, 49, 57, 58].

In rural and urban primary care settings and hospitals, clinicians received CCDSS recommendations for various interventions in adult and geriatric patients including cancer screening, cardiovascular (CV) risk assessment, vaccination, tuberculosis skin tests, counselling, patient education, prophylactic antacids, calcium supplements, and screening for functional independency. Six (60%) trials reported improved process of care [28, 29, 33, 41, 46, 52] including one trial demonstrating higher ordering rates for pneumococcal vaccination (35.8% of patients in the intervention group versus 0.8% of those in the control group, p<0.001), influenza vaccination (51.4% versus 1.0%, p<0.001), prophylactic heparin (32.2% versus 18.9%, p<0.001), and prophylactic aspirin at discharge (36.4% versus 27.6%, p<0.001) in a teaching USA hospital [29]. Conversely, Overhage et al.[42] showed that a CCDSS for 22 preventive care measures in hospital inpatients did not change clinicians' actions for such measures. Only two studies assessed patient outcomes, but neither showed effects [16, 34].

CCDSSs helped clinicians detect and treat dyslipidaemia, diabetes, smoking, obesity, hypertension, and renal diseases as well as calculating coronary risk factor profiles. All nine trials reported improved process of care of which three targeted screening and treatment of dyslipidaemia in primary care [18, 19, 27, 56]. Five trials reported patient outcomes; three showed positive effects [17, 38, 43‐45], and two [27, 54, 55] showed no effects.

Studies in this category covered various CCDSSs for screening and management of mental health conditions in primary, secondary, and tertiary care settings. Only one trial [47] used cluster randomization (see Additional file 1 Table S1) and all but one trial [51] were conducted in a single site. In all six trials, the CCDSSs were stand-alone systems (see Additional file 2 Table S2), and four trials included patient-completed computer-based instruments [15, 37, 48] or paper-based post intervention surveys [47]. Two trials showed positive effects in process of care, including Ahmad et al. [15] who reported that a CCDSS increased opportunities to discuss intimate partner violence in primary care (adjusted relative risk [RR], 1.4; 95% CI, 1.1 to 1.9) and increased its detection (adjusted RR, 2.0; 95% CI, 0.9 to 4.1). Three trials reported on patient outcomes including one with positive [51] and two with no effects [37, 47].

CCDSSs for tetanus, hepatitis, pneumococcal, measles, and influenza vaccinations in children, adults, and the elderly in primary care only improved influenza vaccination among the elderly in one trial [25]. All trials compared 'usual care' with CCDSS alone [25, 32] or in addition to an educational session [1], and no trials assessed patient outcomes.

Two trials reported improved process of care [36], including one that also assessed patient outcomes but found no effects [59]. All studies in this category compared CCDSSs with 'usual care' although in one study, all providers were educated on the importance of HIV screening and trained on CCDSS functions [50]. Lafata et al.[36] showed that, among insured women 65 to 89 years of age, reminders mailed to patients, either alone or with physician prompts, improved osteoporosis screening and treatment rates. Zanetti et al.[59] showed improved intraoperative redose of prophylactic antibiotic, but it was underpowered to demonstrate effects on patient outcomes.

Costs of developing, implementing, and maintaining a CCDSS were partly reported in 6/41 (15%) trials [16, 27, 33, 46, 54, 55, 57, 58]. Among these six studies, when a CCDSS was used, two found costs of care were significantly less [27, 46], three yielded increased cost of care [16, 33, 54, 55], and one showed varied cost minimization data [57, 58]. Rosser et al.[46] did not report detailed costs, although the physician reminder was reported to be the most cost-effective method of improving preventive services, followed by letter reminder, and telephone reminder. Cobos et al.[27] showed that a CCDSS for management of patients with dyslipidaemia including those without coronary heart disease had no effects on lipid profiles, but saved 24.9% in treatment cost per patient and 20.8% in total costs, including costs for physician visits, laboratory analyses, and lipid-lowering drugs. Apkon et al.[16] showed a difference of US $91 more patient resource usage (ambulatory visits, laboratory tests, diagnostic imaging, and pharmacy use) for multiple preventive care procedures in the CCDSS group than usual-care group. Frame et al. showed that a CCDSS for multiple preventive procedures did not increase revenue generation or the number of office visits to a fee-for-service clinic despite its positive effects on provider compliance to such activities [33], and Unrod et al. found implementation costs for CCDSS, including equipment, training, and staff costs, increased costs for smoking cessation counselling [54, 55]. Wilson et al. presented software development costs and the marginal cost for each additional compact disc [57, 58].

Only two (5%) trials reported CCDSS adverse events; one demonstrated greater risk for over treatment than for under treatment in dyslipidaemia because all patients were screened, including low-risk patients who would not normally be screened [18, 19]. Zanetti et al.[59] reported four in 449 (1%) inappropriate alerts to redose prophylactic antibiotics during cardiac surgery and one unnecessary intraoperative redosing [59].

Six (15%) trials reported on provider satisfaction with CCDSSs [15, 16, 30, 40, 49, 50] including two trials [30, 49] on cancer screening where most providers were satisfied with CCDSSs use. Only Apkon et al.[16] reported provider and patient satisfaction when a CCDSS was used, but showed no significant differences between groups in patient satisfaction results and mixed providers' satisfaction within the group using a CCDSS for 12 preventive care interventions.

We built on our previous review by including only RCTs published in any language, and using duplicate study identification, data abstraction, and study evaluation. Our current focus on RCTs provided a more scientifically robust estimate of CCDSS effectiveness, although the potential for publication bias was not assessed. We confirmed our abstractions with primary authors. We collaborated with clinical decision-makers in extracting and analyzing data, and formulating and disseminating findings. We considered the methodological rigor of trials. We could not use meta-analysis to pool effect sizes because included RCTs presented a considerable variety of systems and outcomes. The vote counting approach that we used to summarize study results does not take into account the size or quality of individual studies.

Our decision-maker partners indicated concerns regarding insufficient reporting on infrastructure and contextual factors in which CCDSSs were evaluated, including impact on clinician workflow and the interoperability across different systems. An assessment of available data across all studies in the review set (166 RCTs) is in progress.

Although trial methods improved over time, our review was hampered by the limitations of the primary studies. CCDSSs should target processes of care that have already been shown to be validly related to improved patient outcomes, but not all studies reported the validity of the targeted processes. In addition, most trials did not assess patient outcomes, and even the trials that did were too small to detect clinically important effects. Further, information on study duration was often missing, limiting our ability to assess sustainability of CCDSS effects.