Features predicting the success of computerized decision support for prescribing: a systematic review of randomized controlled trials

Potentially important features were identified primarily from our own e-health research program and clinical experience [16] as well as reviews of the literature [11, 12, 17‐19]. A list of 40 features was generated, 12 features were ultimately removed during the review process due to lack of reporting or inability to assess. The remaining 28 features were grouped into 4 categories: Pure technical features, Technical/user interactions, Logic of decision support, and Developmental and Administrative environment (see Figure 1; Additional File 2).

We included reports of RCTs of RxCDSS published in English. We considered a RxCDSS to be an intervention which utilized a computer to analyze patient-specific information to advise a prescriber (primarily a physician) or pharmacist when they were writing or filling a prescription, respectively. Although the decision support itself had to be generated electronically, the support could be delivered by any means (e.g. computer terminal, fax, mail, patient record insert). We only considered systems which intervened before a drug therapy had been chosen by a physician, or had the ability to suggest alternate therapies (i.e. a drug different then that initially prescribed) to be a RxCDSS. These are the more challenging decisions for which to intervene and change. Systems whose sole purpose was to offer 'fine-tuning' advice on a pre-defined therapy – usually dose modification – were not included in this review. Systems primarily focused on diagnosis, vaccination, or nutrition, were also excluded.

We searched the databases Medline, EMBASE, CINAHL, and INSPEC for articles published since the earliest entry to June 2008. The detailed search strategy is shown in Additional File 3. The search was individually tailored for each database, with search terms from domains of study methodology, general CDSS terms and RxCDSS identifiers. These included: randomized controlled trial, artificial intelligence, decision support systems, computer-assisted therapy, computerized medical records system, reminder systems, hospital information systems, computer systems, decision support techniques, ambulatory care information systems, computer assisted decision making, medical errors, therapeutic uses, drug therapy, drug information services, drug interactions, drug monitoring, guideline adherence, medication systems, drug administration schedule, drug costs, drug dose-response relationship, and computer assisted drug therapy. A pilot test was completed to ensure that known relevant studies were identified. All citations obtained were downloaded into Reference Manager, version 11.0.

The titles of all returned studies were reviewed, and those potentially matching our definition of a RxCDSS were kept. Next, the abstracts were assessed independently by two reviewers to determine whether the studies met the inclusion criteria. Disagreements between reviewers were resolved by consensus and, if necessary, by arbitration of a third reviewer. If uncertain whether a study met the inclusion criteria, it moved to the next stage of assessment in order to decrease the likelihood that a relevant study was overlooked.

During full-text review, articles were once again reviewed independently using detailed data extraction forms which extracted details on methods, study validity, study outcomes and features. Before use, the data extraction forms were critiqued for face validity by a panel of methodologists experienced in systematic reviews and CDSS. The forms were also piloted to improve usability.

Methodological quality of each RCT was assessed using a modified scale adapted from Garg et al [11]. Our rating system assessed studies on four potential sources of bias: unit of allocation, presence of baseline differences between groups potentially linked to study outcomes, objectiveness of outcome, and completeness of follow-up. Each source of bias was rated on a scale of 0 to 2, with 2 indicating the highest methodological quality. The results of this evaluation were summed with a maximum total possible score of 8.

Study outcomes were assessed for success in each of our three domains of focus: 'Implementation', 'Change in Health Care Provider Behaviour', and 'Change in Patient Outcomes'. Implementation was considered successful if the RxCDSS was successfully introduced and utilized by the clinical staff. A successful change in provider behaviours required reporting of changes such as a decrease in inappropriate prescribing or a change to a more cost-effective therapy. Lastly, impact on patient outcomes was considered successful if the study reported improvements in patients' health (e.g. decreases in morbidity or mortality). These domains of outcomes were hypothesized to be conditional and hierarchical, with success required in implementation before changes in provider behaviour would be noted, and so on. Since the concept of minimal clinically important difference in this area of research remains undefined, outcomes were assessed for statistical significance as reported by the original study [20].

Each RCT report was reviewed several times independently to ensure complete abstraction of features of interest. Each feature on our list was rated for each study as present, absent, or could not assess. 'Could not assess' was used when, even after extensive discussion, reviewers could not agree that a feature was present or absent. For the purposes of analysis, features that could not be assessed were considered absent.

Consensus was obtained as described above for methodological quality scores, RxCDSS success and presence/absence of features. Descriptive statistics were used to characterize the studies included, their degree of success, and the number of features reported. Inter-rater reliability for selected methodological quality score, success and features present or not, was calculated and reported as a kappa statistic. We planned to measure the association between our three-tier definition of success of the individual studies and the feature list using univariate binary logistic regression. This method requires roughly equivalent numbers of successful and unsuccessful studies per outcome. Statistical analyses were conducted using SAS 9.1 (Cary, North Carolina).

The 41 RCTs involved a total of 612,556 patients (range 169–407,460 per study) and 2963 providers (range 17–334 per study). The mean methodological score was 5.9 with a range of 2–8, indicating generally good quality studies. Twenty-three studies (56.1%) used a RxCDSS in an outpatient general practice or internal medicine setting, 10 (24.4%) in inpatient hospital wards or emergency rooms, 5 (12.2%) in pharmacies and 3 (7.3%) in specialty clinics (2 for paediatrics and 1 for diabetes). The systems addressed a variety of problems – cardiovascular care (36.6%), general/internal medicine (29.3%), diabetes (9.8%), respiratory disease (9.8%), otitis media (7.3%), depression, osteoporosis and infectious disease (2.4% each). Nineteen (46.3%) of the RxCDSS were integrated with drug order entry, 16 (39.0%) with management/electronic health record (EHR) software and 9 (22.0%) also printed the suggestions. While 20 (48.8%) of these systems appeared to be developed by independent vendors, 21 (51.2%) were developed within an (experienced) home EHR environment such as the Regenstreif Medical System [61] and the Veterans Health Information System [62].

The studies most often employed a traditional 2-arm parallel design (31, or 75.6%), although 6 studies (14.6%) used a 3-arm design and 4 (9.8%) used a 2 × 2 factorial design. The nature of the control arm of studies was also mixed but most often was labelled as usual care (35 or 85.4%). The remaining 6 studies employed a control group intervention that was felt to be much less effective such as distribution of general treatment guidelines or education, reminders regarding an unrelated condition or a RxCDSS that operated with reduced functionality.

Systems were successfully implemented in 37 trials (90.2%) [21‐23, 25‐30, 32, 33, 35‐60], provider behaviour changed in 25 (61%) [21‐23, 25, 26, 28, 29, 32, 33, 35, 37, 41, 43‐45, 47, 48, 50, 51, 53, 55‐57, 59, 60], and patient outcomes improved in 5 (12.2%) [28, 44‐46, 59]. Only 23 RCTs reported on this important outcome and all of the studies reporting success with a patient outcome were published after 2005. Of those studies reporting improvements in patient outcomes, Kucher et al. [44] noted a computer program encouraging prophylaxis against deep-vein thrombosis and pulmonary embolism reduced the risk of these two events by 41% at 90 days (P = 0.001). Javitt et al. [28] found a system that scanned administrative claims and clinical data to detect physician errors reduced hospital admissions by 19% in the intervention group relative to the control group (P < 0.001). Feldstein et al. [59] used patient-specific guideline reminders within the primary care physician's EHR to improve osteoporosis management for HMO patients who had suffered a previous fracture. Lester et al. [45] used e-mail reminders to physicians to encourage the increased use of lipid-lowering medication for patients with coronary artery disease. Roumie et al. [46] examined strategies to improve blood pressure control and found that patient education combined with computerized patient-specific alerts to providers was superior to provider education alone.

These associations are shown in detail in Additional File 1. Overall, studies mentioned an average of 17 features each (see Additional File 6). No trial concentrated on the features and their relationship with success or failure of the intervention nor did any trial systematically identify all of the features of their intervention. Only one study [21] isolated a specific RxCDSS feature through their randomization procedure. In this study, no benefit was found by adding bibliographic citations to electronic reminders. Another study [42] randomized two groups to guideline-based suggestions for treating congestive heart failure versus these suggestions plus others based on symptoms gleaned from the linked EHR, and found that the intervention group fared worse in terms of more hospitalizations. As this study did not have a control arm without a RxCDSS, it was not considered further in the analysis of features.

Because of the small number of studies, the lack of rigorous attention to features descriptions by the trials and, especially, the lack of diversity of outcomes across the 3 domains, we were unable to statistically evaluate whether there are features more associated with success than failure using logistic regression as originally planned. In general, the features most prevalent across the 40 remaining studies included: support of the user's task at hand (95%), provision of decision support at the time and place of decision-making (85%), provision of a recommendation rather than just an assessment (85%), automatic provision of decision support as part of clinician workflow (78%), integration with charting or order entry (75%), and convenient locations for the computers (68%). However, with few exceptions, the prevalence of these features was similar between successful and unsuccessful studies when examining implementation, provider behaviour and patient outcomes. For patient outcomes, there was reasonable separation for the prevalence of a few features. The features which all of the successful studies [28, 44‐46, 59] shared and which most of the unsuccessful studies did not, included: provision of a recommendation rather than just an assessment, justification of decision support via provision of research evidence and the system uses data standards that support integration. However these results should be considered hypothesis-generating at best, given the low number of studies which successfully altered patient outcomes and the poor general reporting of features.