Can computerized clinical decision support systems improve practitioners' diagnostic test ordering behavior? A decision-maker-researcher partnership systematic review

Do CCDSSs improve practitioners' diagnostic test ordering behavior?

The research team engaged key decision makers early in the project to guide its design and endorse its funding application. Direction for the overall review was provided by senior administrators at Hamilton Health Sciences (one of Canada's largest teaching hospitals) and our regional health authority. JY (Department of Medicine) and DK (Chair, Department of Radiology) provided specific guidance for the area of diagnostic test ordering by selecting from each study the outcomes relevant to diagnostic testing. HIRU research staff searched for and selected trials for inclusion, as well as extracted and synthesised pertinent data. All partners worked together through the review process to facilitate knowledge translation, that is, to define whether and how to transfer findings into clinical practice.

We previously published the details of our search strategy [20]. Briefly, we examined citations retrieved from MEDLINE, EMBASE, Ovid's Evidence-Based Medicine Reviews, and Inspec bibliographic databases up to 6 January 2010, and hand-searched the reference lists of included articles and relevant systematic reviews.

In pairs, our reviewers independently evaluated each study's eligibility for inclusion, and a third observer resolved disagreements. We first included all RCTs that assessed a CCDSS's effect on healthcare processes in which the system was used by healthcare professionals and provided patient-specific assessments or recommendations. We then selected trials of systems that gave direct recommendations to order or not to order a diagnostic test, or presented testing options, and measured impact on diagnostic processes. Trials of systems that simply gave advice for interpreting test results were excluded (such as Poels et al.[21]), as were trials of diagnostic systems that only reasoned through patient characteristics to suggest a diagnosis without making test recommendations (such as Bogusevicius et al.[22]). Systems that provided only information, such as cost of testing [23] or past test results [24] without actionable recommendations or options were also excluded.

Pairs of reviewers independently extracted data from all eligible trials, including a wide range of system design and implementation characteristics, study methods, setting, funding sources, patient/provider characteristics, and effects on care process and clinical outcomes, adverse effects, effects on workflow, costs, and practitioner satisfaction. Disagreements were resolved by a third reviewer or by consensus. We attempted to contact primary authors of all included trials to confirm extracted data and to provide missing data, receiving a response from 69% (24/35).

We assessed the methodological quality of eligible trials with a 10-point scale consisting of five potential sources of bias, including concealment of allocation, appropriate unit of allocation, appropriate adjustment for baseline differences, appropriate blinding of assessment, and adequate follow-up [20]. For each source of bias, a score of 0 indicated the highest potential for bias, whereas a score of 2 indicated the lowest, generating a range of scores from 0 (lowest study quality) to 10 (highest study quality). We used a 2-tailed Mann-Whitney U test to assess whether the quality of trials has improved with time, comparing methodologic scores between trials published before the year 2000 and those published later.

Whenever possible, we classified systems as serving at least one of three purposes: disease monitoring (e.g., measuring HbA_1c in diabetes), treatment monitoring (e.g., measuring liver enzymes at time of statin prescription), and diagnosis (e.g., laboratory tests to detect source of fever). We classified trials in each area depending on whether they gave recommendations for that purpose and measured the outcome of those recommendations. Trials of systems for monitoring of medications with narrow therapeutic indexes, such as insulin or warfarin, are the focus of a separate report on CCDSSs for toxic drug monitoring and dosing and are not discussed here.

We looked for the intended direction of impact: to increase or to decrease testing. We considered a system effective if it changed, in the intended direction, a pre-specified primary outcome measuring use of diagnostic tests (2-tailed p < 0.05). If multiple pre-specified primary outcomes were reported, we considered a change in ≥50% of outcomes to represent effectiveness. We considered primary those outcomes reported by the author as 'primary' or 'main,' or if no such statements could be found, we considered the outcome used for sample size calculations to be primary. In the absence of a relevant primary outcome, we looked for a change in ≥50% of multiple pre-specified secondary outcomes. If there were no relevant pre-specified outcomes, systems that changed ≥50% of reported diagnostic process outcomes were considered effective. We included studies with multiple CCDSS arms in the count of 'positive' studies if any of the CCDSS arms showed a benefit over the control arm. These criteria are more specific than those used in our previous review [19]; therefore, some studies included in our earlier review [19] were re-categorised with respect to their effectiveness in this review.

We summarized data using descriptive measures, including proportions, medians, and ranges. Denominators vary in some proportions because not all trials reported relevant information. We conducted our analyses using SPSS, version 15.0. Given study-level differences in participants, clinical settings, disease conditions, interventions, and outcomes measured, we did not attempt a meta-analysis.

A sensitivity analysis was conducted to assess the possibility of biased results in studies with a mismatch between the unit of allocation (e.g., clinicians) and the unit of analysis (e.g., individual patients without adjustment for clustering). Success rates comparing studies with matched and mismatched analyses were compared using chi-square for comparisons. No differences in reported success were found for diagnostic process outcomes (Pearson X² = 0.44, p = 0.51). Accordingly, results have been reported without distinction for mismatch.

Details of our methodological quality assessment can be found in Additional file 1, Table S1. Fifty-four percent of trials concealed group allocation [26, 27, 30, 32‐35, 37‐40, 42‐44, 50, 52‐55, 60‐63, 66‐68]; 51% allocated clusters (e.g., entire clinics or wards) to minimize contamination between study groups [15, 25, 28‐30, 34, 36, 38‐44, 46, 50‐53, 60, 62‐64, 68]; 77% either showed no differences in baseline characteristics between study groups or adjusted accordingly [15, 26‐37, 39, 40, 45‐55, 58, 59, 61‐66, 68]; 69% of trials achieved ≥90% follow-up for the appropriate unit of analysis [15, 25, 28‐35, 37, 39‐41, 45, 50‐56, 58‐61, 66, 67]; and all but one used blinding or an objective outcome [45].

Most studies had good methodological quality (median quality score, 8; ranging from 2 to 10) and 63% (22/35) [15, 25‐38, 50‐55, 58‐61, 65] were published after our previous search in September 2004. Study quality improved with time (median score before versus after year 2000, 7 versus 8, 2-tailed Mann-Whitney U = 44.5; p = 0.002), mainly because early trials did not conceal allocation and failed to achieve adequate follow-up.

Systems' design and implementation characteristics are presented in Additional file 2, Table S2, but not all trials reported these details. CCDSSs in 80% of trials (28/35) gave advice at the time of care [25‐27, 30, 31, 34‐37, 39‐51, 54, 56‐64, 66‐68]; most were integrated with electronic medical records (82%; 27/33) [15, 26, 27, 30‐34, 36, 37, 39, 40, 42‐51, 54, 56‐58, 60‐64, 66‐68] and some were integrated with computerized physician order entry (CPOE) systems (26%; 7/27) [31‐33, 37, 50, 54, 67, 68]; 77% (24/31) automatically obtained data needed to trigger recommendations from electronic medical records [15, 26, 27, 30‐34, 36, 37, 39, 40, 45, 46, 50, 51, 54, 56‐58, 60‐64, 66‐68], while others relied on practitioners, existing non-prescribing staff, or research staff to enter data. In most trials (61%; 20/33) advice was delivered on a desktop or laptop computer [15, 26, 27, 31, 34, 36‐41, 50, 51, 54, 58‐63, 66‐68], but other methods included personal digital assistants, email, or existing staff. Seventy-four percent (26/35) of systems were implemented in primary care [15, 25‐40, 42‐45, 50‐54, 58, 60‐65, 67]; 56% (14/25) were pilot tested [25, 28‐33, 36, 38, 42‐45, 51, 54, 62, 63, 66, 67]; and users of 59% (17/29) were trained [25‐29, 31‐33, 35, 37, 39‐44, 46, 51, 54, 58‐60, 67]. Eighty-three percent of trials (29/35) declared that at least one author was involved in the development of the system [15, 25‐33, 36, 37, 39‐41, 45‐53, 55‐60, 62‐68]. In general, user interfaces were not described in detail. Additional file 3, Table S3 gives further description of the setting and method of CCDSS implementation.

The 35 trials included a total of 4,212 practitioners (median, 132; ranging from 14 to 600, when reported) caring for 626,382 patients (median, 2,765; ranging from 164 to 400,000, when reported) in 835 clinics (median, 15; ranging from 1 to 142, when reported) across 545 distinct sites (median, 4.5; ranging from 1 to 112, when reported).

Three trials did not declare a funding source [31, 57, 60]. Of those that did, 78% (25/32) were publically funded [15, 25‐30, 36‐38, 41‐51, 54, 56, 58, 59, 61‐64, 66‐68], 9% (3/32) received both private and public funding [39, 40, 55, 61], and 13% (4/32) were conducted with private funds only [32‐35, 65].

Each system's impact on the use of diagnostic tests is summarized in Table 1, and Additional file 4, Table S4 provides a detailed description of test ordering outcomes. These outcomes were primary in 37% (13/35) of trials [15, 25‐27, 31, 41, 50‐53, 55, 58, 61‐63, 66].

Fifty-six percent (18/33) of evaluated trials demonstrated an impact on the use of diagnostic tests [15, 25‐31, 41, 52, 53, 55‐57, 59‐64, 66, 67] Two studies [65, 68] met all eligibility criteria and included diagnostic process measures but were excluded from the assessment of effectiveness because they did not provide statistical comparisons of these measures.

Systems in 49% (17/35) of trials (median quality score, 7; ranging from 4 to 10) gave recommendations for monitoring active conditions, all focusing on chronic diseases [25‐49]. Their effectiveness for improving all processes of care and patient outcomes was assessed in our review on chronic disease management. Here we looked specifically for their impact on monitoring activity and found that 35% (6/17) increased appropriate monitoring [25‐31, 41].

In the context of diabetes, four of eight trials successfully increased timely monitoring of common targets such as HbA1c, blood lipids, blood pressure, urine albumin, and foot and eye health [26‐30, 41]. One of two systems that focused primarily on monitoring of hypertension was effective at increasing the frequency of appropriate blood pressure measurement [31]. One of three trials that focused on dyslipidemia improved monitoring of blood lipids [25]. Another three systems gave suggestions for monitoring of asthma [35, 37, 39, 40], angina [39, 40], chronic obstructive pulmonary disease (COPD) [37], and one for a combination of renal disease, obesity, and hypertension [47‐49], but all failed to change testing behavior.

Systems in 23% of trials (8/35) [34, 50‐57] provided suggestions for laboratory monitoring of drug therapy. Trials in this area were generally recent and of high quality (median score, 8.5; range, 2 to 10; 75% (6/8) published since 2005). They targeted a wide range of medications (described in Additional file 4, Table S4) and are discussed in detail in our review of CCDSSs for drug prescribing, which looked for effects on prescribing behavior and patient outcomes. Focusing on their effectiveness for improving laboratory monitoring, we found that 63% (5/8) improved practices such as timely monitoring for adverse effects of medical therapy [34, 52, 53, 55‐57]. However, two of the trials demonstrating an impact were older and had low methodologic scores [56, 57].

Systems in 17% of trials (6/35) [58‐64] gave recommendations for ordering tests intended to aid diagnosis (median quality score, 7.5; ranging from 6 to 9) and 67% (4/6) were published since 2005 [58‐61]. Eighty-three percent (5/6) successfully improved test ordering behavior [59‐64]. Systems suggested tests to investigate suspected dementia in primary practice [60], to detect the source of fever for children in the emergency room [59], to increase bone mineral density measurements for diagnosing osteoporosis [61], to reduce unnecessary laboratory tests for diagnosing urinary tract infections or sore throats [62, 63], to diagnose HIV [58], and to diagnose a host of conditions, including cancer, thyroid disorders, anemia, tuberculosis, and others [64].

Finally, five trials did not specify the clinical purpose of recommended tests [15, 65‐68], or suggested tests for several purposes but without data necessary to isolate the effects on testing for any one purpose. Three of five focused on reducing ordering rates and were successful [15, 66, 67]. Javitt et al. intended to increase test ordering and measured compliance with suggestions, but did not evaluate the outcome due to technical problems [65]. Overhage et al. meant to increase 'corollary orders' (tests to monitor the effects of other tests or treatments), but did not present statistical comparisons of their data on diagnostic process outcomes [68].

Potentially important factors such as user satisfaction, adverse events, and impact on cost and workflow were rarely studied ( see Additional file 5, Table S5). Because most systems also gave recommendations for therapy, we were usually unable to isolate the effects of test-ordering suggestions on these factors, and here we discuss systems that gave only testing advice.

Two trials estimated statistically significant reductions in the cost of care, but estimates were small in one study [37] and imprecise (large confidence interval) in the other [28, 29]. A third study estimated a relatively small reduction in annual laboratory costs ($35,000), but presented no statistical comparisons [66].

Three trials formally evaluated user satisfaction. One study found mixed satisfaction with a system for monitoring of diabetes and postulated that this was due to technical difficulties [26, 27]. Another found that 78% of users felt CCDSS suggestion for ordering of HIV tests had an effect on their test-ordering practices, despite failing to show an effect of the CCDSS in the study [58]. The third study found that, regardless of high satisfaction with the local CPOE system, satisfaction with reminders about potentially redundant laboratory tests was lower (3.5 on a scale of 1 to 7) [66].

Only one study formally looked for adverse events caused by the CCDSS [66]. The system was designed to reduce potentially redundant clinical laboratory tests by giving reminders. Researchers assessed the potential for adverse events by checking for new abnormal test results for the same test performed after initial cancellation. Fifty-three percent of accepted reminders for a redundant test were followed by the same type of test within 72 hours, and 24% were abnormal, although only 4% provided new information and 1% led to changes in clinical management.

One study made a formal attempt to measure impact on user workflow and found that use of the CCDSS did not increase length of clinical encounters [45]. However, this outcome was not prespecified and the study may not have had adequate statistical power to detect an effect.