The use of external change agents to promote quality improvement and organizational change in healthcare organizations: a systematic review

The protocol for this systematic review was designed in accordance with PRISMA guidelines. We engaged a research librarian (M.H.) to assist in developing the search strategy. The librarian ran a number of search strategies to fine-tune the search terms to eliminate off-topic results. This stage consisted of an iterative review and revision process between the first author and the librarian to refine the list of abstracts. The list of search results included 477 abstracts.

With the assistance of the librarian, we searched PubMed, CINAHL, Cochrane, Web of Science, and Academic Search Premier Databases in July 2016 for studies published between January 1, 2005 and June 30, 2016. We incorporated MeSH terms, and CINAHL headings to refine our search results. The index terms and text words we used in our search consisted combinations of the following: quality improvement OR change OR process improvement OR (organizational improvement OR organizational change) AND (outpatient clinics or primary care OR drug use OR drug abuse OR opioid OR substance abuse OR addiction) AND (coaching or facilitation or mentoring OR precept* or consulting OR academic detail*). Originally we thought we would include addiction treatment clinics, but later excluded them because they operate very differently from primary care clinics.

We applied filters for study type, omitting articles classified as case reports, clinical conferences, comments, editorials, letters, lectures, meta-analyses, opinion pieces, or reviews.

We also consulted with field experts to ask about studies that they would suggest we include in our review. The suggestions and reference lists forwarded by the experts were reviewed by the first author according to the search criteria and included in the final list of 339 abstracts.

We included randomized controlled trials that investigate process improvement activities in healthcare organizations (general practices, primary care practices, private practices, and outpatient clinics) through external change agents. Because external change agents are referred by a number of different titles, we included studies that provide facilitating, coaching, mentoring, precepting, and academic detailing to the organizations’ staff members (physicians, nurses, administrators, etc.) through an external change agent. We focused our review on peer- reviewed clinical trials published since 2005 that measure the effects of 1 or more interventions.

We excluded studies describing patient coaching, as we wanted to focus on organizational change rather than individual health behavior change. We also excluded non-randomized studies, studies without a control group, and studies without the full text available in English, as well as reviews, conference proceedings, qualitative studies, and opinion pieces. In general, the initiatives targeted larger, clinic-wide changes rather than changes at the level of the individual practitioner.

We selected articles in 2 phases. In phase 1, two of the authors (EA and MC) independently screened titles and abstracts of the search results after removing duplicates based on the inclusion and exclusion criteria. We obtained full articles for the abstracts that we identified as relevant. After a second round of elimination that included reviewing full text articles based on the inclusion criteria, the final set of studies were identified. A number of clinical trials were excluded because they did not report on the effectiveness of the interventions. For phase 2, we independently reviewed the final list of full-text articles for the following variables: 1. intervention type, 2. setting, 3. study design, 4. intervention components, 5. background of change agents, 6. background of settings, 7. intervention duration and frequency of contact, 8. outcome measures, and 9. significance. A final review of the search results was conducted by the senior author (AQ) to finalize the list of included studies and the set of study variables and outcomes used to summarize them. Table 1 summarizes these variables.

Because the research domains, study methods, and measures were disparate, we could not perform the necessary data pooling for a meta-analysis. We performed a qualitative synthesis to categorize how study interventions –specifically the methods of change implementation-- affected the outcomes of the research. We analyzed the multifaceted interventions employed in each study under 5 separate categories based on our qualitative synthesis.

Definitions of Types of Interventions

Authors’ note: the terms “academic detailing,” “facilitation,” and “coaching” are not well defined nor rigorously applied in the research literature. In this review, ‘practice facilitation’ includes facilitating, coaching, and mentoring. We have supplied general terms for reference here In the study arms column in Table 1, we specify the frequency of contact the external change agent had with clinics.

We also assessed the quality of clinical trials included in this study. The lead author (EA) assessed risk for bias using the Cochrane Handbook for Systematic Reviews of Interventions for randomized clinical trials. The senior author (AQ) randomly selected 4 of the articles to rate for risk of bias. Studies were rated on five elements affecting risk of bias and each study received an overall quality rating of high, low, or unclear risk of bias. Of the 4 randomly selected articles, the co-authors agreed on 85% (17/20) of their ratings in the five subdomains, indicating high concordance in ratings of risk of bias. Of note, the search terms we employed led us to review only articles with rigorous study designs, meaning those with randomization to treatment and control groups. Hence, no studies were excluded on the basis of risk of bias.

The initial search of 5 databases identified 477 articles published from January 1, 2005 to June 31, 2016 (Fig. 1). Three additional titles were identified through other sources (i.e. expert feedback). One hundred forty-one of these articles were duplicates. We screened the remaining 339 titles and abstracts for relevance. Of these, 226 articles were excluded because they did not meet the inclusion criteria. Full text articles of 113 studies were reviewed. Thirty-four of these studies were selected for inclusion. After the final selection, we examined the studies in more detail. During this process, we identified a number of other articles that were not eligible. The reasons for exclusion are as follows. The first article we excluded did not report on the effectiveness of the interventions but reported on the parts of a larger clinical trial. We searched for the larger trial that the study was part of. However, it was published before 2005 [4]. The second study investigated the acceptability of the intervention and did not report on effectiveness [5]. The third study was eliminated because it reported the cost analysis of a randomized controlled study (RCT) conducted in 2005 [6]. We excluded a group of studies that were conducted in disparate healthcare and community settings outside of primary care, including intensive care units, dental care, and addiction treatment clinics [7‐13]. After reading through the articles, we decided to exclude two studies conducted in developing nations [14, 15] to make sure the results we report are compatible in terms of context. We also excluded one study that upon further examination proved not to be a RCT [16]. We extracted data from a final set of 21 studies.

Table 1 summarizes and details the study characteristics of the set of 21 studies that are included in this review. The studies represented 9 countries, with most studies occurring in the United States. Most of the included studies were cluster RCTs in which clinics were randomized. All included studies focused on Primary Care Practices (PCP) and General Practices (GP). Hence, the target clients were mostly physicians and nurses.

The studies reported on a number of interventions, combining the five components described previously. Table 1 lists details of the interventions for each study. Most of the studies focused on changing policies regarding chronic conditions care. Seventeen studies reported patient outcomes while 4 studies reported on measures developed by the investigators (such as surveys, self-defined objectives, etc.) [17‐20]. The studies reporting on patient outcomes pulled the data from an external electronic database (such as an electronic health record). The majority of studies measured the effects of intervention components as a single intervention.

The methodologic quality of included randomized clinical trials is available in Table 2. Most of the studies had low risk of selection and performance bias. The participants and staff were unblinded to the interventions in all cases. However, we decided that lack of blinding would have minor impact on study outcomes because most studies analyzed the intervention effects on patient outcomes while randomizing at the clinic level. A number of unblinded studies utilized self-assessments (either clinician or patient) as outcome measurements. Those cases are reported as high risk. Overall, majority of the clinical trials in this review demonstrated low risk of bias.

All studies except one that focused solely on the effects of academic detailing [21] investigated the impact of multifaceted interventions on practice change in healthcare practices. Nine studies investigated the effects of two-component interventions [10‐13, 16, 18, 19, 21‐26]. Others investigated three or more components. Sixteen studies out of the 21 included in this review used academic detailing (AD) as part of a multi-component intervention strategy [10, 13, 17, 21, 23‐34]. Thirteen studies had a form of audit and feedback integrated into their intervention [17‐19, 22‐25, 28, 29, 32‐35]. Eleven studies employed a type of practice facilitation or coaching during their interventions [17‐20, 25, 27, 32‐36]. All studies that included practice facilitation reported significant effects in one or more study outcomes. None of the studies that demonstrated ‘no effect’ in primary outcomes employed practice facilitation as a component of their intervention except two [9, 36] which showed significant change only in secondary outcomes.

Five studies reported having a form of information technology or system support (IT) [17, 26, 29, 30, 35]. Bertoni et al. [26], Clyne et al. [30] Feldstein et al. [29], and Mold et al. [17] utilized automated reminders that alerted clinicians when there was an error in the system. Smidth et al. [35] provided online forms or informational websites to participants. Only 2 studies utilized regular phone calls [27, 37] with sites. Both studies reported this strategy as not significantly effective. Two studies mailed educational materials to patients [30, 31]. Neither demonstrated a significant effect.

Most of the studies employed pharmacists and pharmacologists to deliver the academic detailing [9‐12, 21, 24, 30]. Physicians and nurses were also engaged as academic detailers [8, 13, 24, 26, 27, 29, 33, 34, 36]. In some studies, the investigators employed quality improvement experts who were trained in organizational change implementation [10, 17, 18, 20, 25, 32, 37]. These studies usually focused on change implementation and targeted practice facilitation as part of their intervention rather than delivering only academic detailing. All of the studies that included practice facilitation through external change agents demonstrated positive effect regardless of the change agent’s specific background [17‐20, 25, 27, 32‐36].

Because the context of each study varied and there was not a uniform reporting measure of effect size, we measured the effectiveness of an intervention based on the p values of change between study arms reported in each study. Thirteen of the 21 studies reported a significant positive change (p < .05) in their primary outcomes on the intervention arm [17‐20, 22, 23, 25, 27‐29, 32, 34, 35]. All of these studies investigated the effects of multi-faceted interventions that included at least two of the intervention categories described under the data analysis section.

Four studies reported that the intervention demonstrated mixed results; there was significant increase in several outcomes whereas there was no improvement in other outcomes. Bertoni et al. [26] reported that two of their four outcome measures had significant improvements. This study tested the effectiveness of academic detailing coupled with system support. Clyne et al. [30] reported a significant decrease in one of the primary outcomes whereas the improvement in second outcome was not significant. Mold et al. [17] reported significant positive impact in one of the three primary outcomes. Ornstein et al. [33] reported positive change in only a specific age group.

Follow-up individualized to the clinic stands out as an integral component of multi-faceted interventions for organizational change. A majority of the studies that reported significant improvement in their results included individualized follow up in their interventions. Studies use a number of titles to describe a change agent. These titles include practice facilitator, practice enhancement assistant, and outreach visitor. Facilitation is always conducted by an expert change agent who is trained in quality improvement tools and methods. The change agents perform the follow-up meetings in-person, and on a regular basis that averages a minimum of once a month. The changes that are implemented in the clinics are always tailored to clinic needs. Only one study that included change agents demonstrated no change [36].

Overall, six of the studies reported that their intervention did not demonstrate any significant improvement on the primary outcomes [13, 21, 24, 28, 31, 36]. One thing that these studies had in common was that all of them except one [36] tested academic detailing coupled with audit and feedback with little to no follow up, which strengthens the importance of facilitation on organizational change. A number of reasons were listed as possible cause for no effect. Hennesy et al. [31] emphasized that lack of intensive follow up may have caused the unfavorable outcomes. Varonen et al. [13] pointed out that some of the practices were already implementing the targeted policies of the intervention at baseline, limiting their outcomes. Two studies [21, 24] that showed no significant difference between study arms reported that both intervention and control groups demonstrated positive impact, which may be attributed to effects of larger scale campaigns taking place simultaneously with the study.

This review should be considered in the context of some limitations. Although our search results include a number of studies that reported no significant effect, it is possible that other clinical trials that did not demonstrate significant effects have not been published, and hence not included in this review; publication bias may have decreased the number of studies with negative findings. A second limitation is that variation in the types of labels used to identify the components of the interventions can act as a barrier to identifying relevant studies. For instance, practice facilitation can be labeled as coaching, consulting, or mentoring, whereas academic detailing can also be referred to using a wide set of terms, including learning collaboratives or problem-based learning. A third limitation is that our systematic review was rather tightly circumscribed through the application of our search criteria (we limited our search to randomized controlled trials featuring the use of external change agents in primary care settings in developed countries). We sought to minimize heterogeneity in terms of settings, study designs, and interventions in order to clearly isolate the effect of external change agents. This decision does not imply that there is not significant knowledge to be gained by studying the role of external change agents in developing countries, in healthcare settings besides primary care, and in otherwise eligible primary care studies that have used non-RCT study designs. Examination of the role of external agents across different contexts should be the subject of future research. Finally, the type, duration, and intensity of follow up (in-person or over the phone) provided through the practice facilitation is not always clearly reported, which may have an effect on the reported outcomes. The studies also vary in terms of how the educational materials are delivered (face to face vs online or mailed). This information was not always explicit in the reports.