A randomized controlled trial evaluating the impact of knowledge translation and exchange strategies

The integration of research evidence into public health policy and program decision making is commonly referred to as evidence-informed decision making [2], and strategies to promote it as KTE. However, it is well known that the decision-making process is complex, and that multiple forms of knowledge impact both the process as well as the decision. In this study, we were interested in exploring the use of research evidence in decisions concerning the provision of public health services for promoting health body weight in children. In Canada, program managers in public health departments typically make recommendations to senior management on the specific interventions and strategies that could be provided to address particular population issues (e.g., healthy weights) [3]. Managers typically explore different options and make decisions about interventions that fit within the social and political climate of their respective regions. We explored whether research evidence influenced these decisions made by program managers concerning whether and which interventions they recommended their health department make available in order to promote healthy body weight in children.

Factors identified previously in the KTE literature known to contribute to clinical and program planning decisions include those related to individual decision makers, the system, patients, and research evidence [4]. At the individual decision-maker level, important factors include past experiences (e.g., clinical or managerial experiences with patients/clients, policy makers, events, or circumstances), beliefs, values, and skills; the environment/system level includes resources (both human and financial), legislation, protocols, and societal norms; patient preferences; and research evidence (e.g., multiple ways of knowing) [5‐8]. The intent of evidence-informed decision making is not to suggest that health policy and program decisions be determined solely by research evidence, but rather research evidence be considered within the context of the setting or circumstance, societal expectations, health care resources, and professional expertise.

Barriers consistently identified to evidence-informed decision making in the KTE literature include: lack of time; limited access to research evidence (e.g., many health departments can identify relevant research evidence in the published domain, but experience significant challenges in obtaining the full text in a cost-efficient and timely way) [9, 10]; limited capacity to appraise and translate research evidence; and resistance to change (e.g., lack of motivation to stop doing what has traditionally been done) [11‐17]. System-level changes needed to support evidence-informed decision making include: researchers gaining a better appreciation of the context in which decision makers function and building more collaborative relationships with decision makers [3, 18, 19].

Three KTE strategies are currently being widely used to promote evidence-informed decision making. These include: freely accessible web-based resources that summarize research evidence; tailored and targeted messages that connect relevant research evidence to specific decision makers [20]; and knowledge brokers (KBs), who work one-on-one with decision makers to facilitate evidence-informed decision making [21]. The internet is established as an essential component of KTE [22], and significant resources have been and continue to be allocated to these strategies. Several web-based resources have been developed with the intent of compiling the best available research evidence by topic area or health care discipline (e.g., Medline Plus, More EBN, health-evidence.ca). Some have gone one step further to synthesize the results of the evidence to answer specific practice-based questions [23]. However, there is a scarcity of literature evaluating the effectiveness of web-based resources in achieving evidence-informed decision making.

Tailored and targeted messages have gained momentum as a popular KTE strategy [24‐27]. 'Tailored' implies that the message is focused on the specific scope of decision-making authority of the intended user, while 'targeted' indicates that the content of the message is relevant and directly applicable to the decision currently faced by the intended audience. Evidence indicates that computer-tailored messages are associated with increased uptake compared to standardized messages [28], and that electronic targeted messages to subgroups with common interests is effective in promoting evidence-informed decision making [29]. While tailored, targeted messages have been shown to improve uptake of systematic reviews [30], questions remain as to what content is most wanted and required for different audiences, what the most effective communication channels are [28], and which organizations will benefit most from such a KTE strategy.

KBs have been implemented widely in private industry [31‐33], and more recently in healthcare settings [21, 34, 35]. In fact, a great many organizations in Canada have quickly moved to adopt KB roles with little more than anecdotal evidence supporting their effectiveness. A KB acts as a catalyst for systems change, establishing and nurturing connections between researchers and end users [36], and facilitating learning and exchange of knowledge [37]. The anecdotal evidence suggests that KBs improve the quality and usefulness of evidence that is employed in decision making [38], while promoting a decision-making culture that values the use of evidence [39, 40]. Furthermore, the heightened degree of interaction with decision makers through knowledge brokering is assumed by many to be the optimal KTE strategy in comparison to less interactive strategies; however, this has yet to be proven [34]. Given the lack of evaluation of each of these KTE strategies individually or in comparison to one another, the timing was right for conducting this study.

The problems of obesity, overweight, and physical inactivity have been identified in children [41]. According to the latest Canadian Fitness and Lifestyle Research Institute Physical Activity Monitor [42], 90% of Canadian children and youth aged five to 17 are not active enough to promote good health. Many of the risks associated with obesity in children cluster in cardiovascular disease risk factors known as the insulin resistance syndrome, and have been identified in children as young as five years of age [43]. In addition, overweight in childhood increases the risk of death from ischemic heart disease in adulthood two-fold over 57 years, and the incidence of Type 2 diabetes is increasing and is attributable to obesity [44]. Most alarming, however, is the knowledge that physical activity patterns and chronic disease conditions track from childhood into adulthood [45‐55]. Canadian research estimates that physical inactivity and obesity resulted in expenditures of $5.3 and $4.3 billion in direct and indirect costs, representing 2.6% and 2.2%, respectively, of total health care costs in Canada [56].

The literature demonstrates that regular aerobic activity increases exercise capacity and plays a role in both the primary and secondary prevention of cardiovascular disease [57‐60]. Furthermore, regular physical activity has been shown to enhance health, reduce the risk for all-cause mortality, prolong life, and improve quality of life [61‐67]. The evidence suggests that the best primary strategy for improving the long-term health of children and adolescents may be in creating a lifestyle pattern of regular physical activity and healthy eating that will carry over to the adult years [68].

Public health departments in Canada are responsible for promoting the health of the population, preventing disease, and providing medical care to treat communicable diseases. They provide services that focus on promoting the health of individuals as well as health promotion within schools and worksites, nutritional counseling, physical activity promotion, development of community strengths to promote/improve health, and the promotion of healthy environments [69]. The public health sector in Canada is structured generally with a medical officer of Health at the head of the organization and who has senior decision-making authority (subsequent to the local/regional board of health) for the services provided by that organization to a designated local community or region. The public health workforce responsible for the promotion of physical activity and chronic disease prevention is comprised primarily of public health nurses, nutritionists, physical activity experts, and health promotion officers. At the time this study was conducted (July 2004 to February 2006), all provinces and territories in Canada held mandates requiring public health departments to develop and implement strategies to promote healthy body weight in children. Despite these mandates, there was limited capacity (time, skill, access) among public health decision makers, and limited resources to utilize the best available research evidence with which to plan and implement effective healthy body weight promotion programs and services.

This randomized controlled trial funded in 2003 by the Canadian Institutes of Health Research, was the first in Canada to evaluate the effectiveness of a KB in comparison to other KTE interventions on promoting evidence-informed decision making in public health departments. Following ethics approval (McMaster University Faculty of Health Sciences Research Ethics Board) and recruitment, participating health departments were stratified according to size of population served and randomly allocated to groups using computer-generated random numbers. Given the background work conducted by the research team, as well as findings from the literature, stratifying public health departments by size of population served prior to randomization was deemed necessary. The three strata were defined as: health departments serving a population size below 50,000; a population size between 50,000 and 250,000; and a population size above 250,000. The Statistics Canada Peer Groups were used to allocate public health departments to each strata. The public health departments were randomly allocated to intervention groups in equal numbers within strata by computer generated pseudorandom draws using standard algorithms. Three health departments that remained unselected after equal allocation within strata were assigned to treatment groups randomly across strata. The health department was the unit of analysis. The study process is shown in Figure 1.

The framework proposed by Dobbins et al. [70] is one of many frameworks [71‐76] that have been developed to illustrate the process of knowledge translation and evidence-informed decision making. Dobbins' framework was used to guide the development of the KTE strategies (tailored, targeted messages and KB) and identify relevant outcomes for this study. The framework demonstrates the complex inter-relationships that exist between the five stages of innovation identified by Rogers, [77] (knowledge, persuasion, decision, implementation, and confirmation), and four types of characteristics, organizational, environmental, individual, and the innovation [78], as the knowledge translation process occurs. The framework also identifies the variety of possible outcomes that can be observed including: knowledge and attitudes; decision making; implementation (e.g., putting research knowledge into public health policy and practice, guideline development); and outcomes (e.g., changes in public health policy and practice). This study focused on the measurement of outcomes, specifically changes in public health policies and programs at the local public health department level.

The hypotheses were: public health departments exposed to tailored, targeted messages and the KB would report greater evidence-informed decision making than those exposed to a repository of quality assessed systematic reviews evaluating public health interventions (health-evidence.ca); knowledge brokering would result in greater evidence-informed decision making than tailored, targeted messages; and characteristics of the organization would have significant impacts on the effect of the KTE interventions on evidence-informed decision making. More specifically, we hypothesized that certain organizational characteristics (e.g., research culture, or the value organizations placed on the use of research evidence in decision making) would have an impact on the effectiveness of the KTE interventions to promote evidence-informed decision making. A previous study with Canadian public health decision makers illustrated that public health departments that valued the use of research evidence in decision making were significantly more likely to use research evidence for program planning decisions than health departments that put less value on research evidence [79, 80]. Therefore, we hypothesized that organizations that placed lower value on using research evidence in decision making would experience less improvement in evidence-informed decision making than those who valued research evidence more highly.

The moderately interactive KTE intervention included tailored, targeted messages plus access to health-evidence.ca (TM group). The TM intervention included sending participants a series of emails that included the title of the seven systematic reviews followed by a link to the full reference, including abstracts, on health-evidence.ca. The online reference offered a link to the short summaries, and finally, the full text of each review. Over seven successive weeks, on the same day each week and the same time of day, participants in the TM group were sent an email indicating that a systematic review related to healthy body weight promotion in children was available in full text at the link provided. At the URL linked within the email message, participants also received access to the PDF version of the systematic review, the published abstract of the review, as well as the short summary written. Finally, the text of the message was worded to say, 'this message is number XX in a series of seven emails you will receive on healthy body weight promotion in children as part of the KTE strategy you are being exposed to in this randomized controlled trial'.

The most interactive KTE intervention included both the HE and TM components and a KB who worked one on one with decision makers in the public health departments. One full-time KB provided knowledge brokering services to all English speaking participants allocated to the KB group (n = 30). A second Francophone KB (0.2 full time equivalent) provided KB services to French-speaking participants also allocated to the KB group (n = 6). The KBs were Master's prepared, had extensive knowledge and expertise in public health decision making, as well as an understanding of the research process. Specific tasks conducted by the KB included: ensuring relevant research evidence related to healthy body weight promotion was transferred to the public health decision makers in ways that were most useful to them, assisting them to develop the skill and capacity for evidence-informed decision making, and assisting them in translating evidence into local practice.

Approximately twenty percent of KB time was spent facilitating knowledge and skill development either through face-to-face interaction such as workshops or online strategies such as webinars, interactive web-enabled meetings, or conferences. Eighty percent of the brokers' time was spent preparing for and directly interacting with participants. The proportion of time the KB spent preparing for interaction with participants was 40% to 50% early in the project and declined to 30% as both public health decision makers and the KB became more skilled in their respective roles. KB activities were classified into the following categories: initial and ongoing needs assessments; scanning the horizon; knowledge management; KTE; network development, maintenance, and facilitation; facilitation of individual capacity development in evidence-informed decision making; and facilitation of and support for organizational change. These activities were carried out through regular electronic and telephone communication, and one site visit to each health department of one to two days in length. As well, each health department was invited to attend a one-day workshop held regionally (four cities) across Canada. A more complete description of the KB intervention is published elsewhere [81]. However, the main activities of the KB intervention are described.

At the start of the intervention, the KB conducted assessments at the individual, organizational, and environmental levels, in order to identify strengths, knowledge, and capacity for evidence-informed decision making. The KB then worked with participants to generate a plan for developing individual and organizational capacity for evidence-informed decision making. In order to facilitate participant access to the best available evidence, the KB consistently scanned the horizon for new evidence and resources of interest to participants. This activity involved maintaining subscriptions to related list serves, electronic distribution lists, and e-table of contents alerts from relevant journals. The majority of the KB's time was spent doing KTE, which was facilitated by developing and maintaining a trusting relationship with participants. The KB-initiated communication with participants occurred at a minimum of once per month, and more frequently as requested. The KB also offered a site visit to each public health department. The purpose of the site visit was to facilitate the building of a trusting relationship between the health department and the KB, as well as to enable the KB to learn more about the local context. This facilitated the tailoring of KB services to the specific needs of each local environment. Furthermore, the activities conducted by the KB during each site visit varied according to specific needs and goals identified by each health department. In many cases, the KB participated in team program planning sessions and assisted in the interpretation of evidence from the tailored, targeted messages and its incorporation into local program plans. The KB also conducted training sessions in many health departments to assist participants and their colleagues in developing their capacity to be critical consumers of different knowledge sources. Opportunities to facilitate knowledge, skills development, and capacity for evidence-informed decision making occurred during all interactions with the KB at the individual (email, telephone, site visit) and group level (site visit, regional workshop, webinars). Finally, during the regional workshops, the KB presented the results of the systematic reviews to participants, facilitated discussion concerning the results as well as implications for local program and public health policy development. KBs also encouraged participants to engage in individual and joint problem-solving related to evidence-informed decision making, and enabled face-to-face contact with the KB to promote credibility and trust.

The data were collected using a telephone-administered survey (knowledge transfer and exchange data collection tool) at baseline (August 2004) and immediately post-intervention (February 2006). Items in the questionnaire were chosen from questionnaires previously tested and used in diffusion of innovation and research utilization studies [11, 77, 82‐88]. We tested the modified questionnaire for reliability and validity among public health decision makers, and have reported a Cronbach alpha of 0.65 for reliability elsewhere [11, 80, 89]. The questionnaire is available from the corresponding author upon request. The questionnaire was administered twice to participants at baseline, one month apart.

Data were collected on organizational, environmental, and individual characteristics shown previously to be related to evidence-informed decision making [79], and measured using seven-point Likert scales. Organizational characteristics included: organizational culture (e.g., research culture, or the value placed on using research evidence in decision making, and the expectation to demonstrate use of research evidence in decision making), staff training in research methods and critical appraisal, and decision-making style. The environmental characteristic included collaboration with other community organizations. Individual characteristics included age, education, position, perceived influence over the decision-making process, and perception of the barriers to using research evidence in public health decision making. All variables were measured in the same direction.

Two dependent variables were evaluated: global evidence-informed decision making and public health policies and programs. For global evidence-informed decision making, participants were asked to report on the extent to which research evidence was considered in a recent program-planning decision (previous 12 months) related to healthy body weight promotion. This is a common way of measuring research use in the KTE field. Participants were asked to quantify their response ranging from one (not at all) to seven (completely). However, given many have suggested that this is not an optimal way of measuring evidence-informed decision making, we developed a second outcome variable, labeled 'public health policies and programs'. This measure was derived as the sum of actual strategies, policies, and/or interventions for healthy body weight promotion in children being implemented by the health department.

Eleven policies, programs, and/or interventions with good evidence of effectiveness were identified from seven systematic reviews assessed as being of high methodological quality [90‐96] (Table 1). Each systematic review was assessed for methodological quality by two independent reviewers using a previously developed and tested quality assessment tool [97, 98]. Reviewers met to discuss ratings, and consensus on all ratings was achieved. Only those systematic reviews attaining seven points or higher out of a total of ten possible points were deemed of sufficient methodological quality to inform public health policy and practice. Participants were asked whether the public health policies and programs were being implemented by their health department (yes/no). The total number was summed and compared across groups from baseline to post intervention.

Mixed-effects models were used to conduct tests of the two hypotheses related to the treatment effects, which is a standard approach to the analysis of designs with repeated measurements [99]. Repeated measurements over time were modeled as nested within participants, and time of observation was coded to estimate the differences between groups in scores at the average of the two baseline observations, and then the change from baseline to the post-intervention follow-up. The interaction of this change with the randomized treatment assignment is the appropriate estimate of the treatment effect, such that we tested whether change following the intervention differs among the intervention groups. These mixed-effects models provide for appropriate adjustment for the repeated measurements with participants when testing treatment effects, and they also allow for flexible handling of missing data. The moderating roles of selected predictor characteristics (hypothesis three) were also tested by evaluating their three-way interactions with time and treatment.

Participation by province and territory ranged from 29% to 100% with the sample consisting primarily of health departments serving both urban and rural populations (46%). Table 2 presents a description of the study sample. Follow-up data were collected from 88 of 108 (81.5%) participating public health departments. Reasons for not participating in the follow-up survey were lack of time and not having anyone working in healthy body weight promotion. Among the HE, TM, and KB groups, similar drop-out rates were observed of seven, six, and seven health departments, respectively.

It is unknown to what extent the HE group accessed http://​www.​health-evidence.​ca. To our knowledge, all those exposed to the TM intervention received 100% of the intervention. For those exposed to the KB intervention, approximately 70% received the full intervention (e.g., frequency, intensity) with approximately 15%, respectively, not engaging at all, or to a limited extent. Organizations were analyzed according to their assigned group.

The estimates from the mixed-effects models are presented in Table 3. The table gives estimated pair-wise differences for the TM and KB groups, relative to control (HE group), as well as overall tests of group differences at baseline and for the change from baseline to follow-up. In addition, the standard deviation in outcome between and within health departments over time is provided. This gives an indication of the degree of variation in the outcomes that remains unexplained after accounting for the intervention. For both outcomes, most of the remaining variation appears as unexplained changes over time within health departments. Table 3 shows that baseline scores do not differ significantly between groups for either outcome, although the TM group possibly had fewer public health policies and programs at baseline compared to the HE group (p < 0.06).

As shown in Table 3, the intervention had no significant effect on global evidence-informed decision making (p < 0.45), although all groups improved to some extent. For public health policies and programs, as is shown in Figure 2, a significant effect of the intervention was observed (p < 0.01). For this outcome, the TM group improved significantly from baseline to follow-up in comparison to the HE and KB groups that showed no significant change. With respect to hypothesis three, some organizational characteristics were shown to moderate the intervention effect, although not always in the hypothesized direction. When organizational research culture was added to the mixed-effects models as a predictor, the group/time/culture interaction was significant (p < 0.03). This three-way interaction is illustrated in Figure 3, with the predictions for each group shown at relatively low (four of seven) and high (six of seven) values of the extent to which health departments reported they valued research evidence.

As Figure 3 illustrates for health departments with low organizational research culture, the intervention effect was as we hypothesized -- the control group was unchanged, the TM group improved somewhat, and the KB group improved most. However, when organizational research culture was high (six on a seven-point scale), the HE group remained unchanged, the KB group decreased (fewer public health policies and programs), and the TM group increased significantly. Similar trends were observed for organizational characteristics, such as expectation to use research evidence and frequency of hearing the term research evidence. However, no significant differences in results were observed when multiple organizational characteristics were included in the models, therefore only the results of organizational research culture have been presented.

Our primary outcome, Global evidence-informed decision making, may not be optimal for measuring KTE effectiveness despite its consistent use in the literature. It is likely that this measure is too vague to elicit reliable and valid responses. One conclusion is that concrete outcome measures, such as public health policies and programs that are tied to specific behaviors and/or programs, may provide a more concrete measure of evidence-informed decision making. However, challenges still exist with the use of this measure because the existence of organizational policies does not necessarily translate into actual services being provided, and it is unclear what the optimal data source is for identifying public health policies and programs that are being implemented. Priorities for future research include: development and testing of data collection tools for measuring more objectively evidence-informed decision making outcomes; and continued exploration of subjective measures so as to better understand evidence-informed decision making and KB processes, as well as indicators of KB success.

The limitations in this study include: the source of data, participant turnover, exposure to the intervention, and self-reported outcome measures. A decision to have just one participant from each organization provide data was made following an assessment of Canadian public health departments that suggested services for healthy body weight promotion in children were coordinated across organizations. In reality, these programs span multiple divisions (e.g., healthy lifestyles, family health) and multiple teams within divisions (e.g., nutrition, physical activity, schools), resulting in many public health professionals working simultaneously on different interventions, programs, and policies, usually with limited knowledge of what others are doing. It is possible that the participants in our study had inadequate knowledge to accurately report on all public health policies and programs provided by their organization. This may have led to both under- and over-reporting of this outcome. One strategy to overcome this issue would be to have multiple participants from each health department participate in data collection and report only on those interventions, programs, or policies directly relevant to themselves. Furthermore, while the KB encouraged multiple decision makers from each public health department to participate in the KB intervention, for many health departments only one decision maker was exposed to the intervention. This likely resulted in insufficient exposure to the intervention among health department staff to facilitate change at the organizational level.

A significant limitation of this study was high participant turnover. While the majority of health departments (81.5%) completed the study, different decision makers completed the baseline and follow-up surveys in 30% of health departments. This reflects the transient nature of public health in Canada and in the United States, and is not something that could have been avoided. This high turnover rate may have resulted in substantial error in outcome measurement and may explain some of the huge variation in the number of public health policies and programs observed from baseline to follow-up in some health departments. This continues to represent a significant issue, and one that we are unable to quantify in terms of its impact and in which direction on the data. Furthermore, given that up to 30% of participants either did not engage with the KB at all or to a limited extent, it is possible that the results of this study are generalizable only to those health departments that would engage with the KB.

The challenges we encountered in conducting this randomized controlled trial raise issues concerning the appropriateness of using empirical designs in evaluating the effectiveness of KTE strategies, particularly in public health settings. Of particular importance is the inability, through randomization, to eliminate all differences (particularly organizational ones) between comparison groups other than the intervention. This poses difficulties with interpretation of the results as it is unclear whether findings truly represent what actually occurred, or if inherent differences in organizational context masked or moderated the treatment effect. While the findings of this study (TM is effective only for organizations with certain characteristics) contribute important knowledge to the field, additional research is needed to better understand the how, what, where, and when with respect to the effectiveness of the KTE strategies. Future research in this field should integrate other designs to better understand in which circumstances KTE strategies work, for whom, and why [116]. Other designs likely to be useful and used extensively in the business literature are case studies, interrupted time series, 'N of 1' studies, and qualitative studies such as grounded theory. Mixed methods designs will allows us to understand more fully not only if KTE strategies are effective, but also more importantly why or why not, how and why they work, what impedes impact, and when KTE strategies will have the greatest likelihood of having a significant and positive impact. Finally, additional research is required to develop and test a tool for assessing organizational factors associated with evidence-informed decision making. While some assessment tools exist, no one tool currently stands out as being optimal.