Evaluation of a tailored, multi-component intervention for implementation of evidence-based clinical practice guidelines in primary care physical therapy: a non-randomized controlled trial

The design of this study was two-fold and comprised a systematic implementation strategy development component and a non-randomized controlled trial in which the effectiveness of the implementation intervention was evaluated in comparison with no intervention. Data were collected at baseline and follow-up. The study took place within the county council Region Västra Götaland in western Sweden, providing first-line primary care of both acute and long-term nature to 1.6 million inhabitants. At the physical therapy practices, PTs primarily treat patients with musculoskeletal disorders; the most common complaints are low back pain, neck pain, and subacromial pain. There are 60 physical therapy practices in five areas within the county council, which were allocated to intervention (three areas, 28 practices) or control group (two adjacent areas, 32 practices) based on geographic location. The study was conducted between November 2010 and November 2011.

All PTs employed at the 60 physical therapy practices were eligible to participate in the study. At the time of the baseline data collection, 425 PTs were employed in primary care by the authority and at follow-up, 454 PTs were employed. The guideline project team members, who were also employed by the county council, were excluded (n = 6). Figure 1 illustrates participant flow through the study.

The initiative to the guideline development and implementation was taken by PT managers within the county council, who perceived a variation in physical therapy practice and therefore a need for evidence-based practice guidelines. An integrated organizational development/research project was planned. Support from senior management of the county council, as well as funding of the project, was secured. Guideline development, needs analysis, and implementation strategy development were carried out in parallel processes within the integrated project (Figure 2). The guideline development was performed by a project team of six primary care PTs, in a systematic process guided by a 7-step guideline development model by Grol et al. [17]. The development process included systematic database searches, critical appraisal of the evidence base using GRADE [42], and formulating evidence-based practice recommendations. The guideline format followed recommendations from AGREE II [43]. The guidelines consisted of a brief summary on the first page; a brief introduction to the topic with up-to-date data on definition of the condition, prevalence and prognosis; recommendations on patient management according to strength of evidence; and a detailed reference list. Detailed search strategy and search results, summaries of reference articles, recommended outcome measures, and patient information leaflets were provided in appendices to the guidelines.

An implementation strategy was developed based on prevailing implementation theory [17], barriers and facilitators identified in the target population [5], and evidence from previous research. The latter included strategies that are theory-based [31, 44], active [38], multi-faceted [39], and tailored [25]. The 5-step model for implementation by Grol et al. [17] was selected as a framework for the implementation, because it is based on a comprehensive overview of theories for behavioral change, integrates several theories that we considered relevant, and provides a structured approach for the implementation process. The model, and how it was adapted for the specific clinical and organizational context and applied to the strategy development, is presented in Figure 3. Steps one and two of the model were carried out in parallel, as the needs analysis in step two guided the choice of topics for the development of the guidelines in step one. As part of step two, a previously reported cross-sectional survey [5] was conducted. The implementation strategy developed in step three focused on addressing the determinants of guideline use identified in the survey. Determinants and how they were addressed in our implementation intervention are shown in Figure 4.

Further, an educational element was included in the implementation strategy, to address misconceptions of the EBP concept that emerged from the survey. To guide topic selection, input was also solicited in the survey regarding which guidelines were needed the most. Among the most commonly demanded topics were guidelines for physical therapy treatment of low back pain, neck pain, and subacromial pain. These diagnoses are also referenced in the literature as three of the most common reasons to consult a PT in primary care [45]. In step four, details of the implementation activities were planned and the plan was executed (see below). In step five, data for the evaluation of the implementation were collected with the same questionnaire [46].

The implementation plan carried out in step four of the model comprised a multi-component intervention with the following components:

The seminar and group discussion sessions were held on nine different occasions over a 3-month period, with 9–28 PTs attending each session. In the group discussion, the guideline recommendations and current practice were discussed. The sessions were conducted by the first author and a project team co-member, both primary care PTs with postgraduate training in EBP skills. The seminars were held during working hours. Participation was supported and encouraged by senior managers but it was up to each unit manager to decide whether to send all or some of the unit’s PTs. Both the website and the seminar content were tailored to address areas in need of improvement that had been identified earlier [5].

Data were collected at baseline and follow-up via a web-based, self-report questionnaire covering various aspects of EBP and guidelines [46]. Follow-up data collection took place three to six months after the implementation seminars were carried out in the intervention group, corresponding to one year after baseline data collection.

The primary outcomes were the self-reported awareness of the existence of guidelines, knowledge of how to find guidelines, access to guidelines, and use of guidelines. The first three outcomes were selected because of their role as facilitators of guideline use, and the fourth outcome because of its importance in measuring the extent to which EBP was applied in practice. Awareness, knowledge, and access to guidelines were assessed with 3-point scales “yes”, “partially”, or “no”. Use of guidelines was assessed on a 5-point scale with response alternatives “never or very infrequently”, “infrequently”, “sometimes”, “frequently”, or “very frequently or always”. Secondary outcomes were self-reported attitudes to EBP and guidelines, assessed with 5-point scales ranging from “strongly disagree” to “strongly agree”.

The questionnaire had, in an earlier study, been found to be valid and of acceptable reliability in a primary care physical therapy context [46]. In that study, it was translated, cross-culturally adapted, and further developed from a previously used instrument to assess EBP [3]. Invitation to respond to the questionnaire was distributed via e-mail, containing a link to the questionnaire. Participants responded on-line and the survey software (EPiServer CMS 5, EPiServer AB, Stockholm, Sweden) logged the responses and added them to a results database. At both baseline and follow-up, three reminder notices were e-mailed at 1-week intervals. Staff turnover was analyzed by comparing the e-mail addresses between the two data collection points.

Ethical approval for the study was obtained from the Regional Ethical Review Board of Gothenburg (Reference 780–11). All questionnaires were filled out anonymously and responses could not be traced back to the respondents.

Sample size was calculated based on the assumption that a relevant difference between groups in the primary outcomes would be 15%. We estimated that we needed 137 PTs in each group to be able to detect a relevant change, with 80% power and a significance level of 5%. Assuming a response rate of approximately 65%, we needed to send the questionnaire to approximately 400 respondents, which corresponded well with the number of PTs employed in primary care in the county council.

Participating practices were analyzed using individual PTs as measurement units. Baseline data were compared between groups for age, gender, years of primary care experience, education level, workplace size, and for primary and secondary outcomes. Differences in proportions between groups were analyzed using Pearson’s χ² test. Before the analysis, the response categories were dichotomized into the two highest and lowest categories. For the primary outcomes, primary analyses of data for all PTs in the intervention group were performed, regardless of whether they had participated in the implementation seminar, the main component of the intervention (“intention-to-treat” analysis). Secondary analysis was restricted to those participants who actually participated in the seminar and responded to the follow-up questionnaire (“per-protocol” analysis).

For the primary outcomes, absolute changes between baseline and follow-up in proportions answering “frequently/almost always” or “yes” were computed. Differences in change between the groups were computed and analyzed using an approximative z-test. A significance level was set at p < 0.05. Analyses were performed using IBM SPSS Statistics, version 20.

At follow-up, a significantly higher proportion of PTs in the intervention group (59%) than in the control group (44%) reported being aware of guidelines (p = 0.030), knowing where to find guidelines (40% vs. 16%; p < 0.001), and having easy access to guidelines (26% vs. 7%; p < 0.001) (Table 2). The difference in guideline use was not statistically significant in intention-to-treat analysis (55% vs. 48%; p = 0.081), but was significant in per-protocol analysis, i.e., PTs who participated in the implementation seminar reported frequent use of guidelines to significantly greater extent than those in the control group (63% vs. 48%; p = 0.043).

Within-group changes in proportions from baseline to follow-up are shown in Table 3. The differences between the groups in this change were also statistically significant for three of the four primary outcomes (Table 3). The proportions of PTs reporting awareness of (p = 0.023), knowledge of (p = 0.007), and easy access to (p < 0.001) guidelines increased more in the intervention group than in the control group. The change in self-reported use of guidelines did not differ significantly (p = 0.297). Differences in proportions were between 9% and 22%.

Table 4 shows that there were no statistically significant differences in secondary outcomes. The only exception was “EBP helps decision making”, but this difference was small. Per-protocol analyses of secondary outcomes produced similar results (not reported).

Our findings support the conclusions of two systematic reviews [38, 39], concluding that PTs’ knowledge and, to some extent, behavior but not attitudes were improved after active multi-component implementation interventions. However, a third systematic review [37] found that multifaceted guideline implementation interventions were no more effective than single intervention strategies across several allied health professions. Two randomized controlled studies (included in the reviews) compared an active implementation intervention versus passive dissemination of physical therapy guidelines for low back pain and whiplash disorders, respectively [40, 41]. The active interventions were similar to ours and included interactive training, group discussions, and reminders. The studies found a 12% and a 44%, respectively, higher self-reported guideline adherence rate after the active strategy; a better result than in our study. However, in the study on whiplash guidelines, the effects on adherence were mixed; two of the guideline’s five recommendations were adhered to more by the intervention PTs than by those in the control group, while this was not the case for the other three recommendations.

A recent study of PTs and other musculoskeletal practitioners in the UK, reported a 60% guideline adherence rate after an implementation of guidelines for low back pain via posted information, versus 55% in the no-intervention control group; only a 5% absolute difference [52].

Hence, effect sizes in the few guideline implementation studies that have been published in physical therapy varied between 5% and 44%. In other healthcare areas, effects of implementation interventions are known to be modest. Systematic reviews [53, 54] have reported average effects of guideline implementation on performance or process of care between 5% and 10%, consistent with the findings of this study.

The post-intervention rate of frequent guideline use in our study, 55%, can still be considered quite low, leaving room for future improvement. Changing behavior to start using guidelines more frequently is a process that takes time, and we measured only a few months after the implementation activities. A Dutch survey examining guideline adherence three years after a postal dissemination of physical therapy guidelines for low back pain, reported a corresponding self-reported adherence rate of 61% [55], only slightly higher. In other cross-sectional surveys, varying rates of guideline adherence have been reported among PTs in different countries and settings, ranging from 40% in the United States [2] and 45% in Australia [12], to 75%-86% in Sweden [13, 14]. Comparisons are however treacherous, as data were collected and analyzed differently in the studies.

There are several possible explanations for the modest effect on guideline use. Rogers [56] has suggested that the adoption of new innovations is highly influenced by the innovation characteristics and has described five perceived attributes of the implementation object: relative advantage, compatibility, complexity, trialability, and observability. Grilli and Lomas [57] found that of those, trialability (the degree to which an innovation may be tried on a limited basis) was the most important attribute of guidelines, enhancing adherence, and that complexity was associated with low adherence rates. Compatibility with existing clinical practice has also been shown to enhance adherence [58]. In our project, trialability and complexity were addressed already in the development of the guidelines. Because the guideline recommendations did not differ much from existing practice, compatibility was also achieved. However, this very fact could maybe explain the modest effect on the use of guidelines; since they did not introduce any new revolutionary methods but rather confirmed existing practice, using them frequently might not be perceived as necessary.

Generally, it has been suggested that the modest effect of most implementation interventions could be due to the complexity of implementation research [59]. The primary health care context involves many stakeholders at multiple levels, creating significant conceptual and methodological challenges such as developing effective study designs and selecting appropriate outcomes. Furthermore, to achieve a practice change, managerial and organizational support for an implementation project is crucial.

The fact that the per-protocol analysis yielded significant effect on guideline use where the intention-to-treat analysis did not, could either suggest that participating PTs did not share the content of the implementation seminar sufficiently with non-attending colleagues, or that there was a diffusion effect between the intervention and control groups. This finding indicates the importance of implementation seminar attendance, which therefore should be more strongly encouraged or mandated by managers.

Naturally, the positive effect on awareness, knowledge, and access were likely related to the components of the intervention that addressed these outcomes, and the specifically created website was a main contributor. Verbal information at the implementation seminars, supplemented by well-structured guideline information and supporting documents on the website as well as regular e-mail reminders, is likely to have contributed to increased awareness, knowledge and perceived easy access.

Attitudes to EBP and guidelines were for the most part not affected. It would have been difficult to demonstrate statistical significance within this study because participants in both groups were already very positive at baseline; hence there was little room for improvement.

The rather high baseline levels of awareness and use of guidelines (in both groups), also likely contributors to the modest effects on guideline use, could be explained by how the survey questions were phrased and the availability of other guidelines. PTs with good search skills and an interest in searching for guidelines could have searched and found guidelines on their own initiative. The questions were phrased in a rather general way: “are you aware of evidence-based clinical practice guidelines relevant for your practice?” and “how often do you use evidence-based clinical practice guidelines?”. We didn’t ask about the study specific guidelines since they were to be developed. At the time of the follow-up, PTs in the intervention group most likely interpreted the questions as referring to the recently developed and introduced guidelines, whereas the control group most likely interpreted the questions in a general way. Thus, the high baseline rates of guideline use imply that other guidelines than those introduced in the intervention, were used in both groups.

The use of a structured model to guide the implementation project was perceived as useful by the project team. Because the content and recommendations of the guidelines did not differ much from existing practice, complex theories were not considered necessary. However, because the model by Grol et al. [17] is rather pragmatic and integrates several theories in a non-explicit way, it may be less suitable for gaining a deeper understanding of how change was achieved. Linking intervention components, or behavior change techniques, to specific theories of behavior change, could be a more effective strategy to reach this goal [60].

Tailoring implementation components to identified determinants was also perceived as a useful and logical strategy. The core component of the intervention, the implementation seminar, together with the website, addressed most of the identified determinants. The inclusion of interactive group discussions has been suggested as one of the possible tailoring mechanisms that could enhance the impact of educational meetings in implementation [61]. Few studies have been performed that use a tailored strategy to implement change in physical therapy practice. Stevens and Beurskens developed and evaluated a tailored strategy to implement measurement instruments in a Dutch physical therapy context [62]. They concluded, based on interviews with a small group of PTs, that the strategy was applicable to physical therapy practice and effective in improving awareness, knowledge, attitude and use of measurement instruments.

As illustrated in Figure 3, we linked the components of our intervention to the identified determinants, as recommended in the literature [28]. In addition to identifying barriers and facilitators in our population via the baseline data collection, we supplemented these findings with a literature search. Using more than one method to identify determinants is recommended [63]. The largest barrier identified, lack of time, was addressed indirectly by attempting to reduce several of the other barriers, such as poor awareness, availability of and access to guidelines, i.e., decreasing the time required to search for evidence. It was also addressed in the guideline development process, as the guidelines were developed with this barrier in mind. They were designed to be brief and concise. The characteristics of the guidelines were carefully considered as they are likely to affect actual use. Guidelines that are easy to understand, can easily be tried out, and do not require specific resources have a greater chance of implementation [19]. Lack of time is however mainly an organizational or contextual barrier that was not possible to address directly in this intervention.

While social cognitive determinants such as the ones addressed in our study have been shown also in earlier research to influence clinical behavior [64], in particular the use of research evidence [65], a less explored factor is the role of habits in clinical behavior. In a fairly stable context such as the one in this study, clinical practice can be assumed to be habitual and habits, i.e., repeated behaviors, can in stable contexts be difficult to change [50]. Especially in a situation where the guideline recommendations did not differ much from existing practice, many PTs may not have felt a necessity to change their habits and start using guidelines more frequently.

Other possible strategies would have been to use audit and feedback or local opinion leaders. However, according to Grimshaw et al’s recent review on various implementation strategies [21], audit and feedback seem to be less effective than educational strategies. The one strategy that stands out, with a median effect size of 12% versus 4-6% for other strategies, is the use of local opinion leaders. This was not known at the time of our strategy development, nor were there any obvious candidates in our study setting. This strategy would require the availability of credible and influential persons, as well as an intact social professional network [21]. With those conditions satisfied, it would be a recommended strategy in future implementation endeavours.

The study population is likely to be broadly similar to the national population of primary care PTs in Sweden. Although the study was conducted in only one county council/geographic region, this particular county council is the second largest in Sweden. Thus, we believe that our results can be generalized to PTs working in primary care in other county councils in Sweden and possibly also to other countries, particularly those in which, like Sweden, PTs have a large degree of autonomy.

The study has several limitations. Only process outcomes at practitioner level were measured and we do not know whether the intervention had, or will have, an impact on patient outcomes. We only measured self-reported behavior, which entails a risk of overestimation and social desirability bias, and limits our ability to draw strong conclusions concerning actual behavior [66].

To minimize social desirability bias, the questionnaires were anonymous and it was therefore not possible to track individual responses and changes between the two test occasions. Statistically, only comparisons between independent groups were possible. However, staff turnover was rather low during the study period, with an average of 90% of PTs remaining the same in the respective groups from baseline to follow-up measurements. The study population can therefore be considered rather stable.

As with all non-randomized studies, it cannot be fully ascertained that the observed changes can be attributed to the intervention. Randomization was in this project not feasible, as a likely diffusion effect between participants would have made data interpretation difficult. Instead, the second strongest design, a controlled before–after design with a comparable control group, was used [59]. Data were collected in both populations contemporaneously using the same method at baseline and follow-up. A ‘between group’ analysis comparing performance in the study and control groups was performed, and the observed differences could be assumed to be due to the intervention [67]. Both groups were similar at baseline and could be expected to experience secular trends or changes in the same extent, increasing the confidence with which the observed changes can be attributed to the intervention [67].

The short duration between implementation and follow-up evaluation is likely to have contributed to the modest effect on guideline use. The introduction of complex workplace change, especially as concerns EBP, takes considerable time [68]. The actual use of, or adherence to, a guideline has been described as the last in a sequence of cognitive and behavioral steps that begin with awareness [47], and to expect a high impact on guideline use so soon after the intervention is probably not realistic. A longer follow-up period would have been useful but was not possible due to the guidelines and website being made available also to the control group after the follow-up survey was conducted.

Another limitation, possibly contributing to the modest effect on guideline use, could be that the education component of the implementation intervention comprised only one session. Two systematic reviews have shown that educational meetings can improve both professional practice and patient outcomes and that sessions with interactive components may have greater impact [23, 69]. However, the effectiveness of various educational strategies is not conclusive and factors such as time and intensity have been proposed to influence outcomes [18]. One single session may not be enough, although we took effort to make it as interactive as possible. Interactive educational strategies have consistently been reported as more effective than traditional, didactic education and engaging the clinician is recommended [18]. Additional educational sessions might improve the effect of the implementation, but this would need to be investigated.

The intervention focused on the individual level. Mid-level managers have a potentially pivotal role as implementation agents and their proactivity and commitment have been suggested to influence implementation effectiveness [70]. A multi-level approach and particularly targeting also unit managers would likely have increased the impact of our implementation intervention.

Measuring the effect of implementation interventions as well as guideline use is very complex, but very important. To build upon the results of this study, further research is recommended that evaluates an implementation strategy that comprises more than one educational session, and that is designed to enable a longer follow-up period, e.g., two years. Using a more rigorous design, i.e., a randomized trial, is paramount to reduce risk of bias and be able to draw stronger conclusions. In view of the modest effect on guideline use and the considerable resources required for an active, multi-component intervention such as the one in this study, a comparison between different implementation strategies with various components could provide valuable knowledge. A strategy that needs to be evaluated in the physical therapy context would be the use of local opinion leaders. Linking intervention components, or behavior change techniques, to specific theories of behavior change could facilitate a greater understanding of what works and what doesn’t. Data should be collected so that individual changes from baseline to follow-up can be analyzed. It is also very important to measure the effect on patient outcomes as well as the cost-effectiveness of guideline implementation strategies. Furthermore, quantitative evaluation studies need to be accompanied by qualitative studies to achieve increased understanding of effect mechanisms of implementation interventions and practice behavior change. Lastly, the patient’s voice is seldom heard in implementation research and patient views and preferences should be included in an evaluation of the barriers and facilitators for implementation, development of the implementation intervention, and in choosing relevant outcomes.