Implementation of internet-delivered cognitive behavior therapy within community mental health clinics: a process evaluation using the consolidated framework for implementation research

This process evaluation aimed to identify barriers and facilitators that influenced ICBT implementation in seven community mental health clinics distributed across one province. A bespoke survey, designed around CFIR constructs, consisting of open- and closed-ended questions, was administered to therapists and managers approximately 2 years after ICBT for depression and anxiety was implemented in the clinics.

Participants included managers and therapists who worked in seven community mental health clinics across Saskatchewan who were involved in ICBT implementation at the time this survey was conducted. Completion of the survey was high, with 11 out of 12 managers (n = 3 psychologists; n = 5 social workers; n = 2 nurses; n = 1 administrators; n = 2 male; n = 9 female) and 22 out of 23 therapists (n = 7 psychologists; n = 11 social workers; n = 3 nurses; n = 1 counselor; n = 2 male; n = 20 female) who were contacted agreeing to participate. On average, therapists who participated had provided ICBT to 11.41 (SD = 7.12) patients at the time they completed the survey. Of note, we did not contact 17 therapists who provided ICBT in the past, but had retired or resigned from their position at the time of the survey.

Research ethics board approval for the study was obtained from the University of Regina. All participants provided electronic informed consent and were offered a $10 coffee card as a token of appreciation. The secure online survey (available in Additional file 1) was administered using the Qualtrics software program. After completing background questions, participants rated their perceptions of the importance of ICBT (e.g., whether residents should have access to ICBT, whether involvement with ICBT implementation was worth their time) on a scale from 1 “strongly disagree” to 5 “strongly agree”. Participants were then asked their opinions regarding the number of patients who should receive ICBT (i.e., “many more”, “somewhat more”, “the same number”, “somewhat fewer”, or “none”). Participants were then provided with a description of each CFIR domain (intervention characteristics, outer setting, inner setting, individual characteristics, and implementation process) and were asked to comment on both positive and negative aspects of the domain on ICBT implementation. For example, the following question was posed: “The inner clinic setting (e.g., structure, communication, culture) can influence program implementation efforts. Please comment on any positive or negative factors within your clinic setting that you feel may have influenced the implementation of ICBT within your health region.” After providing open-ended feedback, participants rated agreement with a series of 47 statements (see Table 5 for items) related to ICBT implementation on a scale from 1 (“strongly disagree”) to 5 (“strongly agree”). The statements referenced facets of each CFIR domain. Example questions included: “It is positive that the ICBT Wellbeing Course was developed externally and the health region did not have to develop our own ICBT program” (Intervention Characteristics); “My health care region is aware of the high need for mental health care” (Outer Setting); “My health region has an adequate number of therapists available to deliver ICBT” (Inner Setting); “Therapists in my health region have adequate knowledge of ICBT” (Individual Characteristics); and, “We spent adequate time planning how to deliver ICBT in advance in my health region” (Implementation Processes).

Responses to open-ended questions were exported from Qualtrics into NVivo Version 10, a qualitative data software tool. The CFIR constructs served as a coding framework with text classified into pre-existing categories (see http://​cfirguide.​org/​constructs.​html for definitions) using deductive principles [17] to identify the presence of such categories in the dataset. Two coders read and coded data together reaching consensus on the CFIR domain and construct. Each sentence could contain multiple codes. Alternatively, concurrent sentences of the same thematic code could be conjoined into one unit. Once initial coding was complete, the coders examined quotes assigned to each construct as a whole and any quotes that were identified as inconsistent with the designated construct were re-examined to determine the most appropriate construct. A third researcher then reviewed quotes representing each construct and identified any points of disagreement that were then resolved through consensus. In addition to coding CFIR constructs, the coders classified comments as either positive (facilitator) or negative (barrier) in tone. Following all coding, NVivo’s query function was used to calculate the total frequency of each construct. The frequency score represents the total number of times the construct was present in the survey (see Table 4 for frequencies and representative quotes). At times, participants made recommendations to improve ICBT implementation. These comments were also classified according to domain to identify CFIR domains requiring improvement. Descriptive statistics were calculated to examine responses to rating scale survey questions (see Table 5). Ratings above 4 were interpreted as positive perceptions of implementation, while ratings below 3 were interpreted as areas of potential improvement.

There was strong indication that both therapists and managers had positive perceptions of ICBT. Specifically, both agreed strongly that Saskatchewan residents should have access to ICBT (M = 4.61/5, SD = .50), that health regions should be committed to ensuring access to ICBT (M = 4.12/5; SD = 1.05), that health regions should identify barriers and facilitators associated with ICBT (M = 4.12/5; SD = 1.05), that health regions should continuously monitor and evaluate ICBT (M = 4.06; SD = 1.25) and, that it was worth their time to be involved in ICBT (M = 4.39; SD = .79). Also reflecting positive perceptions of ICBT, ~45% of participants indicated that a somewhat greater number of patients should receive ICBT in the future, ~21% supported treating about the same number of patients, and promisingly ~15% reported that many more patients should receive ICBT. Very few participants indicated that the health regions were treating too many patients (9.1%) or should not provide ICBT (9.1%).

Table 4 lists the frequency of positive and negative comments along with representative quotes made by participants related to each CFIR construct. Table 5 displays mean ratings on statements related to the CFIR constructs. The findings are summarized below by CFIR domain with attention given to recommendations for improving implementation as well as differences between therapists and managers.

Intervention characteristics and implementation processes appeared to have the greatest positive impact on ICBT implementation. In terms of the intervention, the relative advantages, design quality, and strength of evidence of the ICBT program appeared to be important factors facilitating ICBT implementation. Examination of participants’ responses to rating scales revealed other aspects of the intervention that also likely facilitated ICBT implementation including the transdiagnostic nature of the program, the brevity and effectiveness of the program, and low cost and external development of the program. Processes used to implement ICBT also served to facilitate ICBT implementation, especially in terms of engaging diverse stakeholders in ICBT implementation (e.g., patients, therapists). Participants also reported a positive perception of the diverse actions taken by the external facilitation Unit to implement ICBT in the clinics (e.g., managing the website, providing education and technical and clinical assistance, securing external funding). In comparison, the outer setting, inner setting, and individual characteristics were less consistently regarded as having facilitated ICBT implementation.

The inner setting emerged as representing the greatest barrier to ICBT implementation especially in terms of lack of readiness for implementation and implementation climate. Specifically, ICBT implementation was considered to be negatively impacted by limited resources being allocated to ICBT and perceptions that ICBT was of lower relative priority compared to face-to-face therapy. Of note, while therapists, on average, treated only 11 patients, this was still deemed to be taxing and difficult to integrate with existing patient workloads. Participants also felt inadequate attention was given to setting goals, reflecting and addressing challenges, and providing incentives and rewards to therapists; these factors were considered to hinder implementation.

While the inner setting clearly emerged as the greatest barrier to ICBT implementation, analyses suggested that there were some additional barriers to implementation. Low adaptability of ICBT in terms of support and length of treatment was perceived as a barrier to implementation. Participants also indicated that ICBT implementation would have been greater if additional clinical concerns were addressed with ICBT (e.g., alcohol or drug problems). Other barriers to implementation included negative beliefs about ICBT among some therapists and the need for even greater engagement of stakeholders in ICBT.

The results of the current study show consistency with a recent systematic review on factors that influence implementation of e-health interventions [10]. Consistent with this recent review, we found that ICBT implementation amongst mental health clinics was impacted by diverse constructs and no single construct explained implementation. Furthermore, similar to this review, we identified that the inner setting implementation climate negatively impacted ICBT implementation and implementation processes, such as planning, engaging, reflecting and evaluating, positively impacted implementation. Unique from this review, in the current study, we found that ICBT implementation was significantly facilitated by the relative advantages of ICBT, the quality of the ICBT program, and evidence supporting ICBT. These variables did not emerge as prominently in the review of e-health implementation; this could reflect that Ross et al. examined many forms of e-health (e.g., electronic medical records, decision support) and that intervention characteristics play a more prominent role when it comes to implementation of therapeutic programs. Evidence supporting ICBT may have been a particularly strong facilitator in our study as the implementation approach involved an initial workshop that incorporated evidence supporting ICBT. Furthermore, on a regular basis, we provided clinics with information on outcomes of ICBT and also regularly posted results of studies on ICBT on the Unit website. A further discrepancy is that in the past review, external policies and incentives were identified as highly beneficial for e-health implementation. This variable did not emerge in the current study, but could reflect that participants in our study were therapists and managers as compared to those involved in overseeing health policy. Of interest, in the current study relative priority of e-health within the inner setting emerged as a prominent barrier to implementation; Ross et al., [10] identified no past research on this construct suggesting the present study makes a unique contribution to the e-health implementation research.

Overall, we found CFIR to be a useful framework for guiding and examining multiple contextual factors impacting ICBT implementation. Similar to others, the CFIR terminology and language were coherent [12]. We did not find any need to add categories to the CFIR framework. In the future, it would be valuable to explore whether the variables that were identified as impacting ICBT implementation in the current context also explain implementation in other clinical settings. In this study, we used a bespoke survey that may prove helpful to others who are interested in evaluating implementation with respect to CFIR. This format had the advantage of being cost-effective for capturing information across multiple sites.

It is critical to study ICBT from an implementation perspective given that barriers and facilitators to implementation can impact the utility and sustainability of ICBT. Most past research on ICBT has focussed on ICBT within specialized clinics; however, it is important to consider implementation within settings where it may be more challenging to offer ICBT given multiple demands on therapist time. This is the first multi-site evaluation of the factors impacting uptake of ICBT in community mental health clinics. Lessons learned from implementation studies assist with local implementation, but also support widespread adoption of an intervention, allowing others to consider the strengths and challenges of using the facilitation model to implement ICBT, and to anticipate specific barriers and facilitators to ICBT implementation [12].

An additional strength of this study is that we utilized the CFIR to understand implementation barriers and facilitators; this is a comprehensive model built on the review of many other implementation theories [12]. We used both open-ended questions and rating scales to explore implementation processes and information was collected from two perspectives (managers and therapists). While there was convergence between methods of data collection, the ratings scales appeared most useful for identifying facilitators of ICBT implementation, while the open-ended responses were helpful for understanding barriers and potential areas of improvement. A final strength is the high participation rate.

In terms of limitations, it should be noted that qualitative analyses have the potential to be subjective and influenced by coder bias. In order to offset this bias, however, multiple coders were used and quantitative ratings supplemented the qualitative findings. A further limitation of this study is the sample size which could limit generalizability of the findings. The sample size also limited our ability to examine differences in implementation experiences between managers and therapists and uncover differences between clinics with higher and lower ICBT implementation. Also of note, there was a high degree of staff turnover and 17 therapists who provided ICBT had either resigned or retired prior to conducting the survey and therefore did not participate. It should be noted that facilitators and barriers to ICBT implementation are likely linked to the model of implementation, the local population, and the health care system. For example, with different implementation models, there may be less emphasis placed on sharing evidence regarding the intervention and this factor may impact implementation in a different fashion than what was found in our study. As another example, in a different region, funded by private insurance or with better access to mental health care, different facilitators and barriers to ICBT implementation may emerge. Future research is needed to examine ICBT implementation in different settings, with different populations, using different models. The open-ended questions and rating scales were developed specifically for this study. It is possible that findings may have varied if we used a different framework [19], formulated different questions, or utilized a different method of study (e.g., focus groups). Given our interest in examining facilitators and barriers to ICBT implementation, all participants in this study were directly involved in ICBT implementation. Perceptions of ICBT were highly favorable among this group and could be quite different among providers without experience with ICBT. It should also be noted that we assessed barriers and facilitators retrospectively after participants had experience with ICBT. Given the retrospective observational methodology, it is difficult to fully disentangle which facilitators and barriers were most important.

There are a number of strategies that could potentially improve ICBT implementation in the future that are important for other groups to consider who have an interest in using a similar facilitation model to implement ICBT. In terms of the intervention, it would be beneficial to explore if implementation could be improved if the ICBT program was adapted to meet the needs of the patients, especially in terms of program length and amount of therapist support. Rather than offering standardized ICBT it may be valuable to explore if reach, outcomes, and implementation of ICBT could be improved if ICBT were personalized to the patient. This would be consistent with the notion of “patient-centered care”, which refers to “providing care that is respectful of and responsive to individual patient preferences, needs, and values” , p. 6 [20]. There is considerable research demonstrating that patient-centered care is associated with greater patient satisfaction and self-management [21]. Although it seems intuitive that personalized ICBT would improve adherence and outcomes, it is important to examine this systematically. Another possibility is that personalization may contribute to non-therapeutic discussion and drift away from the key tasks of ICBT, ultimately resulting in a reduction in patient adherence and outcomes.

In terms of the inner setting, it would be valuable to systematically examine if uptake could be improved if adjustments were made to resources dedicated to ICBT. Some participants suggested that ICBT should only be offered by a centralized unit while others suggested that settings other than mental health clinics should deliver ICBT (e.g., primary health care clinics). Others felt that within the clinics it would be desirable to have either fewer therapists or alternatively more therapists deliver ICBT. Still others felt that therapists should be given more time to offer ICBT. Of interest, within the OTU, consistent with how ICBT is described in the literature [18], most therapists need 15 to 20 min, per patient, per week to provide support. This reflects that ICBT materials reduce the need for therapists to provide information to patients. A number of therapists in the clinics, however, reported taking 45 min or more to provide support. This could reflect lack of experience or that therapists are having difficulties with the model of care and attempting to provide the same level of support that they provide in face-to-face care. It is possible that spending additional time on addressing inner setting barriers to ICBT implementation, such as workflow, could have resulted in greater use of ICBT beyond current levels. Past research has shown, for example, that compatibility of e-health programs with workflow positively facilitates e-health implementation [10]. In general, in the current trial, no specific goals were set for implementation, internal champions were not systematically identified, and incentives for delivering ICBT were not used; these approaches could represent additional strategies that could be examined to increase uptake in the future. It is also quite possible that addressing beliefs about ICBT, such as beliefs that face-to-face care is of greater priority than ICBT, could also serve to improve ICBT uptake in the future.

Another suggestion for improving ICBT implementation would be to explore if greater time and resources for engagement would improve uptake and use of ICBT. Among participants, there was a call to systematically examine ICBT as a first step in care prior to being offered face-to-face services. Currently, ~15% of the patients who receive ICBT report they are on waiting list for face-to-face services. It would be valuable to systematically examine whether need for face-to-face care is reduced if patients receive ICBT while waiting for face-to-face care. A previous study did not find offering ICBT to patients who were waiting for face-to-face treatment reduced the need for subsequent face-to-face care [22]. This finding requires replication, however, as the research was observational in nature. In terms of other directions, many participants, in the current study, requested more information about ICBT utilization and outcomes presented in a more engaging format. It would be valuable to examine whether providing this information would improve uptake and use of ICBT beyond current levels. Lastly, it would be valuable to examine the generalizability of the current findings to other regions in Canada or across the globe.