Background
Sustainment, ‘the continued use of program components and activities for the continued achievement of desirable program and population outcomes’ [
1], is an understudied area within implementation science [
2],[
3]. Fortunately, the burgeoning of prospective implementation trials has poised the field to explore sustainment following implementation [
2]. An understanding of sustainment and associated factors can be informed by an ecological perspective, such as the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework [
4]. The EPIS is one of the first frameworks to delineate a separate stage for sustainment, as well as offer guidance regarding contextual factors likely to impact sustainment [
4].
A seminal review of sustainment found that many naturalistic observational studies (N = 125) have been conducted following implementation of an innovation. However, the authors caution that these studies are hampered by their insufficient methodological rigor, making it difficult to draw conclusions and necessitating more empirical study [
2]. Importantly, there are several areas of weaknesses in the studies conducted to date. First, and most strikingly, only 6% of the studies investigated sustainment following experimental manipulation of implementation strategies. Second, less than half of the studies reported quantitative outcomes, such as the proportion of sites or providers sustaining an EBP. Third, of those studies that did report quantitative outcomes, the majority of investigations only examined penetration [
2], which refers to ‘the integration of a practice within a service setting and its subsystems’ [
5]. Other important sustainment outcomes were not examined. Fourth, when examining predictors of sustainment, only 30 studies used quantitative methods, and of those studies, only 20 were guided by a conceptual framework [
2]. Given these limitations, the current study contributes to the empirical literature on sustainment by: investigating sustainment following an experimental manipulation of an implementation strategy (
i.e., training); examining a number of sustainment outcomes quantitatively (
i.e., penetration, components of treatment utilized, knowledge, and attitudes); and examining a predictor of sustainment guided by a conceptual framework (
i.e., EPIS) [
4].
The primary aims were to examine sustainment for clinicians two years following receiving training and consultation in cognitive-behavioral therapy (CBT) for youth anxiety [
6], an evidence-based practice [
7]. This time length was chosen given that it takes two to four years for programs to be institutionalized [
8]. We selected penetration, the self-reported percentage of anxious youth treated with CBT [
2], as our primary sustainment outcome given that it is an important implementation outcome [
5] and to build upon the previous literature [
2]. In an effort to broaden understanding of sustainment, we also examined sustainment of specific components of CBT and of two individual adopter characteristics from the EPIS framework [
4]: knowledge of CBT and general attitudes regarding evidence-based practices (appeal, requirements, openness, and divergence). Although knowledge and attitudes are typically described as moderators of implementation, we were interested in examining knowledge and attitudes over time to see if initial changes in these constructs at the implementation phase maintained into the sustainment phase. Furthermore, consultation session attendance was examined as a moderator of the change in CBT knowledge and EBP attitudes across all time-points, given that consultation attendance was associated with an increase in treatment fidelity (
i.e., skill; adherence) in the initial study [
6].
Discussion
The present study examined sustainment of CBT for anxious youth two years following participation in an implementation trial using educational implementation strategies (
i.e., training and consultation). Guided by the EPIS framework, the sustainment of two individual adopter characteristics (
i.e., knowledge and attitudes) was also examined [
4]. Time spent in consultation sessions was examined as a potential moderator of change in knowledge and attitudes over time based on previous literature. Given that participation was largely driven by individual choice, consultation participation can also be considered an individual adopter characteristic [
4]. Findings indicate sustainment of penetration rates from post-consultation to follow-up although not all intervention elements were equally delivered. Therapists reported low usage rates of specific components of the treatment, specifically exposure tasks, the active ingredient of CBT for child anxiety [
24]. Although knowledge scores at follow-up were significantly higher than at baseline, maximal gains in knowledge achieved by post-consultation were not sustained. Attitudinal improvements toward EBPs following training and consultation were also generally not sustained. Time spent in consultation significantly moderated changes in knowledge and attitudes toward EBPs, suggesting an important mutable target for implementation and sustainment strategies moving forward. Taken together, these results suggest important implications for future implementation endeavors.
Self-reported penetration of CBT with anxious youth was sustained from post-consultation (79%) to follow-up (83%). Fidelity was not measured at the two-year follow-up, but therapists were asked to report on components of CBT for child anxiety that they had used with anxious youth. These self-report ratings suggest that CBT may have not been implemented with high fidelity. Clinicians reported extensively implementing identification and management of somatic arousal and anxious cognitions, rewards, and problem-solving. Exposure tasks, a key component of CBT for anxiety, were implemented much less extensively. This is concerning given that the exposure tasks are critical and are viewed as core components of the intervention [
24]. When considering the treatment components needed to attain fidelity, even with ‘flexibility’, [
25], exposure tasks are required [
26]. In light of these findings, implementation efforts using training and consultation as implementation strategies may benefit from spending additional time discussing the importance and application of exposure tasks, including how clinicians might tailor exposures within their organization. Of note, approximately 30% of the consultation sessions in the present study focused on exposure tasks, suggesting that spending more time in consultation on exposure is necessary [
21], although the specific amount of time needed in training and consultation to dedicate to exposure remains unknown.
Knowledge increases throughout the training and consultation phase were not sustained at two-year follow-up, although knowledge remained higher than at baseline. In order to identify contextual factors that contribute to sustainment, and in accord with the EPIS framework, an individual adopter characteristic (
i.e., time spent in consultation) was examined as a potential moderator given previous findings (see Beidas
et al., 2012) [
4],[
6]. Greater participation in consultation resulted in greater sustainment in knowledge at the follow-up period. This finding is consistent with previously reported findings and provides further evidence of the benefits of consultation following workshop training in order to improve and sustain knowledge [
6]. It is worth noting that follow-up participants spent more time in consultation than non-participants. Thus, higher than average participation in consultation within a group already evidencing greater consultation than the general group appeared to contribute to positive results, further highlighting consultation as an important educational implementation strategy.
Disappointingly, improvements in attitudes gained through the training and consultation did not sustain over time. The pattern observed across two of the four attitudinal subscales measured by the EBPAS (appeal, requirements) suggested that attitudinal gains made by participating in training and consultation were not maintained. A similar pattern emerged with regard to openness to new practices, but surprisingly, not only were gains not maintained, but clinicians exhibited less openness to new practices at follow-up when compared to baseline. Belief in the utility of EBPs exhibited a different pattern. This attitude did not change from baseline to post-training, decreased from post-training to post-consultation, and then increased to baseline levels from post-consultation to follow-up. There is little literature to ground these findings within because few studies have investigated changes in attitudes over this long of a period (
i.e., sustainment). Many studies measure attitudes pre- and post-training [
27], and, typically, attitudes do improve following training as observed from baseline to post-consultation in this study (see Beidas & Kendall, 2010) [
28]. However, this study demonstrates that new information is gleaned through measurement of attitudes at the sustainment time-point. With regard to the curious finding that openness to new practices dropped at follow-up when compared to baseline, one explanation may be that therapists who were initially very open to new practices became aware that a significant time investment (
i.e., training and ongoing consultation) is necessary to gain mastery in a new practice, making them less open to new practices in the future. It is also possible that clinicians encountered difficulty implementing exposures on their own during the follow-up period, which may have resulted in less openness to CBT given its emphasis on exposures. With regard to the finding that belief in the utility of EBPs increased from post-consultation to follow-up, it is possible that after having two years to apply CBT for youth anxiety, therapists had more time to see the potential benefits of the treatment and generalized this to other EBPs.
Consultation emerged as an important moderator with regard to both knowledge and attitudes toward EBPs. As noted above, attitudinal increases from training and consultation were not maintained at follow-up. However, number of minutes spent in consultation impacted sustainment of attitudes. In other words, spending more time in consultation resulted in higher attitudes with regard to requirements, appeal, and divergence as measured by the EBPAS. It may be that greater exposure to CBT for child anxiety through consultation highly impacted therapists’ attitudes toward whether or not they would be likely to adopt EBPs if they were required to, found them appealing, or believed in their utility. Importantly, the results from this study suggest that consultation is an implementation strategy that can be leveraged to change both knowledge and attitudes towards EBPs over time. In other words, the impact of three months of consultation provided in relatively low dosage (maximum of 1 hour a week for 12 weeks) can be continued to provide a return on investment up to twoyears later. It is important to also observe if consultation has a similar impact on fidelity and/or client outcomes in future studies. Also, given that consultation participation relies on individual choice to an extent, and thus can be considered an individual adopter characteristic, it is important to understand what factors predict greater participation in consultation.
This study adds to the limited literature regarding the sustainment of penetration, knowledge and attitudes over time. Strengths of the study include the two-year span of time examined as well as the inclusion of an individual adopter characteristic (
i.e., consultation participation) as a potential moderator of sustainment. Additionally, this study looked at numerous sustainment outcomes, including outcomes not commonly examined (
e.g., knowledge and attitudes). However, limitations should be noted. Despite best efforts, only 43% of the original sample participated. A lower proportion of Hispanics/Latinos participated in the follow-up study as compared to the original study. Also, a difference in consultation session attendance was found across samples. Thus, it is possible that findings from the current study do not generalize to the full sample. Given the specific focus on CBT for youth anxiety, it is possible that findings do not generalize to other interventions [
29]. Other limitations pertain to measures used, including investigator-created measures and the use of self-report.
Additionally, the current study only examined one individual adopter characteristic as a potential moderator of sustainment, whereas the EPIS framework encourages examination of numerous additional inner context factors, intervention characteristics, and outer context factors [
4]. Inner context factors include intra-organizational and individual adopter characteristics, whereas outer context factors include characteristics of the service environment, inter-organizational environment, and consumer support/advocacy [
4]. Also, given that participants were not randomized to amounts of consultation attendance, it is possible that attendance was confounded by motivation; participants who spent more time in consultation may have had higher motivation as well as greater knowledge and more positive attitudes. Furthermore, due to time and resource constraints, the follow-up study did not include an assessment of treatment fidelity, preventing conclusions regarding the sustainment of this important implementation outcome. Future work on sustainment is needed to address these limitations. To gain a better understanding of change over time, future investigations should study sustainment over even longer periods of time. With regard to measurement, future work should incorporate multiple outcome measures, including observational fidelity (
e.g., coded therapy sessions) and client outcomes. Examination of potential moderators of sustainment, informed by an ecological framework, such as EPIS, will allow for identification of mutable targets to improve implementation and sustainment efforts [
4].
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
JME conceptualized the follow-up study, contributed to data collection and coding, and was the predominant contributor to this article. RSB conceptualized the training study and follow-up study, contributed to the data collection, and was a key contributor to this article. KLR contributed to coding, completed data analyses, and assisted with interpretation of results. VAR and DMB contributed to coding and assisted with interpretation of results. PCK assisted in conceptualization of both the original training study and the follow-up study and assisted in interpretation of results. All authors read and modified drafts and approved the final manuscript.