Background
Numerous interventions for adolescent substance use disorders (SUDs) have been developed, tested and supported by empirical evidence, yet of the two million 12- to 17-year-olds in need of SUD treatment, only about 8% actually receive it [
1]. Providing high quality care to those youth who access addiction treatment can mitigate the adverse consequences of substance use, including both short and long-term violence, accidents, disease, and criminal behavior [
1],[
2]. Ensuring the provision of quality care also can bolster community confidence in treatment: this, in turn, may lead more families, courts and schools to refer youth to treatment and help ensure that adolescents in need of care receive it.
One strategy policymakers use to ensure high quality treatment is to offer discretionary monies that encourage community-based programs to adopt treatment protocols deemed efficacious in experimental settings (
i.e., evidence-based treatments or EBTs). For example, government agencies such as the Substance Abuse and Mental Health Services Administration (SAMHSA) have offered discretionary grant funding in order to help facilitate EBT implementation. In one of the largest such efforts to date, the SAMHSA’s Center for Substance Abuse Treatment (CSAT) provided over 80 million dollars to support community-based SUD treatment organizations to implement the Adolescent Community Reinforcement Approach (A-CRA; [
3]), an EBT that has yielded positive outcomes in relation to adolescent alcohol use, mental health, and social functioning in three randomized controlled trials (RCTs; [
4]-[
6]). These discretionary grants provided on average $300,000 annually for approximately three years to support local implementation and evaluation including a multi-day A-CRA training, technology-assisted performance feedback, and a standardized certification process for both clinicians and supervisors provided by the treatment developers.
While EBT dissemination using intense support from treatment developers has been shown to improve implementation quality by clinicians in private practice [
7],[
8] and those who work in community-based organizations [
9],[
10], little is known about: a) the extent to which such federal grant initiatives lead to sustained EBT implementation and b) the factors that are associated with sustained implementation [
11]-[
14]. To understand A-CRA sustainment, that is ‘the continued use of an innovation in practice’ [
15], we developed a measure consistent with Fixsen
et al.’s [
11] and Scheirer and Dearing’s [
16] definition of implementation quality by assessing both the organizational supports and delivery components of the treatment. Understanding EBT sustainment using both factors has not been well-utilized in previous sustainment research [
17]. To date, most reported program sustainment studies have relied on self-reports rather than more objective measures of implementation quality.
There is a lack of empirical evidence on the factors that predict EBT sustainment, and this study is uniquely positioned to study several factors that have been theorized to be related to sustainment. When planning this study in early 2009, we turned to the public health literature on program sustainability to identify factors to examine. We discovered that conceptual frameworks and models of change have been slow to develop, partially due to the diversity in the definition of terms to characterize program sustainment [
17]-[
19]. We reviewed literature regarding the seminal work on innovation diffusion in organizations [
20] and its application to health service organizations [
21], the uptake and implementation of evidence-based practices in medical, mental and public health contexts [
11],[
18],[
22], organizational change and the adoption of EBTs within SUD treatment programs [
23],[
24], and social or health service program sustainment [
25]-[
28]. In reviewing this wide spectrum, we identified four main factors theorized to influence program sustainment: a) the broader community environment, external to the organization implementing the EBT; b) the level of implementation during the funded period; c) factors within the organizational setting, such as leadership support; and d) intervention/innovation characteristics. These factors are consistent with Damschroder and colleagues’ Consolidated Framework for Implementation Research (CFIR), a conceptual approach that is based on a systematic review of the implementation literature published soon after we conceptualized our study [
29].
In the CFIR model, the broader community environment is considered the ‘outer context’, that is, those factors outside of the organization under study that influence implementation and sustainment. In this study, we examine policy, regulatory, or fiscal facilitators or barriers, such as whether providers are reimbursed for providing the EBT to their clients [
11], community leader involvement [
26],[
30], and the degree of community need for the treatment [
31].
The level of implementation refers to how well an EBT was delivered during the funding period. In this study, we assess level of implementation in reference to employing clinical and supervisory staff who have demonstrated competency through the certification processes and an organization having recruited and treated a sufficient number of participants during the initial implementation period, also termed as ‘penetration’ [
32],[
33]. Integral to Rogers’ [
20] innovation diffusion theory is that the implementation experience will have a large influence on sustainment. Poor implementation may lead to ‘discontinuance’ (
i.e., rejection) of an innovation. We have available participant process (
e.g., treatment initiation, engagement, satisfaction) and outcome data to help explicate implementation during the funding period. Opinions are mixed as to whether these data will help predict level of sustainment [
34]. Intuitively, it is sensible to expect that the extent to which clinicians are able to engage participants in an intervention or participants are able to achieve a treatment’s objectives would be related to sustainment. However, such factors as external (
i.e., community) or internal (
i.e., organizational) support may trump any efforts to sustain an effective EBT. Therefore, we examine these issues in our study.
Factors within the organizational setting refers to characteristics of the ‘inner context’ [
29], such as institutional (
e.g., leadership support, resources within the organization for EBT delivery), organizational climate [
35], and staff attributes (
e.g., presence of a program champion, motivation and skills to deliver the EBT). Although studies have shown that staff turnover is not predictive of sustainment [
17], we believe that with a complex treatment like A-CRA, attrition of trained staff will impede sustainment. Intervention characteristics, such as community-based providers’ perceptions of the treatment, may influence whether it is sustained [
20]. It is not known whether program leadership will still be in support of the treatment at the end of the grant period and whether staff will be motivated and trained to deliver it, so we plan to assess these factors in this study.
Relationships among variables
It is important to acknowledge that the four factors described above do not operate in isolation from one another [
22]. Researchers have emphasized the dynamic context of the community in fostering sustainment, whereas both Gruen
et al.’s [
25] and Shortell’s [
22] work emphasized the interaction between components (
e.g., tailoring a treatment to organizations and the community context) and over time (
e.g., integrating the treatment within existing structures). To better address this point, we will assess these factors through multiple methods across time, including qualitative semi-structured interviews and standardized survey questions that will help us better ascertain the association among these factors. For example, we will ask staff about how the treatment was or was not integrated into their existing organization and whether adaptations were made to the treatment to make it feasible to continue its delivery without the support provided during the funding period.
Previous studies
It has been historically difficult to study EBT sustainment. After initial funding for EBT implementation ends, resources are seldom available to continue studying implementation [
17]. Therefore, most knowledge about sustainment relies on anecdotal evidence, case studies, or highly controlled experiments that have limited external validity [
36]. Recently there have been increased attention to assessing program sustainment [
16],[
37],[
38], but little research has been published in the SUD treatment field. An exception is the work examining pharmacotherapy treatments [
39]-[
43]. Behavioral treatments, however, are much more likely to be used in SUD treatment settings [
44], and the examination of behavioral treatments warrants different approaches than medication treatments, as the resources needed to implement a medication regime differ from a behavioral treatment. For instance, access to physicians is critical to medication regimes [
39], but not to the implementation of behavioral interventions. Furthermore, previous studies have relied on self-report data from program administrators to characterize sustainment. In this study, we plan to determine which factors lead to A-CRA sustainment using more complex implementation quality measures.
In sum, this study will address an important gap in implementation research by examining whether and to what extent an EBT is sustained in usual care practice settings after initial support ends. Additionally, this study will be able to assess factors that predict sustainment because of the implementation and organizational data that have been collected during the funding period and the longitudinal study design that examines program sustainment over a relatively large sample of organizations.
Study aims and hypotheses
The first study aim is to characterize levels of A-CRA implementation longitudinally for 84 programs that received federal funding to implement A-CRA. In doing so, we will identify which aspects of A-CRA are sustained, innovations in its implementation, and potential facilitators and barriers to implementation. We will also characterize the trajectory of implementation quality among the 84 programs over time. We hypothesize that substantial changes in implementation quality will occur after funding ends. Specifically, we expect to observe an overall decrease in sustainment over time.
The second study aim is to empirically evaluate factors that predict the degree to which programs sustain A-CRA. We hypothesize that the extent to which A-CRA is sustained will be associated with community/outer context (e.g., reimbursement for services), organizational/inner context factors (e.g., presence of trained supervisors at the end of the funding phase), and intervention-specific characteristics (e.g., staff support for A-CRA).
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
SH, BH, BG and SG conceptualized the study. SH is the PI and has overall responsibility for the execution of the project. LA was primarily responsible for the manuscript draft and is responsible for study coordination. BH conceptualized the study’s analytical plan. BG and SG contributed to the study proposal and manuscript draft. All authors were involved in developing and editing of the manuscript and have given final approval of the submitted version.