Background
The last two decades have shown increasing emphasis on the implementation of evidence-based practices (EBPs) in publicly funded behavioral health systems nationally [
1,
2]. Policy-makers in public behavioral health systems (e.g., City of Philadelphia, Los Angeles County, Washington, Hawaii, New York) have committed to using EBPs [
1,
3‐
6] to improve the quality of psychosocial services and client outcomes [
7‐
9] using various approaches including tying reimbursement to EBP use (i.e., financial incentives), building EBPs into contracts, and policy initiatives [
2,
10,
11]. Although many public behavioral health systems have invested in implementing EBPs, very few of these efforts have been systematically and rigorously evaluated, thus limiting the ability to understand the effect of these efforts on clinician and organizational behavior and subsequent client reach [
2]. Thought leaders in implementation science have recommended learning from natural experiments enacted by systems via observational research designs in order to produce generalizable knowledge to advance implementation science [
12]. Thus, rigorous evaluation of system-wide EBP implementation can produce valuable information to achieve this objective.
The majority of what is known about system-wide EBP implementation is largely descriptive in nature (i.e., if systems are implementing EBPs, how they support EBPs). One set of studies takes a broad perspective and reports on national trends across system EBP implementations. For example, a study reporting on a set of national surveys conducted with state mental health directors found increases in states offering EBPs for youth from 2001 to 2012 [
2]. Another survey study found that the majority of state mental health directors endorsed using financial incentives to promote EBP use in their system [
11]. Another set of studies takes a more granular perspective and reports on specific strategies used within one system such as the City of Philadelphia [
10], New York [
11], Hawaii, and Illinois [
13]. For example, Powell and colleagues describe how the City of Philadelphia Department of Behavioral Health and Intellectual Disability Services (DBHIDS), which oversees behavioral health services for over 600,000 Medicaid-enrolled consumers, began implementing EBPs in 2007 via “EBP initiatives” [
10,
14] and through the creation of the Evidence-based Practice and Innovation Center (EPIC) which included policy, fiscal, and operational changes to encourage EBP implementation [
10].
Although these perspectives have enriched the field’s understanding of whether systems are implementing EBPs and how EBPs are supported, there is a gap in the literature with regard to how system-wide efforts to implement EBPs are related to clinician practice over time. Only a few studies have attempted to evaluate the effect of system-wide EBP implementation. One national study within the Veterans Health Administration, a large system supporting EBP implementation, found that medical record documentation suggested that only 20% of veterans with post-traumatic stress disorder (PTSD) (total
n = 255,968) received at least one session of EBP for PTSD. Another study using administrative claims data found that there was an increased rate of EBP claims over time within the context of a fiscally mandated implementation effort in Los Angeles County [
15]. By leveraging existing data sources, these studies provide preliminary insights into how system-wide efforts to implement EBPs may be related to patterns in clinician and organizational behavior, but additional work is needed to understand the effect of such efforts.
Another focus of research inquiry includes investigating how system implementation of EBPs interacts with characteristics of the organizations nested within the system, such as organizational leadership, culture, and climate. This line of research can both elucidate potential mutable targets of implementation strategies in future implementation efforts and advance the science of implementation by providing empirical evidence for implementation science frameworks that posit the criticality of these constructs [
16]. Leading determinant frameworks [
17] such as the Consolidated Framework for Implementation Research [
18] and the Exploration, Preparation, Implementation, and Sustainment framework [
16] suggest the importance of the relationship between implementation and organizational characteristics, such as leadership (i.e., extent to which leaders are capable of guiding, directing, and supporting implementation) [
19], culture (i.e., shared norms, behavioral expectations, and values of an organization) [
20,
21], and climate (i.e., shared perceptions regarding the impact of the work environment on clinician well-being). A growing body of literature explores the relationship between these constructs and implementation (e.g., [
19,
22‐
27]). However, findings have been somewhat mixed [
19] and few studies have prospectively investigated the relationship between these factors and implementation—which would provide the most compelling evidence for potential mutable targets of implementation strategies, as well as build causal theory, a key imperative in implementation science [
19,
28,
29].
The current study builds on previous work by investigating how a centralized system effort to support implementation in the City of Philadelphia is related to clinicians’ EBP use and how organizational characteristics, specifically implementation leadership, implementation climate, and organizational culture, might moderate these effects [
16,
23,
30]. We measured clinicians’ self-reported use of psychotherapy techniques for youth in outpatient clinics over 5 years within the context of a system-wide effort to implement EBPs. We measured cognitive-behavioral therapy (CBT) techniques, which have evidence for their effectiveness for youth psychiatric disorders [
31] and comprised the majority of EBPs implemented by DBHIDS, and psychodynamic techniques, which are frequently used [
32] but have less evidence for youth psychiatric disorders [
31,
33‐
37]. We hypothesized that (a) clinician CBT use would increase over time, whereas psychodynamic technique use would remain static; (b) clinician participation in system-sponsored EBP initiatives would increase CBT use over time; and (c) baseline organizational variables would predict variability in clinician CBT use over time [
14].
Discussion
This study represents an opportunity to learn from a system encouraging EBP implementation [
10,
14] and can inform future policy and research. First, in a public system supporting EBP implementation, EBP use increased over time and clinicians who participated in system-sponsored training initiatives increased their EBP use even more. Second, proficient organizational culture modified the effect of system efforts to increase implementation, which elucidates a potential future target for implementation strategy trials [
28,
74]. While prior work has identified correlational associations between determinants like proficient organizational culture and outcomes, this study advances the field by prospectively elucidating the relationship between proficient culture and change over time. Despite our enthusiasm about these findings, it is important to note that they are preliminary given study limitations (i.e., self-reported measure of practice use and lack of a comparison system).
As expected, clinician use of EBPs modestly increased over the 5 years in which the system created a centralized infrastructure to de-silo EBP implementation [
10]. Although only half the clinicians in the study participated in system-sponsored EBP initiatives, there was a significant increase in clinicians’ use of EBPs system-wide during the study period. Potential explanations include that supervisors trained in these EBPs through EBP initiatives may have supported clinicians not formally trained in applying these techniques, that peer interactions may have increased clinician interest in these techniques, or that the changing system culture might reflect new organizational priorities. Clinicians participating in system-sponsored EBP training initiatives increased their use of EBPs twice as much as those not formally trained. Although these increases are promising, the effects were not large in magnitude and raise questions about clinical significance. In a large system serving over 30,000 children and families annually, an increase of 6% might have a population mental health impact, but further research is needed to understand the clinical impact of small, system-wide increases in use of EBPs. Future studies evaluating the impact of system-wide implementation must include client outcomes using hybrid effectiveness-implementation designs [
75] to ensure that the end goal of implementation efforts (i.e., client reach and outcomes) is achieved and that questions related to clinical significance and cost-effectiveness can be answered.
Consistent with the literature [
53,
56,
76‐
78], clinicians working in organizations with more proficient cultures at baseline exhibited greater increases in CBT use. This extends previous findings by prospectively linking proficiency culture to increased EBP use over time. Clinicians working in such organizations may be more motivated to improve their competence in up-to-date practices and have more opportunities to participate in EBP training because of leaders’ EBP prioritization. Although proficient culture is a more distal construct on the causal implementation pathway, these preliminary results suggest the importance of attending to general organizational factors in the implementation process. Future work should clarify if proficient culture is a more powerful target for implementation efforts versus training and ongoing support initiatives or if they result in a synergistic effect.
Clinicians’ self-reported non-evidence-based technique use remained stable. Given that there is little knowledge of the effect of delivering EBPs alongside potentially contraindicated approaches [
23], these findings point to the importance of attending to deimplementation [
79,
80]. Further, this finding provides evidence of discriminant validity and suggests that the relationships observed between initiative participation and proficient organizational culture are not due to spurious findings or common method error variance [
81].
Study methodological limitations include that this study was only conducted in one system and that CBT increases observed may be a national trend; that results may be influenced by cohort effects; that we relied on self-reported clinician use of techniques [
82,
83] [
84] rather than actual clinician behavior or patient outcomes; that the response rate was only 60%; that implementation strategies were not experimentally manipulated; and that these results may not generalize to smaller organizations and/or single clinician organizations given that we focused our sampling on organizations with larger programs. Study analytical limitations include that we conducted multiple tests of moderators given our exploratory aims, which increased the likelihood of a type I error; that the study may have been underpowered to detect effects of organizational moderators given the sample of 20 organizations at level 3; and that results included large confidence intervals on all effects that almost overlap with zero.
Acknowledgements
We are grateful for the support that the Department of Behavioral Health and Intellectual Disability Services has provided us to conduct this work within their system, for the Evidence Based Practice and Innovation (EPIC) group, and for the partnership provided to us by participating agencies, therapists, youth, and families.