The role of the outer setting in implementation: associations between state demographic, fiscal, and policy factors and use of evidence-based treatments in mental healthcare

In previous research [19], we used data from regular surveys of US state mental health authority (SMHA) program directors (individuals charged with overseeing mental health policy, fiscal, workforce, and other initiatives at a statewide level), to examine state trends in EBT adoption and penetration rates and EBT implementation supports by states from 2001 to 2012. While results found moderate levels of EBT adoption for three child-focused EBTs (25–50% of the states reported adopting each of these) and three adult-focused EBTs (65%–75% of the states reported adopting each of these), these services did not reach the vast majority of those in need. There were very low penetration rates (0.75–2.5% of those in need) as well as trends indicating decreasing levels of fiscal support for EBT. States varied greatly, however, across reported rates of EBT funding, supportive policies, and EBT adoption.

Health policy scholars have proposed an array of factors in the fiscal and policy context (i.e., outer setting) that may influence uptake of EBT [9, 20]. Outer setting variables are notoriously important for implementation, but simultaneously difficult to evaluate and influence. For instance, the Society for Implementation Research Collaboration’s (SIRC) comprehensive Instrument Review Project identified only four instruments that capture outer setting processes [21]. Efforts to operationalize these variables and their taxonomies have also been scant. For example, initial versions of the widely applied Consolidated Framework for Implementation Research [9] characterized “external policies and incentives” as a “broad construct” within the outer setting domain, but dedicated only a single sentence to its definition.

Other, more recent, theories, such as the Exploration, Preparation, Implementation, and Sustainment (EPIS) framework [8], more explicitly identify factors within the outer context (e.g., leadership, funding/contracting, inter-organizational networks) that are proposed to be influential across phases of implementation, as well as “bridging factors” that may link the outer context to influential factors at the organizational (or inner setting) level. Other theories characterize factors in the outer setting as including both “unmodifiable” (e.g., economic, political) versus “modifiable” (e.g., funding, regulations, networks) factors [22], potentially providing guidance around how these theories might be applied to action. To date, however, there has been little robust application of such theoretical drivers to implementation research or “on the ground” use [18].

Nonetheless, a small number of previous studies (mostly that predate the publication of the CFIR and/or the first volume of Implementation Science) have attempted to identify and characterize the types of strategies used by states to promote EBT [23]. Other scholars have used an expert consensus process to identify a comprehensive array of potential outer setting implementation strategies [24]. No previous empirical studies, however, have asked what characteristics of states or state behavioral health systems are associated with EBT adoption or investment. Such a study could inform the emerging field of EBT dissemination and implementation policy [16] as well as provide insight into how best to focus efforts to promote EBT in public systems.

This article presents the results of a study to help fill this knowledge gap, using longitudinal survey data of SMHA directors from all 50 US states and territories compiled by the NASMHPD Research Institute (NRI) and other data sources to explore the strengths of relationships between state factors and state activities and the dependent variables of EBT investments and adoption. As a first step, we developed a conceptual framework for the study (Fig. 1) that organized data relevant to state efforts to promote EBT adoption (“modifiable” external factors, mostly from NRI surveys) as well as state data from other sources on factors proposed by previous theory and research [9, 20, 22] to be influential (“unmodifiable” factors).

As depicted in Fig. 1, we hypothesized that certain unmodifiable state factors would be associated with the degree to which policy-makers modify the implementation ecology for EBT (e.g., through fiscal investments and supportive policies). We also hypothesized that both modifiable state factors (e.g., state fiscal and policy supports) and unmodifiable factors (e.g., state spending patterns and party in political control) would be associated with degree of EBT adoption. Importantly, because of the lack of previous research on this topic, and our use of secondary data, we did not hypothesize specific variables within these major domains that would emerge as significant predictors.

The study sought to answer two broad research questions that extend from this conceptual framework. First, what unmodifiable state factors (e.g., economic, geographic, political, structural) are associated with state-level EBT financial investment and supportive policies? Second, what unmodifiable (e.g., economic, geographic, political, structural) and modifiable (i.e., SMHA structures, social processes, EBT-related investments and policies) factors are associated with actual state-level adoption of six leading behavioral health EBTs (three focused on adults and three focused on children)?

The current study was conducted in the USA. Data originated from five publically available sources. The count of adults and children in each state and the per capita income of each state come from the US Census Bureau [25] and US Department of Commerce [26], respectively. Each state’s status with respect to Medicaid expansion under the Affordable Care Act was obtained from publically available data from the Kaiser Family Foundation [27]. Each state’s budget surplus or deficit as well as controlling political party come from Harvard University’s public online “Dataverse” [28]. For controlling political party, we calculated a “controlling party index” based on which party controlled the two houses of the legislature and the executive (Governor) branch (Republican control = 0; Democratic = 1). Thus, this variable could range from 0 (all three branches Republican) to 3 (all three branches Democratic).

All other variables come from the State Profiles System (SPS) and Uniform Reporting System (URS), datasets made available by NRI. The SPS is administered to representatives of SMHAs and asks a wide range of questions about the structure and functioning of the SMHA. The URS is a reporting system as part of SAMHSA’s Community Mental Health Block Grant (CMHBG). It compiles numbers and characteristics of clients served by the each state. It also asks about the use of six EBTs identified in the early 2000s by SAMHSA as being particularly relevant to the populations of focus of the CMHB—adults with serious mental illness (SMI) and children with serious emotional and behavioral disorders (SEBD). Additional details can be found from recent reports [13, 15]. We used 34 items from these NRI datasets: 6 related to the provision of specific EBTs, 11 related to “state characteristics,” 12 about policy supports to promote EBT adoption, and 5 about fiscal supports to EBT adoption. A complete list of items included in analyses can be found in Table 1. Below, we provide more details on variables from the SPS and URS.

SPS and URS data were collected for most variables in 2002, 2004, 2007, 2009, 2010, and 2012. Data on the provision of EBTs, however, were only available from 2007 to 2012.

Data cover all 50 states and the District of Columbia. SPS data were available for most states in most years; response rates vary from a low of 84% (43 of 51) in 2005 to a high of 98% in 2009 and 2010 (50 out of 51). The average response rate across all 7 years was 92%. Data from the URS were available from every state from 2007 to 2012. Data from all other sources were complete.

Descriptive statistics were run using STATA 13.1 [38]. Two-level hierarchical models were conducted using HLM 7.0 [39] to account for the nested structure of the data, where time and year were nested within state. We ran individual models using each individual predictor variable and linear time, as our sample size did not provide the power for multivariable model building or testing curvilinear time trends. Normal distributions were used for continuous outcome variables, and Bernoulli distributions were used for dichotomous outcome variables (availability or unavailability of individual EBTs within the state). Models were run using full maximum likelihood (FML) estimation due to unbalanced data timepoints within states. Randomly varying intercepts were permitted but slopes were fixed due to failure to achieve model convergence as a result of low power (only 50 states) and small variance (there was limited within-state change over time for most variables). Essentially, this resulted in analyses similar to repeated-measures ANOVAs with random effects for state intercept but not slope [40], using FML estimation to permit the inclusion of states with missing data points.

All models were adjusted for time centered at 2002, except those for the outcome EBT availability index. These models were adjusted for time centered at 2007 because data on EBT availability was limited to the years 2007, 2009, 2010, and 2012. We report the most basic models that only include the predictor and time variables. Because only 4 years of data were available for models examining EBT availability, the total possible sample size was n = 200; actual sample size for most models when including missing data points ranged from n = 135–152; budget surplus and per capita expenditures were only asked in three of those years (n = 99). For all other models, with six possible years of data, total possible sample size was n = 300, and actual sample sizes ranged from n = 258–270.

Although the aims, methods, and data sources of this study are unique, many of its findings align with previous research. For example, Purtle and colleagues [7] found that Democratic state legislators were significantly more likely than Republicans to identify University-based research as a credible source of information. Similarly, a national scan by the Pew Charitable Trusts found that leading states in promoting evidence-based policy-making were more likely to be under Democratic than Republican control [41].

The finding that higher per capita income is associated with fiscal investments in EBT but that state budget strength was inversely related to such investments may seem counterintuitive on the surface; however, economic policy research has generally found that state-level infrastructure spending (which could extend to EBT implementation infrastructure) (1) tends to be unrelated to state per capita income [42‐44]; (2) is more common among Democratically controlled states (also related to EBT investment) [44, 45]; and, not surprisingly, (3) is associated with higher per capita state debt [46]. Although discussion of these dynamics is beyond the scope of this paper, such findings about the influence of “unmodifiable state factors” on EBT implementation investment provide a starting point for discussion.

Findings also reinforce the salience of propositions embedded within certain extant implementation frameworks. For example, the EPIS framework [8] identifies “bridging factors” such as community-academic partnerships and use of purveyors and intermediary organizations as potentially important strategies for linking the outer setting to organizational (inner setting) efforts to implement EBT. As discussed below, these factors emerged in the current study as potentially critical predictors of EBT uptake. Given the increasing use of frameworks such as EPIS [18] to actively facilitate implementation efforts internationally, evidence of such concordance is encouraging.

The potential importance of the current findings for EBT and public health—combined with the lack of previous research on outer setting drivers of EBT investment and uptake—points to the critical need for additional, more rigorous, implementation policy research. The research and funding gap is stark: while a recent review found 366 published implementation science measures, only four of these focused on the outer setting (compared to 98 for the inner setting and 90 for individuals) [21]. Meanwhile, a recent review found that policy research constituted only 10.5% of all of National Institute of Health (NIH)-funded dissemination and implementation research, which, in turn, represented less than 0.1% of all research funding [16]. The current study also highlights the likelihood that outer setting implementation determinants and strategies may differentially affect children’s services vs. adult services and thus a need to attend to these differences in future research.

This is not to say that efforts have not been undertaken. Powell and colleagues, for example, used an expert consensus process to identify 73 potential EBT implementation strategies for use in research and practice. Upwards of 32 of these strategies primarily target the outer setting [24]. Many are at the policy level and provide more complete and operationalized descriptions of implementation support activity than is included in the NRI surveys.

Efforts to extend efforts such as Powell and colleagues’ [24] to new or existing data collection efforts by federal agencies, such as the one that provided the data for the current study, would greatly improve our understanding of the outer setting of the implementation ecology. As one step, state implementation variables such as those incorporated into the NRI surveys could be cross-walked to the strategies identified by Powell et al. and/or better aligned with implementation science in other ways. In general, a more systematic approach to continually assessing state-level factors and strategies at the outer setting—along with sentinel indicators of EBT uptake and client outcomes—could be a highly productive way to expand the research base and shape more effective policy.

A clear limitation of the current study is the extremely limited nature of the measures of EBT implementation and the lack of any measure of impact on child outcomes. Because of the lack of consistent data across states, we were forced to rely on self-report of a small number of SMHA representatives for the majority of our independent variables as well as the dependent variable. SMHA respondents may not have been fully informed about the array of state behavioral health initiatives when responding to NRI-administered surveys. Moreover, their perceptions of the presence or absence of supports for EBT may have been colored by their perceptions of the availability of the specific EBTs (and vice-versa), or other factors, such as their time in office, role in implementation activities, clinical experience, or political affiliation. And, as with all such studies, there is a chance that the associations found in our data were upwardly biased due to response bias of participants being consistently positive or negative across items.

In the future, federal efforts to understand EBT implementation would benefit from more robust data collection methods, such as via mandates around consistent coding of EBT utilization in Medicaid and/or insurance claims. Although challenging to adopt nationally, some states have adopted such procedures [58], which could be encouraged nationally. At the very least, entities such as NRI could enact small steps toward greater rigor in its surveying methods, such as collecting information on the characteristics of reporting officials, to include in multi-level models.

Another limitation is that SMHAs are not the only systems that provide EBTs in a state. Vocational rehabilitation, child welfare, and/or juvenile justice agencies may support EBTs such as SE, TFC, MST, and others. Relatedly, we only examined relationships at a state level, when fiscal, political, policy, and other factors—as well as EBP adoption—are all likely to vary greatly within states, across jurisdictions such as counties and metropolitan areas. Another limitation is that, due to the NRI’s funding mandate to focus on services funded by federal block grants, EBT data focus only on a limited number of service types for adults with serious mental illness or children with serious emotional disturbance, not populations with other conditions or less intensive needs. The extent to which these findings are applicable to EBT implementation for other conditions, less intensive needs, or early intervention and prevention is unknown.

Given the current study capitalized on the unique federalist policy environment of the USA, it is also unclear the degree to which the current findings might apply to implementation efforts outside the USA. Nonetheless, as described above, findings from this study were found to align with implementation frameworks such as EPIS (particularly related to “bridging activities” from the outer to inner contexts that may be needed to support EBT implementation), which has been applied to implementation efforts across a range of international contexts [8].

This study is also limited in that it was exploratory and featured a large number of analyses, so is prone to familywise error. Finally, for most models reported in the tables, terms were fixed due to limited power.