Research, Data, and Evidence-Based Treatment Use in State Behavioral Health Systems, 2001–2012

Over the past several decades, a steady stream of articles, reports, and national calls to action have concluded that public behavioral health systems are characterized by ineffectiveness (1–3). Many of these reports called for increasing the availability of evidence-based treatments (EBTs) in public behavioral health systems and systematic use of data and research for continuous quality improvement (1–5). States are in a clear position to lead mental health service and system reform efforts, including investment in EBTs and application of research to improve outcomes (6–8). Thus states represent a logical focus for research on public system investments in research and evidence. Overall, however, empirical study of state support of EBT implementation and data use is scant (9).

One potential metric for gauging use of research evidence is the extent to which states implement EBTs. EBT implementation has been widely promoted as a partial solution to improving efficiency and increasing the likelihood of effectively addressing the behavioral health needs of children and adults (10,11). Because commitment to EBTs requires a commensurate commitment to practitioner and organizational capacity to deliver EBTs, state infrastructural supports (for example, policy and fiscal incentives, workforce development supports, data systems, and research centers or centers of excellence) represent a second potential metric (12–14).

In 1987, a not-for-profit research center for the National Association of State Mental Health Program Directors (NASMHPD) was established. In 1993, the NASMHPD Research Institute (NRI) initiated regular surveys of state representatives about the characteristics of state mental health authorities (SMHAs), including (starting in 2001) the nature and extent of their investment in EBTs and support of research, evaluation, and data use. To date, these data have not been used to examine state trends in EBT implementation, EBT implementation supports, or methods for promoting data and research use—all potential indicators of research-based decision making.

Using the NRI data, this study examined the degree to which state systems invest in EBTs and research-based supports and how such investments have changed from 2001 to 2012. This period represents the initiation of NRI tracking of specific EBTs as well as a time of increased awareness of—and federal initiatives to support—behavioral health EBTs (3,15–17). We addressed three research questions. First, how have the rates of use and penetration of EBTs by SMHAs changed from 2001 to 2012? Second, what kinds of support for EBTs are provided by SMHAs, and how has this changed over time? Third, what infrastructure is in place to support use of data by SMHAs, and how has this changed over time?

Methods

Data Source and Elements

NRI data originate from two publicly available sources. The State Profiles System (SPS) asks questions such as whether the SMHA conducted research on client outcomes, implemented a statewide client outcomes–monitoring system, engaged in initiatives to build awareness of EBTs, and other similar items. These questions referred to the state’s public mental health system in general and were not specifically tied to individual EBTs. Data elements from the Uniform Reporting System (URS) focus on counts of individuals receiving specific EBTs and estimates of youths with serious emotional disturbance and adults with serious mental illness. Prevalence estimates are overall estimates, not specific to the number of people eligible for each EBT. Additional details can be found in recent reports (18,19). Because data are all publicly available, formal review by an institutional review board was not required.

Sample

Information was provided by SMHA representatives in all 50 states, the District of Columbia, Puerto Rico, and the Virgin Islands. Response rates by states and territories were high, ranging from 87% (46 of 53) in 2001 to 98% (52 of 53) in 2005.

EBTs tracked.

EBTs tracked include three interventions for children with serious emotional disturbance— therapeutic foster care (20), multisystemic therapy (21), and functional family therapy (22)—and three treatments for adults with serious mental illness—supported housing (23), supported employment (24), and assertive community treatment (25). Because the NRI survey is intended to provide federal funding agencies with information on populations of specific interest (adults with serious mental illness and children with serious emotional disturbance), all of these interventions are intended for individuals with complex needs, and most are multimodal (that is, they include multiple strategies that address a range of factors that may influence individual and contextual needs).

Years examined.

Data were collected for most variables in 2001, 2002, 2004, 2007, 2009, 2010, and 2012. Data on numbers and rates of individuals served by EBTs, however, are available only from 2007 to 2012, when the URS was implemented as a state-level accountability mechanism by the Substance Abuse and Mental Health Services Administration.

Outliers and Missing Data

Continuous variables were examined for possible outliers or data entry errors, and 115 outliers (out of 1,543 total responses, 7%) were deleted or replaced. [More information on identification and handling of outliers is included in an online supplement to this article.]

Data analysis

Two-level multilevel models (MLMs) (26), with time and year nested within state, were used to examine change over time. Three models were run for each dependent variable to test linear, quadratic, and cubic time trends, and the best-fitting parsimonious model was selected for each. [More information on distributions used, determination of model fit, and software used is included in the online supplement.]

Results

State Use of Evidence and Activities to Promote EBTs

Table 1 presents data on the states that endorsed each specific practice concerning EBT implementation, training and workforce support, or data and research support. Time trend estimates are presented in Table 2. Significant linear time trends were found for five variables. Increasing percentages of states reported funding an external research center or institute, funding initiatives to increase EBT awareness, promoting the adoption of EBTs through provider contracts, providing financial incentives for offering EBTs, and conducting research or evaluation studies examining client change in functioning. The two most dramatic increases (based on the slope size) were in conducting research on change in client functioning and promoting adoption of EBTs through contracts.

Significant quadratic time trends were found for three variables. Significantly increasing and then decreasing percentages of states reported promoting EBT adoption through consensus building among stakeholders, through monitoring fidelity, and through specific budget requests. Reported use of these practices peaked in 2007–2009. A significant cubic trend was found for states conducting research and evaluation on client outcomes, which showed a sharp increase from 2001 to 2002 and then a slight decrease and slight increase from 2002 to 2009. No significant change over time was found for implementing a statewide outcomes-monitoring system, integrating client data sets with client data sets from other agencies, producing a directory of research or evaluation projects, operating a research center or institute, modifying information technology (IT) systems and data reports, conducting research on or evaluations of utilization rates, or conducting research on or evaluations of penetration rates.

States With EBT Services Available

Figure 1 shows the percentage of states that reported availability of the six EBTs tracked. On average across 2001–2012, EBTs serving adults were more commonly available. On average, 72% of states reported the availability of assertive community treatment, 70% reported supported employment, and 66% reported supported housing. For children and youths, on average, 54% of states reported that therapeutic foster care was available, 39% reported multisystemic therapy, and 27% reported functional family therapy.

Model fitting confirmed that piecewise linear time trends fit the data better than exponential time trends. Table 3 presents the results of individual piecewise MLMs. For all EBTs with valid data from 2001 to 2012, there were significant increases in the proportion of states using therapeutic foster care, multisystemic therapy, supported employment, and assertive community treatment from 2001 to 2005 and then no significant increases or decreases from 2007 to 2012. Data on functional family therapy and supported housing were not collected until 2005; tests for these practices found no significant increases or decreases from 2005 to 2012.

Number of Clients Served by Specific EBTs

Because of positive skew (a few states reporting large numbers served), we examined state medians for clients served and used data only from states that reported any availability of the salient EBT. On average across 2007–2012, a median of 1,029 clients per year per state used supported housing, 950 used assertive community treatment, and 669 used supported employment. For EBTs serving children and youths, a median of 371 clients used functional family therapy, 279 used therapeutic foster care, and 230 used multisystemic therapy.

Figure 2 shows the median number served per state each year, and Table 2 provides MLM estimates for best-fitting time trends. Results indicate a significant but small linear increase in the number of multisystemic therapy clients, a significant quadratic change in therapeutic foster care clients (an increase followed by a decrease), and a significant cubic change in the number of supported housing clients (a decrease followed by a flattening). No time trends were found for functional family therapy, supported employment, or assertive community treatment.

Rates of Clients Served

Rates of clients served by EBTs were calculated by dividing the reported numbers of clients served by the reported number of adults with serious mental illness or children and youths with serious emotional disturbance in the state. Averaging across 2007–2012, the median rates of clients served were, in descending order, 3% for supported housing, 2% for functional family therapy, 2% for assertive community treatment, 2% for supported employment, 1% for therapeutic foster care, and 1% for multisystemic therapy.

Table 2 presents the MLM estimates for best-fitting time trends for rates of clients served. Results indicated no significant changes for supported employment or assertive community treatment and significant cubic changes for therapeutic foster care, multisystemic therapy, functional family therapy, and supported housing. Therapeutic foster care and supported housing both showed flat trends followed by a substantial decrease and then another flat trend. Multisystemic therapy showed an increase from 2007 to 2009, followed by a decrease and a brief increase. Functional family therapy showed a flat trend, a sharp increase, and then a slow downward trend from 2009 to 2012.

Discussion

Although there is evidence that many states have invested in infrastructure supports to encourage and support EBT implementation, the impact on availability and reach of EBTs has been relatively small. In 2012, between 65% and 80% of states reported using the EBTs for adults examined in this study. However, the median numbers of clients served by states were only about 1,000 for each EBT, and penetration rates ranged from only 2% to 3%. Uptake of child EBTs was lower: only 25% to 50% of states reported use of the interventions studied, and median numbers of clients served were only 250 to 400, reaching only 1%−3% of youths with serious emotional disturbance. It is not possible from these data to explain the reasons for the discrepancy in use of adult- versus child-focused EBTs; however, the discrepancy could result from the long-standing trend in state behavioral health systems to invest more heavily in services for adults (27).

The results also pointed to large gaps between need and available EBTs for both adults and children. Some of this observed gap may be the result of study limitations. Because the NRI survey focuses on a small number of multimodal interventions for individuals with complex needs, data for a range of well-known interventions for individuals with less complex or intensive behavioral health needs (for example, cognitive-behavioral therapies and behavioral parenting interventions) were not available. Thus the actual number of EBTs being used across states is likely higher. In addition, there have been significant changes since 2001 in states’ authority; for example, the increasing penetration of managed care has led to less direct state control over services (18,28). Thus states have less authority to oversee EBT initiatives or may be unaware of EBTs that are being used through managed care.

Nonetheless, the scant uptake of EBTs found in this study is consistent with other studies of specific EBT implementation, which demonstrate significant challenges in getting research-based services installed in real-world systems (29–32). Results highlight the fact that the population entrusted to public mental health care is not deriving benefits from receipt of evidence-based therapeutic services. Although each of the interventions examined has been proposed as a strategy to meet the behavioral health needs of individuals with the most complex presentations, and all are tailored to clients served by specific public systems (for example, foster care for therapeutic foster care and juvenile justice–involved youths for multisystemic therapy), only a very small portion of individuals actually receive one of the EBTs.

Perhaps more worrisome, adoption by states, numbers served, and penetration rates were found to be flat or declining. Even if these interventions are merely considered indicators or proxies of EBT uptake in states, trends examined here suggest that after a burst of initial adoption and expansion in the early 2000s, EBT uptake has leveled off, if not diminished. This study also found similar “inverted U”-shaped trends in state EBT adoption drivers, including consensus building among stakeholders, state-led fidelity monitoring, and specific budget requests for EBTs.

Given that research-based strategies require investment in resources, both types of downward trends are not surprising given the magnitude of budget shortages that hit most states from 2007 to 2012. In fiscal year 2013, 13 SMHAs still had expenditures below 2007 levels (33), and these recent cuts occurred on top of historical and long-standing budget shortfalls that have consistently affected SMHAs (27).

Regardless, the trend in reduced state support for EBTs and their implementation is troubling, given that resources beyond standard fee-for-service payments are typically necessary for providers to implement EBTs well (34–36). Moreover, the goal of reducing overall public expenditures “downstream” (for example, corrections services, emergency room care, and hospitalization) (37) is theoretically linked to maintaining investment in EBTs that reduce the need for these expensive services. Thus the trend to limit EBT investment may ultimately result in increased state spending. In future analyses, we plan to use NRI data along with other data to examine the relationship between fluctuations in state fiscal investments and use of research-based strategies.

Some EBT implementation support strategies increased across the study period. Building EBTs into provider contracts showed the single greatest increase among the variables examined, from 43% of states in 2001–2004 to 70% in 2009–2012. This phenomenon was also found in a study by Finnerty and colleagues (38), which found that in 2007–2009 the majority of states opted to promote substance use disorder–related EBTs by encouraging providers to deliver these practices with contract funds.

States also reported increased use over the study period of financial incentives for provider EBT use. This may be an encouraging trend, because it has been highlighted as a proactive fiscal policy that encourages EBT adoption (38–40). However, the number of states reporting such incentives in 2012 was only 35%, barely more than the 31% that reported using the strategy in 2004.

Despite substantial improvements in IT since the turn of the millennium, the percentage of states implementing a statewide outcomes-monitoring system, integrating client data sets across agencies, producing directories of research or evaluation projects, modifying IT systems to support EBTs, and conducting research and evaluation did not change from 2001 to 2012. In 2012, only half of states reported having integrated data systems or a client outcomes-monitoring system. This is very troubling given broad consensus that true accountability is achieved not merely through investment in EBTs (41) but also through investment in data systems, data integration, and outcomes monitoring that promotes feedback, understanding of system functioning, and continuous improvement (2,42,43).

Unfortunately, however, to preserve services, SMHAs reported making their cuts in “administrative” expenses, such as data and research use, IT systems, outcomes monitoring, evaluation, and fidelity monitoring. Among evaluation and data use variables, only “funding an external research center or institute” showed an increase over time, from 13% to 31%. Although this could be interpreted as a positive trend, it is mirrored by a decrease in SMHAs’ internal operation of research centers, from 17% to 8%.

The study had some limitations. SMHAs are not the only systems that provide EBTs in a state. Vocational rehabilitation agencies may provide supported employment, and child welfare or juvenile justice agencies may support therapeutic foster care or multisystemic therapy. When responding to NRI-administered surveys, SMHA respondents may not have been fully informed about all state behavioral health initiatives or about localized efforts or pilot projects. In addition, estimates of the penetration rates for EBTs were based on general estimates provided by SMHA respondents of the overall number of youths with serious emotional disturbance or adults with serious mental illness, which may not accurately reflect actual rates of individuals eligible for these EBTs.

Another limitation is that because of the NRI’s funding mandate to focus on services funded by federal block grants, EBT data focus only on services for adults with serious mental illness or children with serious emotional disturbance, not on populations with other conditions or less intensive needs. The extent to which these findings are applicable to EBT implementation for other conditions, less intensive needs, or early intervention and prevention is unknown. In addition, although most of the EBTs studied are well defined and manualized, therapeutic foster care is a broader service type, which may have affected the accuracy of the findings and interpretation of utilization trends. Similarly, other important drivers and indicators of research and EBT use were not surveyed and thus could not be examined, including workforce support and training, the role of changing state leadership, differential costs associated with implementing EBTs, and shifts over time to managed care. These are all important areas for future research. Finally, the study aimed to examine national trends, and thus we present aggregate results across all states. Although beyond the scope of this study, future analyses should examine patterns of individual state trends and predictors of these variations.

Conclusions

State investment in EBT implementation infrastructure and use of data are critical and are only likely to grow in importance as health care reforms call for more effective services and greater quality and accountability. Results of this study suggest, however, that SMHA investment in EBTs and implementation infrastructure is flat or declining in many areas. Consistent, reliable, and valid measurement of these constructs will assist states in ensuring that their systems of services are capable of meeting the needs of the populations for which they are responsible. Such data may also serve as a basis for renewed investment and a metric for reform efforts.