Background
Widespread evidence supports integrating behavioral health and substance abuse treatments for those who are dually diagnosed with both mental illnesses and substance abuse disorders [
1]. However, several large-scale initiatives have demonstrated that it is challenging to provide integrated care in typical treatment settings for those who are homeless [
2‐
5]. For example, a multi-site study that integrated dual disorders treatment and Housing First found that while overall scores for Housing First fidelity were high, scores of the item specifically addressing “Integrated, Stage-wise Substance Use Treatment” were rated 3 or lower on a 5-point scale [
5]. In the Department of Veterans Affairs (VA), up to 80% of the approximately 48,000 Veterans [
6] who are homeless suffer from mental health and/or substance use disorders, threatening their housing stability and leading to higher rates of relapse, treatment dropout, poor community integration, and utilization of costly emergency and inpatient services [
7]. This paper presents a cluster-randomized trial of a specific implementation strategy aimed at supporting the use of a complex integrated dual disorders model for Veterans who have experienced homelessness.
Experts in dual diagnosis research have carried out several initiatives attempting to document the degree to which integrated treatments were adopted and implemented. For example, the National Evidence-Based Practice (EBP) Implementation Project was a non-experimental effort to use in-person consultant/trainers for 2 years to help 53 public-sector community mental health agencies adopt two of five EBPs for those with serious mental illnesses (SMI), one of which was an integrated dual disorder model. Although the sites did improve their fidelity, only 18% of the sites who chose the integrated treatment model had high fidelity [
2,
3]. Other large-scale initiatives that employed similar implementation strategies found similar results [
4,
5]. Building upon these contributions, more research is needed to specifically test implementation strategies in settings that serve those who are not only dually diagnosed but also homeless.
An ideal setting for such research is the VA program called HUD-VASH (Housing and Urban Development—Veteran Affairs Supportive Housing). Veterans at all VA medical centers who are homeless can receive subsidized housing from HUD and VA case management. In 2012, HUD-VASH adopted the Housing First philosophy, which states that individuals do
not need to demonstrate complete sobriety in order to receive housing and case management services. In addition, HUD-VASH recently accepted a significant number of new Veterans to help support the goal of ending homelessness among Veterans by 2015 [
7], straining available resources, especially staff. Both changes increased the number of Veterans in HUD-VASH with a need for integrated treatment for both substance abuse and mental health diagnoses.
Although many evidence-based integrated treatment models are available, one in particular—Maintaining Independence and Sobriety through Systems Integration, Outreach, and Networking—Veterans Edition (MISSION-Vet)—was developed specifically for homeless or formerly homeless Veterans [
8,
9]. MISSION-Vet is a manualized, integrated, co-occurring disorder treatment model grounded in the Health Belief Model [
10]. In MISSION-Vet, a Veteran, case manager, and “peer specialist” work together for about 2.5 h a week. Peer specialists are individuals who have recovered from their own mental health and substance issues and are trained to provide support to others with the same difficulties [
11]. Wide-scale implementation of MISSION-Vet in HUD-VASH has not occurred, despite a strong evidence base, the fact that both share a Housing First treatment philosophy [
12], and free, web-based training and manuals. While implementation models such as the Consolidated Framework for Implementation Research (CFIR) [
13] show that research efficacy, concrete tools, and compatibility with host sites are important factors that facilitate implementation, these factors alone are usually not sufficient to promote adoption by settings like HUD-VASH [
14]. Because systems frequently do not adopt new practices even when they are known to improve outcomes [
15], innovative strategies are needed at both individual and organizational levels to encourage adoption.
To facilitate the adoption of MISSION-Vet within HUD-VASH, we employed and then tested the Getting To Outcomes (GTO) approach [
16], modified for use within the VA [
17]. GTO is both an implementation
model—specifying steps practitioners should take when carrying out an EBP—and an implementation
strategy, providing ongoing implementation training, technical assistance, and data feedback to improve practitioners’ capacity to complete those steps [
18]. GTO has been found to improve the capacity of individual drug and teen pregnancy prevention practitioners and the performance and fidelity of prevention programs in both quasi-experimental [
18] and randomized controlled trials [
19‐
22]. However, most of those studies involved non-evidence-based programs. This study is the first to empirically test a version of GTO in the VA with a rigorous design and also the first instance GTO was used to support an EBP in a clinical setting. We received funding from the VA Quality Enhancement Research Initiative to compare MISSION-Vet implementation with and without GTO support in three large VA HUD-VASH sites.
Results
MISSION-Vet services by RE-AIM domains
Adoption
As shown in Table
3, no case managers in the Implementation as Usual (IU) group at any study site adopted MISSION-Vet while 68% (25 of 37) of case managers in the GTO group adopted MISSION-Vet, a difference that was statistically significant. Likewise, there was a significant difference between the GTO and IU group with respect to the adoption of MISSION-Vet within teams 1 and 2, but not team 3. Teams 1 and 2 had similar rates of case manager Adoption (100, 92%), much more than team 3 (39%). Team 1, which had the highest Adoption among the GTO group (100%), also had the smallest number of consented case managers in the GTO group (
n = 6). Team 3, which had the lowest Adoption rate among GTO groups (39%), had the largest number of consented case managers (
n = 18).
Table 3
Adoption and reach of MISSION-Vet by study group
Adoption |
% CM tried MISSION (n) | 100* (6) | 0 (8) | 92* (13) | 0 (8) | 39 (18) | 0 (6) | 68* (37) | 0 (22) |
Reach |
Eligible Veteransa, n
| 236 | 151 | 271 | 210 | 489 | 206 | 996 | 567 |
Received any MISSION-Vet sessions, % (n) | 11.4* (27) | 0 | 10.0* (27) | 0 | 3.3 (16) | 0 | 7.0* (70) | 0 |
Received 10% of MISSION-Vet sessionsb, % (n) | 7.2* (17) | 0 | 7.4* (20) | 0 | 2.9 (14) | 0 | 5.1* (51) | 0 |
Received 25% of MISSION-Vet sessionsb, % (n) | 5.5* (13) | 0 | 4.8* (13) | 0 | 2.2 (11) | 0 | 3.7* (37) | 0 |
Received 50% of MISSION-Vet sessionsb, % (n) | 2.5 (6) | 0 | 1.5 (4) | 0 | 1.4 (7) | 0 | 1.7* (17) | 0 |
Reach
No Veterans with case managers in the IU group (
n = 567) at any study site received any MISSION-Vet sessions. Seven percent of Veterans with case managers in the GTO group (
n = 996) received at least one MISSION session. Only 1.7% of Veterans with case managers in the GTO group received at least half of the available 24 structured DRT or peer specialist MISSION-Vet sessions. The differences between the GTO and IU groups with respect to the Reach outcomes were all statistically significant when collapsing across all teams, although not all within-team comparisons were significant. Again, teams 1 and 2 had similar Reach rates, double or more the rates for team 3 across multiple levels of MISSION-Vet received (any, 10%, 25%). As one would expect, the teams with higher Adoption, also had higher Reach (see Table
3). For all teams, the percent of Reach declined as the RE-AIM criterion increased (any, 10%, 25%, 50%).
Implementation
Across all teams, 73% of Veterans receiving MISSION-Vet were given the Consumer workbook. Veterans receiving MISSION-Vet received on average 4.2 DRT sessions, 3.3 peer specialist sessions, 1.5 self-guided exercises, and referrals to 1.5 of the categories of referral services. Teams 1 and 3 had similar patterns of case manager implementation (see Table
4). Team 2 provided less case manager services to MISSION-Vet Veterans but more peer specialist services. No team’s Veterans received more than half the sessions of any type called for by MISSION-Vet. Team 1’s Veterans received about a third of the DRT sessions, and a quarter of the self-guided exercises and peer specialist sessions. Team 2’s Veterans received a fifth of the DRT sessions, five percent of the self-guided exercises, and a third of the peer specialist sessions. Team 3’s Veterans received the most: nearly half of the DRT sessions, about a third of the self-guided exercises, and about a quarter of the peer specialist sessions.
Table 4
Implementation of MISSION-Vet
Percent received workbook | 79 | – | 63 | – | 78 | – | 73 |
DRT sessions done |
Number | 4.5 | 0–11 | 2.6 | 0–10 | 6.0 | 0–13 | 4.2 |
Percenta
| 34 | 0–85 | 20 | 0–77 | 46 | 0–100 | 32 |
Self-guided exercises done |
Number | 1.7 | 0–6 | 0.4 | 0–3 | 2.7 | 0–7 | 1.5 |
Percentb
| 24 | 0–86 | 5 | 0–43 | 38 | 0–100 | 21 |
Peer specialist sessions done |
Number | 2.8 | 0–11 | 4.0 | 0–10 | 2.9 | 0–8 | 3.3 |
Percentc
| 25 | 0–100 | 36 | 0–91 | 27 | 0–73 | 30 |
Number of referrals | 1.6 | 0–6 | 1.1 | 0–4 | 2.1 | 0–6 | 1.5 |
Ratings of CFIR implementation factors
In this study, the domain of Intervention Characteristics addresses characteristics of the MISSION-Vet model (see Table
5). All teams perceived that the decision to use the MISSION-Vet model was reached externally to HUD-VASH (Intervention source), which is typically an implementation barrier. Teams 1 and 3 found MISSION-Vet to be complicated to implement while team 2 was neutral on Complexity. Across all teams, MISSION-Vet was perceived as highly adaptable, such that case managers could implement different aspects of MISSION-Vet in different ways for different Veterans, and this was seen as a major facilitator because of the flexibility it afforded. Further, all teams rated Relative Advantage as a facilitator, as case managers reported that MISSION-Vet offered concrete, useful ideas for how to treat those with dual diagnoses that were good additions to their current strategies.
Table 5
CFIR ratings of study teams
Intervention characteristics |
Intervention source | −1 | −1 | −1 |
Evidence strength | 0 | 0 | 0 |
Relative advantage | +1 | +1 | +1 |
Adaptability | +2 | +2 | +2 |
Complexity | −1 | 0 | −1 |
Design quality | 0 | −1 | 0 |
Inner setting |
Networks and communications | −2 | +1 | −1 |
Compatibility | −1 | −2 | −1 |
Relative priority | −2 | −2 | −2 |
Org. incentives and rewards | −1 | 0 | −1 |
Goals and feedback | 0 | 0 | −1 |
Leadership engagement | −1 | +1 | −1 |
Available resources | −1 | −1 | +1 |
Access to knowledge and info | −1 | 0 | +1 |
Outer setting |
Patient needs and resources | +1 | +1 | +1 |
Cosmopolitan | 0 | 0 | +1 |
Process |
Planning | +1 | +1 | +1 |
Engaging key stakeholders | +1 | 0 | +1 |
Engaging Veterans | −1 | −2 | −1 |
The CFIR Inner Setting domain refers to the setting where the intervention took place, in this study, the HUD-VASH teams. At this level, the low relative priority given to implementing MISSION-Vet by the teams’ leadership, in comparison to the high pressure to house Veterans quickly, was seen as a major barrier to implementation across all three study teams. Further, many case managers found MISSION-Vet was incompatible with their work on the HUD-VASH team, as urgent pressure to house Veterans, and large caseload sizes left case managers with little time for the intensity of providing MISSION-Vet services. Case managers reported no positive influence from Incentives or Rewards by leadership for using MISSION-Vet (negative at two teams, neutral at another) or by any feedback provided about use of MISSION-Vet (neutral at two teams, negative at another). Team communications were also challenging at two of the three sites, especially so for the one site that hired peers specialists via contract (not as VA employees), which added an additional level of communication complexity.
There was only sufficient data available to provide ratings for two constructs in the Outer Setting domain. Understanding of patient needs and resources across all three teams were a positive factor in implementation. However, staff at team 2 made repeated statements indicating their belief that the MISSION-Vet intervention was inappropriate for the population they served because their Veterans were struggling to meet basic needs (housing, food, and clothing) and were not ready to address issues related to their dual diagnosis. This perception was seen as negatively affecting the implementation of MISSION-Vet on this team.
In the Process domain, planning for implementation—operationalized by the GTO implementation strategy—was seen as a strength—as GTO helped organize the teams, kept them on track, and provided a forum to troubleshoot implementation. All teams reported significant challenges engaging Veterans into MISSION-Vet treatment, which is reflected in the negative rating on the Engaging Veterans factor.
Discussion
We presented data assessing the impact of an implementation strategy (GTO) on the use of a clinical treatment (MISSION-Vet) in three domains specified by the implementation heuristic, RE-AIM (Adoption, Reach, Implementation). To provide additional context, we also assessed various factors that could hinder or facilitate implementation according to CFIR. As hypothesized, the GTO group had greater rates of Adoption, Reach, and Implementation than the IU group. Regarding Adoption, most case managers assigned to GTO attempted MISSION-Vet, while no case managers in the IU group did. Given that case managers in the IU group had zero Adoption, they also had zero Reach and Implementation. These findings are consistent with a great deal of research that shows that passive approaches to implementation do not usually result in uptake of new practices [
42]. Although case managers in the GTO condition attempted to deliver MISSION-Vet more than IU case managers, they only implemented MISSION-Vet with a small number of Veterans (Reach) and delivered a small amount of MISSION-Vet services compared to what the EBP typically requires (Implementation). The findings regarding the low level of implementation are different from past GTO studies [
22] but similar to past studies attempting to facilitate the delivery of integrated dual diagnosis treatment, as described above [
2‐
5]. Compared to MISSION-Vet, however, the EBP in GTO studies have been much less complicated to implement—e.g., an eight-session teen pregnancy prevention program [
22]. As we show below, the CFIR ratings highlight the numerous challenges on the teams that made the use of MISSION-Vet more difficult and some factors which aided implementation.
Intervention characteristics
Although most case managers appreciated the guidance and materials of MISSION-Vet, some case managers found the treatment complicated and confusing. In particular, some stated that it was challenging to know when and how to integrate the various components of MISSION-Vet into the treatment for each Veteran. MISSION-Vet was adapted quite often, which facilitated case managers trying it out, but appeared to have made it “acceptable” to not implement it in full.
Inner setting
There were several factors associated with the HUD-VASH setting that made MISSION-Vet implementation more challenging. First, although approval for introducing MISSION-Vet was secured from both medical center and HUD-VASH levels, there was little leadership involvement from either level once the project began. We primarily worked with frontline staff to implement MISSION-Vet. Without higher level leadership buy-in, there was little accountability when implementation lagged or reward for those who did implement. Through GTO, service delivery goals were established, but leadership did not track performance compared to those goals. The teams recognized that many Veterans needed more comprehensive dual diagnosis treatment; however, there were mixed feelings about whether it was reasonable for case managers to deliver this treatment given constant pressure to meet goals of housing Veterans within specified time frames. MISSION-Vet is a comprehensive treatment that requires 2.5 h a week of clinical time, which was incompatible with the increasingly large case load size of the HUD-VASH case managers.
Another significant challenge was that the data feedback system established through the GTO strategy experienced a number of difficulties which undercut its effectiveness and impact. First, case managers and peer specialists did not always use the note template, and it is likely that more services were delivered than recorded for many of those staff. Second, the VA data infrastructure made capturing the data extremely difficult and required searching, accessing, and merging records across multiple systems. Although the data was checked for errors, it is likely that some data errors remained. Third, because the VA system did not have an easy mechanism to capture the MISSION-Vet service data, there were often discrepancies between what case managers reported they were delivering and what the data reports stated. All together, these circumstances made the resulting feedback reports less accurate and thus the case managers were mixed on the utility of the feedback reports.
Outer setting
Serving patients was a priority for HUD-VASH case managers, and that facilitated trying MISSION-Vet according to the CFIR ratings. Although not officially rated, two other factors in the CFIR Outer Setting domain, Peer Pressure (competition between teams), and the presence of external policies and incentives may have been working against implementation. That is because there was not a large-scale effort within VA to use MISSION-Vet within HUD-VASH—i.e., there was little national momentum for its implementation. Thus, there were no external policies calling for MISSION-Vet across the VA nor were there any external incentives to deploy it. In the context of President Obama’s pledge to end Veteran homelessness by 2015, the incentives were tightly aligned to securing housing.
Implementation process
Ongoing planning meetings of the GTO teams at each site helped keep the HUD-VASH case managers focused on trying out MISSION-Vet. The GTO technical assistance staffperson, who ran those meetings, was the external change agent. However, case managers reported great difficulty in engaging Veterans to try MISSION-Vet. As stated, MISSION-Vet is intensive, and many Veterans in HUD-VASH who have substance abuse disorders may not have been ready to engage in such an intensive treatment.
Limitations
Certain limitations should be noted. First, peer specialists were not available to teams in the IU condition in two of the three sites. The activation provided by GTO facilitated peer specialists being added to the intervention (GTO) teams, but the roll out of peer specialists varied: one team used contract peer specialists which involved greater complexity and coordination, one team had significant turn over in their peer specialists during the study, and one team experienced challenges in peer specialists recording notes into the templates. In addition, the shortage of peer specialists in this study meant that the case managers in all three GTO teams had to “share” peer specialists, which is not typical for MISSION-Vet implementation. Second, we were able to interview only one of the eight peer specialists who had been involved over time at the three teams. Third, the data from the electronic medical record were limited and relied upon case managers, not trained data collectors. It was not possible to completely confirm accuracy of the data, although we were able to make some data corrections by getting feedback from the case managers (via the data feedback process). Fourth, we know that the services documented underrepresented the actual amount of services delivered. Case managers and peer specialists in all GTO sites reported delivering some services not recorded on the CPRS note template. Lastly, using all Veterans on the HUD-VASH sub-teams as the denominator for calculation of Reach likely made Reach rates appear lower and could be considered a “lower bound” for Reach. That is because not all HUD-VASH Veterans have a dual diagnosis and would be appropriate for MISSION-Vet services.
Conclusions
This study suggests that while GTO implementation support was critical to the launch of MISSION-Vet, this alone does not equate to service delivery with high fidelity for a complex intervention like MISSION-Vet. Despite the support for the launching of MISSION-Vet services, a constellation of implementation factors on the ground (highlighted by the CFIR model) exerted tremendous influence on the amount of MISSION-Vet delivered and consequently reduced fidelity. This project was launched right as the VA was tasked to reduce the number of Veterans who were homeless to zero and HUD-VASH had been asked to adopt a Housing First philosophy (accepting Veterans into services much earlier in their recovery). As a result, case managers were under a great deal of pressure to house more Veterans who had more significant impairments than ever before, which left little room to take on new initiatives, regardless of their utility. It is possible that through the GTO approach, more could have been done to engage leadership levels beyond the team at each site; however, in the absence of any national momentum or external policies, it was unclear what incentives to leverage.
It could be concluded that MISSION-Vet was not an optimal fit for the HUD-VASH program at the time this study was conducted. Nonetheless, while choice in service engagement is a core philosophy for homeless clients in Housing First programs, it is also important to address co-occurring metal health and substance abuse to prevent housing instability and loss, an area with which the field is still struggling to optimally deliver support [
43]. Thus, given its evidence base and philosophical overlap with HUD-VASH, it is possible that under different circumstances, MISSION-Vet could be successful. Since the end of the trial, the national HUD-VASH program has funded additional training for Critical Time Intervention (a key component of MISSION-Vet), multidisciplinary teams, and shared caseloads as a way to improve access to a variety of services including integrated dual disorders treatment. As the study ended, a GTO team at one of the sites was in the process of creating a special intensive case management sub-team in which case managers with very low caseloads would use MISSION-Vet to treat high-acuity patients. This idea appears to be more favorable as it has high leadership support, has high priority, and would be more compatible with the new case load configuration—CFIR factors that were implementation barriers during the study. Overall, this project shows that GTO-like support can help launch new practices but that multiple implementation facilitators are needed for successful implementation of an evidence-based program like MISSION-Vet.
Acknowledgements
The contents of this paper are solely from the authors and do not represent the views of the US Department of Veterans Affairs or the US Government. We would like to acknowledge the contributions of Vince Kane, Jesse Vazzano, Julianne Siegfriedt, Brittany Walker, and Rachel Mullins for their assistance with this project.