Overview
This project is a Hybrid Type III, cluster randomized controlled trial [
19] that compares MISSION-Vet Implementation as Usual (IU—standard training and access to the MISSION-Vet treatment manuals) to MISSION-Vet implementation augmented by GTO. This study is being carried out in HUD-VASH teams at three large VA Medical Centers over a 2-year period. Within each site, existing sub-teams (case managers and Veterans they serve) are the clusters that will be randomly assigned, which is particularly appropriate when desiring to lessen the risk of experimental contamination [
20] of the implementation strategy. Blinding in the traditional sense—e.g., data collectors blinded to subjects’ treatment assignment—will not be possible; however, all data will be coming from secondary sources. As a Hybrid Type III trial, it will emphasize the test of an implementation strategy (GTO), while using VA’s existing HUD-VASH data monitoring system to also assess the effectiveness of the treatment (MISSION-Vet) [
21] called
HOMES (Homeless Operations, Management and Evaluation System). This design is appropriate when the implementation strategy has strong preliminary evidence, the EBP in question has solid evidence but could yield weaker outcomes in new or less controlled environments than was present in efficacy trials, and the EBP has “implementation momentum” [
21]. These conditions are present for implementing MISSION-Vet in HUD-VASH, as will be shown below. The trial assesses three sets of variables: a) MISSION-Vet implementation (adoption, dose, reach, fidelity), b) outcomes among HUD-VASH Veterans (mental health and substance use, community functioning, housing), and c) MISSION-Vet implementation barriers and facilitators organized by the CFIR.
MISSION-Vet clinical intervention
MISSION-Vet is a flexible, integrated, co-occurring disorder treatment model rooted in the Health Belief Model [
22] and is accompanied by a Treatment Manual [
4], which serves as a how-to guide that describes MISSION-Vet’s core components, suggestions for service delivery, and includes a number of appendices with additional didactic materials. Optimally, each Veteran receives about 2.5 h of manualized services a week from a case manager and peer specialist team. The core service components of MISSION-Vet are critical time intervention (CTI), integrated co-occurring mental health and substance use disorder treatment using dual recovery therapy (DRT) delivered by a case manager, and peer support delivered by a peer specialist.
CTI [
23], MISSION-Vet’s core intervention, is a time-limited assertive case management model intended to reduce the risk of homelessness by providing additional support to individuals with mental illness during the transition from institutions (e.g., inpatient psychiatry units, residential treatment programs, and homeless shelters) to community living. The assertive outreach in CTI is consistent with the role of case managers in HUD-VASH. Unlike HUD-VASH, however, CTI services are delivered in three phases of decreasing intensity: Transition to community, try-out, and transfer of care. In the transition to community phase, services are intended to reinforce community living. In the try-out phase, the case manager/peer specialist team begins to reduce service intensity to help the Veteran test and readjust the community-based support systems to fill any gaps. Visits in the transfer of care phase are used to fine-tune the connections established with community-based resources.
MISSION-Vet supplements CTI with 13 DRT [
24‐
26] sessions delivered by the case manager. These sessions are highly structured, include the use of motivational interviewing [
27] and relapse prevention [
28] techniques, and specifically target the co-occurring mental health and substance abuse issues commonly facing homeless Veterans. In addition to the 13 DRT sessions, peer specialists deliver 11 structured psycho-educational sessions that are designed to empower Veterans to plan for a life of stability, sobriety, and community integration. Peer specialists give out the MISSION-Vet Consumer Workbook [
5] at the onset of treatment to promote treatment engagement. Designed to serve as a compliment to the Treatment Manual, the Consumer Workbook helps Veterans integrate DRT and peer support concepts and increase their engagement in outpatient services through homework assignments, readings, and checklists.
In addition to CTI, DRT, and peer support, the MISSION-Vet Treatment Manual provides case managers/peer specialists with information regarding employment supports [
29] and trauma-informed services [
30], both of which have been found to be effective when used with the target population. While each component of MISSION-Vet has demonstrated efficacy on its own, when integrated using the MISSION-Vet platform, they work synergistically to increase treatment engagement, improve mental health and substance abuse outcomes, and reduce ER visits, re-hospitalizations, and recurring homelessness [
31‐
34].
Besides the MISSION-Vet Treatment Manual and Consumer Workbook, MISSION-Vet also includes a structured webinar training found in previous studies to be successful in conveying needed information about the model (Smelson et al., unpublished manuscript). These 2-h trainings provide an overview of the MISSION-Vet treatment model and offer an introduction to the developed materials (manuals and fidelity measure). The training also covers the role of the supervisor whose job is to provide ongoing consultation ensuring fidelity to MISSION-Vet. This training is part of the IU strategy, along with key information on how to access and use the MISSION-Vet Treatment Manual and Consumer Workbook.
GTO implementation model and strategy
GTO strengthens the knowledge, attitudes, and skills practitioners need to carry out evidence-based programs. GTO does this by posing a series of steps practitioners should follow in order to obtain positive results and then provides practitioners with the guidance necessary to complete those steps with quality (i.e., to perform each task as close to the ideal as possible). According to GTO, “carrying out” an evidence-based program involves much more than service delivery to patients, and the steps roughly correspond to three general areas: (1) planning—e.g., developing goals and performance targets, ensuring the staff are trained in the evidence-based program; (2) implementation—e.g., monitoring program activities, maintaining adherence to an evidence-based program model, supervision; and (3) self-evaluation—e.g., tracking patient outcomes, using data to improve program operations. All of these steps are designed to be logically linked so that the goals and performance targets are linked to program activities that will meet those targets, which in turn are linked to process and outcome measures to assess if the targets are being met, which in turn is linked to quality improvement activities that makes use of the process and outcome data. For all these areas, the guidance from the GTO approach comes in tools (GTO Implementation Handbook), face to face training for program (i.e., HUD-VASH) staff, and ongoing technical assistance provided by a GTO representative who meets regularly with the staff to help complete the needed steps. The goal is to work with leadership and staff to integrate the practices GTO targets into routine operations, closing the gap between research and practice.
GTO draws from multiple theories. It [
16] is an operationalization of empowerment evaluation theory [
35], which states that positive results are more likely when evaluators collaborate with program implementers and provide them with the tools and opportunities to evaluate and improve outcomes themselves. Consistent with social cognitive theories of behavioral change [
36,
37], GTO training and technical assistance (TA) enhances knowledge about GTO-related activities, which improves attitudes towards these activities, improves execution of GTO-related behaviors, and supports strong implementation of EBPs [
38] and outcomes.
The GTO implementation strategy is also grounded in implementation theory, for example, operationalizing the CFIR to ensure that all the major domains that influence implementation are considered [
16]. Intervention characteristics (e.g., evidence strength and quality, relative advantage, complexity, adaptability), the first domain, can influence whether practitioners adopt an intervention. Without adaptation, many interventions are a poor fit. GTO helps practitioners tailor interventions to fit with their target population, organization, and broader community. The next three domains of the CFIR comprise the outer setting (e.g., broader social, political, and economic context including policies, incentives, and resources), inner setting in which the intervention is implemented (e.g., context of the specific organization or group implementing the intervention including the structural characteristics, relationships, and implementation readiness), and the characteristics of the individuals involved (e.g., knowledge, skills). In this study, the inner setting is the HUD-VASH team, the outer setting is the broader VA, and the individuals involved are the HUD-VASH staff. To create capacity and conditions for successful implementation of programs, the GTO implementation strategy targets both individual staff and its leadership (it is beyond GTO’s scope to specifically alter the broader social, political, and economic context of the outer setting). The active change process designed to facilitate individual and organization use of the intervention as designed is the last domain in CFIR. GTO proactively engages both the individual and program levels to establish a systematic implementation process.
A great deal of research has been done to examine GTO’s effectiveness. With practitioners of drug prevention programs, GTO has been found to improve the capacity of individual practitioners and the performance of prevention programs in both quasi-experimental and experimental trials. In addition to homelessness, GTO has been adapted to a number of other content areas including drug prevention, underage drinking prevention, teen pregnancy prevention, and positive youth development [
16,
39‐
41]. Since its inception in 2004, more than 100,000 visits have been made to the website where the GTO model is described and over 100,000 GTO Implementation Handbooks have been downloaded (
www.rand.org/gto).
Participation sites and recruitment
Site composition
This study includes three large HUD-VASH sites: site A (450 HUD-VASH vouchers and 18 case managers), site B (850 HUD-VASH vouchers and 27 case managers), and site C (810 HUD-VASH vouchers and 24 case managers), for a total of 2110 HUD-VASH vouchers and 69 case managers. Case manager and voucher numbers were based on a point in time and are expected to change as new vouchers are issued over time. Each site has naturally occurring sub-teams of case managers (sites A, B, and C each has two clusters), which will be randomized with equal probability to IU or GTO by the team statistician using a random number generator. We chose to randomize case managers within each of these sites because it holds constant the variation due to site-level characteristics, funding streams, regulations, data collection activities, and political climates.
During the planning of the study, the three HUD-VASH programs were selected because they were similar across many key factors. For example, the number of vouchers allocated to each site is close to 500 with 85–95 % of these vouchers in use at each site. HUD-VASH case managers at each site are typically expected to have contact with each Veteran at least once per month—with a goal of contacting the Veteran at the Veteran’s residence once he/she has been housed. While some Veterans need more frequent contact, no Veteran gets less than this amount at any site. Among the Veterans in HUD-VASH at these sites, the VA Homeless Network Coordinators report approximately 90 % having substance use and/or mental health diagnoses. Additionally, each of these sites reports that approximately 75 % of Veterans in HUD-VASH receive other services at the VA or in the community beyond standard HUD-VASH case management services.
Recruitment
The subjects in this study are the HUD-VASH case managers although data about the Veterans on their caseload will also be collected. All case managers were invited to participate and consented before randomization by the study staff. The two sub-teams from each site that were assigned to GTO or IU have approximately equal numbers of case managers and vouchers, although the exact numbers shift frequently due to staff turnover and Veteran dropout, graduation, and the issuing of new vouchers. Regardless, consistent with cluster random assignment, each case manager is linked to a certain sub-team (and thus study condition), and each Veteran is in turn linked to a specific case manager. If sites decide to add new staff to a sub-team, that case manager would then automatically be assigned to the condition of that sub-team.
All HUD-VASH case managers in both groups will be invited to the webinar training on MISSION-Vet. After MISSION-Vet begins, case managers will be responsible for engaging their Veterans into MISSION-Vet services. Any Veteran participating in HUD-VASH at each of these three sites is eligible to receive MISSION-Vet services. Through training and TA, case managers and peer specialists will be guided to follow the recommended inclusion and exclusion criteria of MISSION-Vet: (1) has a current substance abuse or dependence disorder and a co-occurring mental illness and (2) is willing to participate in MISSION-Vet services. However, the decision with whom to deliver MISSION-Vet will ultimately be made by the case managers. MISSION-Vet takes about 1 year to fully deliver; therefore, Veterans will be invited to take part in MISSION-Vet in the first year of the 2-year intervention period. The study received a waiver of consent to use client-level data already being collected as part of the homeless data program management system of HUD-VASH.
Measures, procedures, and analyses by study aim
All of the below measures will be collected for 3 years: during the 2-year GTO period of support and 1 year after that support ends to assess sustainability (see
aim 4 below).
Aim 1: compare MISSION-Vet fidelity between GTO and IU groups
Data source. Veterans served by case managers in the GTO and IU groups.
Measures and data collection. The MISSION-Vet Fidelity Measure tracks all the core elements of the MISSION-Vet treatment model, including CTI, DRT, peer support, vocational supports, and trauma-informed care for each individual Veteran. The fidelity index consists of 78 items assessing the presence or absence of certain activities within MISSION-Vet. Taking the responses from all non-missing items, we will compute a fidelity score for each Veteran. The data will be extracted from the VA Computerized Patient Record System (CPRS) on all HUD-VASH Veterans served by case managers in the IU or GTO conditions, at each of the three HUD-VASH sites.
Hypothesis. We expect that Veterans served by case managers who were in the GTO group will receive MISSION-Vet with greater fidelity compared to those who are in the IU group.
Data analysis. We will fit cross-sectional linear random effects (to account for clustering) models for the fidelity score, measured at 12 months post-baseline.
Aim 2: compare the effect on the proportion of time the Veteran is housed between GTO and IU groups
Data sources. Veterans served by case managers in the GTO and IU groups.
Measures and data collection. For each Veteran entering HUD-VASH, a case manager must complete a HUD-VASH Referral Worksheet that contains the program’s first assessment of days of homelessness and housing. Then, case managers provide data every 3 months on the number of days housed in the last 90 days using the Housing Progress Report form. Case managers are also required to report when a Veteran loses their housing voucher each month. These reports are stored in national database called HOMES (Homeless Operations, Management and Evaluation System).
Hypothesis. We expect that Veterans assigned to case managers in the GTO group, who receive MISSION-Vet, will have more days housed in the prior 3 months compared to Veterans who receive MISSION-Vet from case managers in the receiving IU group.
Data analysis. Again, we will fit repeated measures linear random effects models in this analysis. As a secondary analysis, we will perform a z test for the comparison of the proportion of Veterans at the end the study who have retained their housing (versus lost their voucher) between GTO versus IU.
Aim 3: compare mental health, substance use, and functional outcomes among Veterans served by HUD-VASH case managers in GTO and IU groups
Data sources. Veterans served by case managers in the GTO and IU groups.
Measures and data collection. In addition to housing, the HUD-VASH staff complete data collection forms for HOMES on four outcomes each month. HOMES requires case managers to make separate ratings on mental health, substance abuse, and associated problems for each Veteran using components of the Addiction Severity Index [
46] as well as the Global Assessment of Functioning (GAF) [
47].
Hypothesis. We expect that Veterans served by HUD-VASH case managers supported by GTO, who received MISSION-Vet, will have improved mental health, substance use, and functional outcomes compared to Veterans served by HUD-VASH case managers in the IU group.
Data analysis. We will again fit our repeated measures linear random effects models to the response variables mentioned above.
Aim 4: analyze MISSION-Vet implementation using the factors specified by the implementation model RE-AIM (Reach, Effectiveness, Adoption, Implementation, and Maintenance)
RE-AIM [
48] states that successful deployment of an intervention is mediated by the following factors:
Reach—the proportion of individuals reached by the intervention;
Effectiveness—the efficacy of the program under ideal circumstances;
Adoption—the proportion of patients that used the intervention;
Implementation—the extent the intervention was well-implemented; and
Maintenance—whether the intervention was maintained. While GTO provides guidance for successful implementation, the RE-AIM model provides a heuristic to judge the success of that implementation.
Data sources. All aspects of the RE-AIM framework will be informed by data from case managers in the IU and GTO groups and the Veterans they serve.
Measures and data collection. Each RE-AIM measure below, using data collected from aims 1–3, will be calculated for both the IU and GTO groups:
-
Reach—Obtained from aim 1, the Reach measure will be the percent of all eligible Veterans who receive MISSION-Vet with adequate fidelity (defined as the endorsement of at least 75 % of the items on the MISSION-Vet Fidelity measure).
-
Effectiveness—Obtained from aim 3, effectiveness will be the percentage of all eligible Veterans who receive MISSION-Vet with adequate fidelity who improve on each outcome in an amount consistent with a statistically significant effect.
-
Adoption—The proportion of case managers who have more than 50 % of their eligible Veterans engaged in MISSION-Vet will be judged to have “adopted” MISSION-Vet as a standard protocol for Veterans with mental illness and co-occurring substance abuse disorders.
-
Implementation—The extent to which the MISSION-Vet intervention was well implemented will come from aim 1 and be expressed as the percentage of eligible Veterans who actually participated in MISSION-Vet who had adequate fidelity (endorsement of at least 75 % of the items on the MISSION-Vet Fidelity measure).
-
Maintenance—The degree to which the successful deployment of MISSION-Vet was maintained will come from data collected through the previous components of the RE-AIM model but extended a year beyond the end of the GTO support. Thus, this study will be able to assess whether MISSION-Vet maintains a high Reach, Effectiveness, Adoption, and Implementation between the end of the GTO support and the last data collection wave, which will be a year after the GTO support ends.
Hypothesis. We expect better results across all elements of the RE-AIM model in the GTO group than in the IU group.
Data analysis. The scores for R, E, A, and I will be calculated among the Veterans in both the GTO and IU groups during the 2-year period of GTO support. For Reach, we will perform a z test for the comparison of the proportion of Veterans at the end of the GTO period with adequate fidelity between GTO versus IU. The Effectiveness analysis will involve t tests comparing mean scores at the end of the GTO period for the proportion of time housed and homeless (on the logit scale), components of the ASI, and the GAF. For Adoption, we will compare the proportion of case managers through a z test who have “adopted” MISSION-Vet according to the specified measure. We will also perform a logistic regression of whether a case manager adopted MISSION-Vet as a function of study arm, one of the moderating variables, and their interaction, testing specifically for the interaction term. The analysis of Implementation will compare through a z test the proportion of Veterans who participated in MISSION-Vet who had a fidelity score of at least 75 % between the GTO versus IU groups.
Maintenance—The analyses described above for R, E, A, and I will be similarly applied to the data collected and summarized a year after the end of the GTO support period.
Power analysis
We demonstrate that for aims 1, 2, and 3, we will have at least 80 % power to detect intervention effects at the 0.05 significance level (aim 4 is exploratory and thus we are not able to calculate power for that aim). Our power calculations were performed using the power analysis software G*Power 3.1.2 [
49]. Previous studies examining the effect of GTO used different measures than in the current study. We therefore assumed effect sizes based on a previous study of the effect of GTO on capacity to carry out evidence-based drug abuse prevention programming. The measures used in that study were of provider knowledge (effect size = 0.21 after 2 years of GTO), attitudes (effect size = 0.38 at 1 year of GTO and 0.59 at 2 years of GTO), and skills (effect size = 0.07 at 1 year of GTO and 0.28 at 2 years of GTO) all related to the performance of practices associated with evidence-based prevention programming [
16]. These measures are relevant because GTO will attempt to impact knowledge, attitudes, and skills in the proposed study as well, but in this case around the use of MISSION-Vet. We use these values as a rough guide for the effects we might expect to detect for the current study, along with conventional effect size interpretations.
For aim 1, we expect a total of 1604 Veterans to be available for the fidelity measure computations. This sample size assumes a reduction of 20 % from the initial (approximate) 2110 vouchers available due to Veterans not having co-occurring SUD and then another reduction of 5 % due to case manager refusal to participate. The latter reduction brings the number of case managers down from an initial estimate of 69 down to 66. Because the study is cluster-randomized by case manager, the sample size to detect effects is reduce by a “design effect” (Deff). The formula for the design effect is given by Deff = 1 + (
m − 1) ICC where
m is the average number of Veterans in the study per case manager and the ICC is the intra-class correlation. The ICC of MISSION-Vet + GTO effects clustered by case managers is unknown but likely to be fairly limited as GTO is a system-wide intervention. To be conservative, we use a conventionally large value of ICC = 0.15 [
50]. Based on the information above, the value of
m is approximately
m = 1604/66 = 24.3. Thus, the design effect assumed is Deff = 1 + (
m − 1) ICC = 1 + (24.3 − 1) (0.15) = 4.495. Therefore, the effective sample size for aim 1 controlling for clustering is 1604/4.495 = 356 = 265 Veterans. For an unclustered design with an available sample of 356 Veterans randomized to two study arms, the smallest detectable effect size (ES) at the 0.05 significance level with 80 % power assuming a two-sample
t test is 0.298. While this effect is of comparable or larger size to those in the drug abuse prevention study described above, this effect size, according to [
51], is considered conventionally to be between small (ES = 0.2) and medium (ES = 0.5). Therefore, our sample size seems to be sufficient to detect a small-to-medium effect size. The actual data analyses will control for socio-demographic and other important confounders, so in fact we expect greater power to detect fidelity differences due to MISSION-Vet + GTO.
For aims 2 and 3, we will focus on the subsample of Veterans in the study for whom we have 12 months of data collected over consecutive 3-month intervals. Because these Veterans serve as their own controls relative to time, they provide the greatest power in detecting the effect of MISSION-Vet + GTO. With the same design effect as in aim 1, the effective sample size available for aims 2 and 3 controlling for clustering is 356 Veterans. For a linear repeated measures model with two treatment groups and five measurements per unit (that is, one every 3 months for a year, including baseline), and assuming a sample of 356 units to randomly divide approximately equally between the two groups, the time-by-intervention effect size that can be measured with 80 % power is 0.082. By convention, this is considered a very small effect size, and it is also small compared to the ones found above in the drug abuse prevention study. We therefore anticipate detecting effects with sufficient power for aims 2 and 3.
Design challenges
An important design issue is whether to randomize at the site, provider, or Veteran level. We decided to randomize at the HUD-VASH case manager level for several reasons. First, the analysis of national HUD-VASH data showed that the HUD-VASH site was the strongest predictor of housing stability [
6]. Therefore, it was important to hold the site constant by having IU and GTO conditions equally represented within each of the sites. Further, we account for the effects of case manager clustering in our statistical analyses. Second, we believe that the possible contamination that may result between IU and GTO conditions being present at the same site is manageable. In the past studies of GTO, contamination did occur but was moderate (across multiple programs, mean = 0.5 to 1.0 on a 6-point index of possible GTO activities in which one could engage) [
16]. We have however, purposely selected large sites in an effort to minimize the contamination as it is believed that we can minimize discussion of GTO implementation strategies more easily among non-GTO case managers in larger sites. To minimize contamination further given its potential confound, we decided to use naturally occurring HUD-VASH sub-teams at each site so that team meetings and other routine discussions between case managers are handled in separate IU and GTO “teamlets.” Other limitations of the study include staff turnover and changes in policy directives. Regarding staff turnover, we anticipate some level of turnover, and our power analyses show that we have a sufficient sample. Furthermore, new staff in the IU sites will receive training on MISSION-Vet. As for changes in the program directives, we acknowledge that this is also a limitation as there is always ongoing tension between research and the need for patient care services to make and respond to new directives to end homelessness. However, our design—comparing an organizational intervention like GTO to usual implementation strategies (regardless of how they may change)—is therefore a genuine test of the impact of the GTO intervention (and MISSION-Vet as well) in real-world conditions. Thus, the findings that are produced may have greater ecological validity—or relevance to real world conditions, where circumstances do change. In addition, even if the VA did make changes to HUD-VASH, the changes would apply evenly to all HUD-VASH sites and all HUD-VASH case managers and thus be equal across IU and GTO conditions. Further, we would hypothesize that GTO would be able to help HUD-VASH case managers incorporate the changes for improved Veteran impact better than case managers assigned to the IU condition.