The comparative effectiveness of Core versus Core+Enhanced implementation strategies in a randomized controlled trial to improve substance use treatment receipt among justice-involved youth

A high proportion of youth involved in juvenile justice (JJ) exhibit problematic levels of alcohol and other substance use (SU). For this reason, and because SU services are linked to outcomes including reduced recidivism [1, 2], addressing SU has become a focus of JJ agency efforts to lower recidivism and address youth needs. While some agencies provide SU treatment directly, most refer youth to external service providers [3], which requires cross-system collaboration in identifying and addressing SU needs [4, 5]. The current study, from the Juvenile Justice-Translational Research on Interventions for Adolescents in the Legal System (JJ-TRIALS) Cooperative, examines the effectiveness of two implementation intervention bundles for promoting SU service uptake within complex systems involving JJ and behavioral health (BH) agencies.

Despite considerable variability across justice agencies in Cascade-related practices, this investigation detected an impact of the JJ-TRIALS intervention on youth treatment service outcomes, adjusting for youth, justice, and county characteristics. Findings document that Core intervention strategies (training teams on the BH Services Cascade, DDDM, and supporting goal selection) were effective at increasing service receipt over time relative to baseline (H1). When comparing cohorts of youth who entered during different study periods, Initiation rates increased 5.7% from Baseline to Early/Late Experiment. Engagement and Continuing Care rates increased significantly between Baseline and Early Experiment periods and extended to later periods. There was no difference between Core and Core+Enhanced conditions on receipt of treatment services over time (H2), neither study phase nor its interaction with condition was significant. For H3, youth initiated treatment 51.3% faster in Core+Enhanced sites than in Core sites (H3).

Findings also document that Cascade Penetration (last service stage achieved) increased from Baseline to Early and Late experiment in both conditions (H4), and the Core+Enhanced intervention (external facilitation of team progress toward goals using DDDM) was more effective in Penetration across study periods compared to Core (H5). These data suggest that efforts to improve services along the Cascade are more effective when teams work iteratively and collaboratively with external BH providers and have the guidance, support, and accountability that external facilitation provides. This type of outer/inner context collaboration is one critical bridging factor that can lead to more effective system-level implementation efforts [47]. It is worth noting, however, that improvement was modest even within Core+Enhanced sites; most youth in need did not receive treatment. Given higher costs associated with Core+Enhanced strategies, future research should examine its cost-effectiveness compared to Core.

This study extends prior research on SU Referral (under the purview of JJ) to services most often delivered by community partners. The pattern showing greater increases in referral rates over time in Core+Enhanced sites [39] was similar to findings for Penetration in this study but was not replicated for treatment service receipt. A number of distinctions between these two investigations may explain the differences. The Belenko et al. study included 30 sites with data through referral, whereas this study included 20 sites with data covering the full Cascade. The smaller sample included sites that prioritized documenting service utilization outside of JJ and could differ in terms of inter-organizational collaboration and capacity to implement change [4, 5, 48]. Moreover, treatment referral was at the discretion of the JJ agency and may have been particularly sensitive to the JJ-TRIALS intervention. The focus was primarily on JJ system change, with representation from BH on implementation teams to provide consultation and support. Expectations that BH agencies would modify their programs and practices (not specific to JJ referrals) were only implied. Future efforts should attempt to balance these partnerships or expand to focus on BH system change, with consultation from JJ. Additional implementation supports that could foster greater sustainment might include a longer intervention period (one year may be insufficient to develop effective cross-system coordination); financial and technical support for improving data infrastructure and quality; virtual tutorials or technical assistance in using data in real time (e.g., charting trends using MIS records); additional training and incentives for probation officers to engage with BH providers; and ongoing feedback for probation officers (based on records audits) regarding service receipt among youth on their caseloads. Beyond these implementation factors, later Cascade events are likely influenced by a complex set of determinants, including JJ sanctions, BH waitlists, and youth/family considerations (e.g., motivation, transportation barriers, family disorganization). Therefore, future studies should also consider other outcomes related to treatment Engagement such as parental support, motivation for change, adjudication and disposition decisions, etc.

Perhaps the predominant finding was the ubiquity of site variability across jurisdictions, target services, and implementation over time. Observing between-site differences is more rule than exception for multisite behavioral intervention trials [49], at least in JJ settings given the variability in legal codes and jurisprudence goals [42]. Although outcomes research characteristically places site variability in the background (via statistical control procedures) in order to emphasize generalizability of results, it can be equally illuminating to foreground site differences to better understand dynamic interactions of salient inner and outer context factors on targeted outcomes. Future studies should consider using individual-differences analytic approaches [50] and leverage qualitative data to investigate the myriad of causes of and impacts on site variability. For example, variations in how sites determined need for SU treatment (e.g., corroborating evidence from two or more sources versus relying on a single source) could affect referral decisions and Cascade retention. As described herein, the Cascade assumes that youth will ideally remain in services through continuing care; however, some referrals may be deemed inappropriate by treatment staff (e.g., when referral is based primarily on parent perception). Future studies should attempt to address the possibility of appropriate exit out of the Cascade through prospective data collection and/or qualitative inquiry.

Site variability highlights the need to further understand change processes, including strategies used by implementation teams as they planned and orchestrated system change [51]. For example, implementation teams in some Core+Enhanced sites spent months developing and refining tools to standardize and inform referral decisions, whereas some piloted changes before implementing practices agency-wide. Such activities might have produced qualitative improvements (e.g., more efficient and accurate identification) that take months or years to be reflected in service receipt patterns. While some Core sites applied DDDM, many tackled simpler modifications to JJ practices with immediate benefits such as modifying referral forms or offering JJ office space to contracted service providers. While contracting between outer and inner context service organizations is an EPIS bridging factor, the nature of contracts, including language, statements or work, and fiscal incentives, may affect service results [18, 47].

Sustainment is a critical concern in implementation research, and beginning with sustainment in mind – as a critical goal – cannot be understated [18]. While the expectation of continued gains during the Maintenance study period was not stated explicitly in hypotheses, EPIS-inspired study design and intervention components included an emphasis on sustainment [20]. In these data, means for Maintenance cohorts were low across Cascade events, likely because the study concluded without adequate time for youth entering sites to progress past screening/assessment. The degree to which practice changes were sustained after the study concluded is unknown. Future work should examine how gains attained in demonstration projects can be institutionalized in service systems [52, 53]. Attaining sustainment requires going beyond usual funding and accountability mechanisms (e.g., grants) to consider how practices and approaches fit within the complex and dynamic contexts of the broader system.

It is important to acknowledge potential limitations. First, it is not certain how much service receipt data were missing and for what reasons (e.g., not collected, not available) [13], nor the degree to which use of dichotomous measures limited power to detect significant effects. Records data include errors that may operate in multiple directions, and summarizing over multiple items helps to cancel out random error and focus on the underlying signal or “effect.” Summarizing across Cascade steps within a site (i.e., Cascade Penetration) results in a more stable estimate of the effects within (and therefore across) sites due to less variance, larger effects, and consequently more power than analyses focused on individual steps. To sufficiently identify and address existing barriers to Cascade penetration and determine the success of change efforts, JJ systems must invest in quality data captured throughout the Cascade, incorporating data from BH providers where feasible.

Second, the sample was not nationally representative, so findings may not generalize to other jurisdictions; however, youth and agency characteristics were sufficiently diverse and comparable to a nationally representative survey of JJ agencies [3]. Third, the subset of sites that collected data across the full Cascade may have better relationships with service providers compared to those who do not capture that data. They could also have leadership at the facility, county, and/or state level that prioritized and funded SU and/or other services. Fourth, the EPIS-inspired study design examined change over time by assigning cohorts of youth to study phases based on date of entry into the system. The lower means among the maintenance cohort could be attributed to the shorter time span available for youth to receive services. For example, receipt of services for youth in the Baseline cohort occurred over a 2-year period, compared to only 6 months (and sometimes less for youth entering late in the period) for youth in the Maintenance cohort. Given that on average youth were screened 16 days after JJ entry and 156 days elapsed between Screening and treatment Initiation, many youth in Maintenance (and some in Late Experiment) may have initiated after the study concluded. Finally, several measures that could potentially help explain these findings were not captured in the records (e.g., change in youth supervision status over time, positive random urine screenings, history of SU or mental health treatment).

Results from this study demonstrate that change is possible in complex service systems involving JJ and BH agencies. Improvement in SU treatment initiation occurred across Core and Core+Enhanced conditions, the former without external facilitation. Furthermore, Core implementation activities appear beneficial even when instruction is not specific to an evidence-based practice and target goals are allowed to vary along the Cascade. Better Cascade penetration occurs when external facilitation is provided, but wide variation exists in the degree and nature of change across service systems.

Findings demonstrate the criticality of early EPIS phases from exploration through implementation, suggesting that Core strategies provided early on are effective at producing some improvement in treatment initiation rates. Implementation-focused activities in the Core+Enhanced condition (facilitation of implementation teams) are effective at incremental improvement in moving youth farther along the Cascade. Therefore, using a collaborative, multi-agency approach to system change, that utilizes a data-driven approach (with or without intensive facilitation), can be useful in improving SU treatment initiation rates, although substantial gaps remain in engaging youth in treatment after initiation. Focusing on improving SU identification and service receipt among justice-involved youth translates to benefits for public health and public safety.