In this study, we found no evidence that the CDT implementation strategy achieved higher overall implementation compared to that for IND using either a composite score or assessments of how many stages were completed, how fast they were achieved, whether a county achieved placement of any youth, or whether a county achieved full competency. These findings were dominated by an overall low rate of MTFC placement; 35% in the CDT condition and 32% in IND. Among those counties that did place at least one youth in MTFC, there were indications in both quality of implementation as well as quantity that CDT performed better than IND. Thus it appears that CDT's effect was negligible in achieving first placement as well as speed or extent of moving through implementation states. CDT did, however, appear to increase the number of placements and increase the quality of implementation once implementation began. For these counties, CDT impact appeared to result in more robust MTFC programs as indicated by having significantly more youth placed in care during the study period among counties that began placements, and by having completed more implementation activities. None of the other secondary hypotheses were confirmed in this study.
Limitations
Even with 51 counties in two states, this study of non-early adopting counties still had limited statistical power to examine some outcomes. Some of the non-significant findings might be affected by sample size issues; analyses showed some non-significant gains in implementation from CDT compared to IND. In particular, more CDT sites (n =4) were certified during the study period than IND sites (n =1), but this difference was not significant, and statistical power was clearly influenced by the overall low rate of certification. It is well recognized that the statistical power for time to event and binary measures (e.g., milestone attainment) have lower statistical power than do most analyses of continuous quantity and quality measures. Even for the primary analysis based on a composite measure whose distribution was close to normally distributed, the effect size was 0.28 while the p value was not close to significant (p =0.42). Such an effect size is considered small but could be meaningful in large-scale implementation. Also, despite the comparatively high cost and length of this randomized trial, the low numbers of cohorts made it very difficult to assess variation across time, especially for counties assigned to CDT. We may need to find unique opportunities when evidence-based interventions are being rolled out to share the expense of large-scale implementation trials.
Another limitation in this study is that we did not explicitly examine variation in the patterns of implementation across counties, but rather considered variation more as “nuisance” parameters in our analysis. Further analyses are needed to understand whether there are distinct patterns of implementation that occur (e.g., a long pre-implementation period), and whether such patterns can be predicted and therefore provide opportunities for individualized feedback and intervention with counties that are not making adequate progress. Such analyses are currently under investigation as part of an ongoing R01 (PI: Saldana).
It is not clear from our study whether using a different learning collaborative other than CDT to support the implementation of another evidence-based program would yield more or less improvements in implementation.
The cost of using the CDT approach is a major potential caveat because this cost is added to the business as usual costs enacted in the IND condition. Cost has high relevance to policy and system leaders and is being examined in additional analyses [
16],[
33].
Interpretations
The Community Development Team implementation strategy, which uses trained consultants with knowledge of local policies and conditions to guide problem solving in teams of counties facing similar implementation challenges, may be particularly important in implementing complex mental health interventions within social service settings, such as MTFC, in non-early adopting counties. In extending Rogers’ finding that innovations that can be implemented relatively simply have high potential for rapid diffusion, we find that a more intensive implementation strategy, such as CDT, is partially helpful when implementing complex interventions to less innovation-seeking organizations, such as the social service agencies responsible for delivering mental health interventions to the targeted populations [
34]. The notion that routine face-to-face and telephone conferences allowing for peer-to peer exchanges with consultants who are knowledgeable about state and county conditions, regulations, policies, and politics seems intuitively obvious as a means to boost implementation prospects, but this is the first trial to definitively show some effects from this process. In fact, to our knowledge, this is the largest county-level randomized trial to compare two implementation strategies against each other to examine implementation effectiveness.
There are two dimensions where this study differs from other important implementation efforts involving evidence-based programs. First, the design of this study with its head-to-head randomized trial and the assessment of implementation success/failure using the SIC allows us to make empirical comparisons of distinct implementation strategies. CDC's Dissemination of Effective Behavioral Interventions (DEBI) program [
35], for example, used a national training center to provide training in HIV prevention programs to all community-based organizations that are interested, so there is little opportunity to test implementation effectiveness. Similar to the two arms of this implementation trial, DEBI provides training on the delivery of the program itself. Unlike the two arms in this study, DEBI does not provide much technical assistance pertaining to capacity building at the community level, nor on supervision once training has ended. By not including an implementation measure such as the SIC, the DEBI program has limitations in learning how important these additional steps are.
A second unusual feature is that CDT involves a peer-to-peer process of addressing challenges in delivering an evidence-based intervention. Many other implementation strategies rely solely on technical support delivered to a sole system. For example, the Blueprints Replication Initiative provided extensive capacity building support to communities to implement those top-tier programs that Blueprints had identified and whose providers also had, at that time, sufficient capacity to deliver implementation training [
36]. In the Blueprint implementation project, they concluded that their training program led to high fidelity program delivery. Unlike our study, it is not immediately clear how many communities this Blueprints Replication Initiative initially contacted to participate in this study, nor was there any indication that implementation would be successful with any communities other than the early adopters.
We note that a broad-based prevention support system, Getting to Outcomes (GTO), has used a non-randomized design to evaluate their implementation strategy [
37] in six organizations that received GTO support contrasting these to four that did not [
37]-[
42]. They reported no significant effect on self-efficacy across the two conditions but strong increases within GTO communities as a function of GTO participation. Three of the six hypothesized process components, evaluation-decision making, process evaluation mechanics, and continuous quality improvement mechanics showed higher scores among those sites that received GTO compared to control.
Our own trial provides important findings regarding the use of a particular learning collaborative, CDT, in implementing a mental health intervention within social service settings. While we have not decomposed the effects of the different components of CDT, the improvements in quality and quantity of implementation that we have found suggest limited optimism for the use of certain aspects of quality improvement collaboratives. In Nadeem et al.'s review of quality improvement cooperative research, they identified five randomized controlled trials of such implementation strategies, three of which used active control comparisons such as we have done [
22]. This study thus contributes to this small but important literature.
In addition to providing some evidence for the hypothesized outcomes for CDT, this study also succeeded from a design point of view. The study used a novel rollout randomized implementation trial design in two states to compare two implementation strategies focused on one evidence-based mental health intervention. In this head-to-head randomized implementation trial, counties were randomized to both the timing of implementation and implementation condition. We had no difficulty obtaining consent from counties to participate, and throughout the design, we were able to keep counties true to their assigned condition. No county dropped out of this design once they began, although several elected not to implement MTFC. This is not completely surprising given that all of the participating counties in both conditions had previously been given opportunities to implement MTFC and they had declined to do so; these counties are described as non-early adopting counties.
The assignment of counties to cohorts allowed county leadership to plan in advance for implementation, and our protocol, which allowed counties to move to later cohorts and fill vacancies while remaining in the same implementation condition, provided sufficient flexibility for counties to make timing adjustments. Our protocol of weekly research meetings involving the CDT and IND consultants who supported implementation in both conditions minimized the potential contamination across conditions. More detailed social network analysis in the California counties demonstrated that trust and influence relationships between county leaders in the two conditions were similar and not likely to affect the conclusions of this study [
43]. Finally, we note that this trial took place during a major economic recession, which did reduce the willingness of counties to implement a new program model. However, because of the randomized trial design, we could still make valid causal inferences comparing the two implementation strategies. Had this study been conducted under any design other than a randomized trial, we would not have been able to disentangle the effects of the extreme economic changes from the implementation condition effects.
We also note that this study introduces more sophisticated modeling of implementation processes than often is done. The SIC allowed us to measure implementation across multiple stages and milestones and across multiple levels of participants from county government to foster parents in the MTFC team. The SIC provided information on the quality and quantity of implementation as well. By assessing timing, quality, and quantity of implementation, we were able to pinpoint much more accurately what changes in implementation process occurred, including progress and lack thereof. We believe that this methodologic approach of measurement with a SIC scale of the three dimensions of quality, quantity, and timing is appropriate for a wide range of implementation studies. In a recently funded study, Saldana (R01 MH097748) is adapting the SIC for other child and family evidence-based programs for service sectors including schools, juvenile justice, and substance abuse treatment [
16]. The purpose is to evaluate the common or universal implementation activities that are utilized across evidence-based programs in their implementation strategies and to examine whether these universal items are equally important in achieving implementation success. Similarly, the study examines if the stages of the SIC are stable across evidence-based programs even when the activities defining SIC stages might differ. These adapted SIC tools will then be evaluated for adequate psychometric properties, including predictive ability, in order to further examine the value of implementation process and milestones in achieving successful program implementation.
The SIC scale, as well as the analytic models described here, is also relevant to the field of translational research, which has focused particularly on milestone attainment and less on quality and quantity [
44]. The traditional view of implementation as one single stage of translational research, concerning the “bedside to community” translation that begins with “bench” research, can be enriched and viewed from a broader perspective. Indeed, the SIC measurement system and the analytic methods described here, which were developed around implementation, could also be used to monitor the entire translational process from bench to bedside to community.