Quality improvement, implementation, and dissemination strategies to improve mental health care for children and adolescents: a systematic review

Approximately one in five children and adolescents living in the USA has one or more mental, emotional, or behavioral health disorders according to the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) criteria in any given year [1]. These disorders contribute to problems with family, peers, and academic functioning. They may exacerbate coexisting conditions and reduce quality of life. They also increase the risk of involvement with the criminal justice system and other risk-taking behaviors and suicide [2].

Several key publications in the mid- to late-1990s suggested that usual care in children’s mental health had, at best, no [3] and sometimes harmful effects [4]. Since then, mental health interventions that improve outcomes for children and adolescents with mood disorders, anxiety disorders, disruptive behavior disorders, psychotic disorders, eating disorders, and substance use disorders have been tested and shown to yield varying degrees of benefit [5, 6].

Despite advances in the evidence base [5, 7], some outcomes for children with mental health problems remain suboptimal. Reasons include issues with access to care and the failure of systems and providers to adopt established quality improvement (QI) strategies and interventions with proven effectiveness (e.g., evidence-based practices (EBPs)). Studies using nationally representative data on US adolescents show that only approximately one in five children with mental health problems receives services and only one third of treatment episodes are considered minimally adequate (at least four visits with psychotropic medication or at least eight visits without psychotropic medication) [8‐10]. The current health care system continues to provide fragmented care to children and adolescents in numerous uncoordinated systems, rendering inefficiently the delivery of needed services [11]. Moreover, clinicians—particularly primary care practitioners—may lack the time, knowledge, or training to identify and treat or refer patients with mental health problems [12].

Given the gap between observed and achievable processes and outcomes, one way to improve the mental health care of children and adolescents is to adopt QI practices; another is to develop strategies to implement or disseminate interventions with known effectiveness. Such strategies target changes in the organization and delivery of mental health services [13, 14]. They seek to improve the quality of care and patient outcomes by closing the gap between research evidence and practice [15‐17].

Some investigators consider implementation and dissemination strategies as a particular subset of initiatives to improve the quality of care. However, the field of implementation and dissemination is so new that the conceptual framework and terminology in relationship to QI efforts have not been fully standardized yet [18]. We do not take a position on the taxonomy of these terms. More information about the definitions used in this review can be found in the full report on this topic [19].The ultimate goal of these strategies is to improve patient health and service utilization outcomes for children and adolescents with mental health problems through system interventions, not clinical ones. Intermediate outcomes in this context include changes to health care systems, organizations, and practitioners that provide mental health care. Targeting multiple, interrelated, nested levels—such as the macro environment (e.g., states), organization or system (e.g., specialty mental health clinics), program (e.g., selected interventions), practitioner (e.g., psychologists), and patient (e.g., children or adolescents and their families)—typically increases the effectiveness and sustainability of a particular strategy [20, 21]. These outcomes represent implementation, dissemination, or QI outcomes and are distinct from but can influence patient-level outcomes [20], For instance, changes in intermediate outcomes such as practitioners’ attitudes [22] or organizational climate [23] may influence the successful adoption of and fidelity to EBPs. These practices in turn influence patient health outcomes, such as behavior or quality of life.

We developed the topic, key questions (KQs), outcome list, and analytic framework for this systematic review through a comprehensive, evidence-based, and public process. We refined review criteria used by two recent reviews [24, 25] and built on extant literature [20‐23, 26‐28] on this topic to focus the review on a narrower set of inclusion and exclusion criteria. We aimed to decrease the heterogeneity of findings, add more recent studies, and seek studies that may have examined differential efficacy by patient, provider, or system-level characteristics. A panel of key informants gave input on the proposed scope and methodology. Public comment was solicited in response to online posting on AHRQ’s Effective Health Care Website. We revised the KQs, outcomes, and analytic framework in response to all gathered information.

We drafted a protocol for the systematic review and recruited a panel of technical experts to provide high-level content and methodological consultation throughout the review. The final protocol was posted on the Effective Health Care website at http://​effectivehealthc​are.​ahrq.​gov/​search-for-guides-reviews-and-reports/​?​pageaction=​displayproduct&​productid=​2030 on December 30, 2014, and registered on PROSPERO (Registration number: CRD42015024759). Following release of a draft of the systematic review and peer review, we amended our protocol to include additional review and analysis methods suitable for complex interventions (described below in the “Methods” section).

Our key questions included the following:

Figure 1 depicts our analytic framework. Note that all KQ focus on the effectiveness, harms, and moderators of outcomes (effectiveness or harms) of strategies to implement, disseminate, or improve quality of mental healthcare for children and adolescents. The benefits and harms of strategies accrue at multiple levels: systems or organizations, practitioners, and patients. Clinical interventions that focused solely on improving health outcomes were not eligible.

KQ 3 was intended to evaluate the effect of moderators on any outcomes reported in KQ 1 (either intermediate or ultimate patient health or service utilization benefits to systems or organizations, practitioners, or patients) or KQ 2 (harms to systems or organizations, practitioners, or patients). Because the effect of the moderator may vary based on the nature of the outcome, our planned analyses did not combine categories of outcomes.

We followed procedures specified in the Methods Guide for Effectiveness and Comparative Effectiveness Reviews from AHRQ (available at https://​www.​effectivehealthc​are.​ahrq.​gov/​search-for-guides-reviews-and-reports/​?​pageaction=​displayproduct&​productid=​318). The review uses the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist to facilitate the preparation and reporting of the systematic review [29].

We summarize results by KQ below for searches through February 17, 2017. The full report [19] provides detailed descriptions of included studies, key points, detailed synthesis, summary tables, and expanded strength of evidence tables for studies via original searches conducted through January 14, 2016.

Overall, 12 of the 19 studies demonstrated effectiveness as measured by moderate or mainly low strength of evidence for benefit for at least one outcome of interest. Our confidence in these results is limited by the paucity of studies on any strategy. We graded benefits of pay for performance as moderate strength of evidence [53]. We graded the strength of evidence of benefit as low for at least one outcome among strategies that contained:

We found low strength of evidence of no benefit for intermediate outcomes for strategies that involved the following combinations of professional components:

We were unable to judge the potential for harms associated with these strategies that might mitigate benefits. We had only a single study on early intervention for first-episode psychosis, and it reported no adverse events and no differences in false-positive referral rates. In addition, the available evidence from four studies on two moderators does not permit us to make general conclusions about the conditions under which these strategies might work optimally.

The studies varied with respect to the numbers and types of active components, i.e., we observed considerable differences in components in treatment group strategies and comparison group strategies. In some studies, the treatment group contained several components and the comparison group contained none of those components. In other studies, both the treatment and comparison groups tested strategies with multiple components, with different numbers of variations in components across arms. Because both arms often received active interventions, the Hawthorne effect may explain lack of effectiveness. We did not find any consistent patterns of effectiveness involving the number of active components. That is, we did not find that studies that employed strategies with a single active component had any better or any worse effect on outcomes than those that employed multiple active components.

Additional heterogeneity arose from several other sources and precluded any quantitative synthesis of our findings. Except for two studies reported in one publication [55], two trials (three publications) reporting variants of a similar strategy [14, 51, 60], and two trials reporting different types of feedback strategies [13, 61], no other studies tested similar strategies. The outcomes of the studies varied widely. Similarly, settings differed greatly (community-based hospitals and clinics, general practice and primary care, home-based mental health systems, schools, substance abuse treatment). Finally, the targets of each strategy, such as practitioners, practices, or systems, also differed considerably.

Challenges in this systematic review arose with defining the intervention of interest, constructing the search strategy, and applying prespecified inclusion/exclusion criteria. The lack of consistency in the terminology used in the published literature meant that the use of self-selected descriptors such as “QI,” “implementation,” or “dissemination” by study authors did not conform either to our a priori definitions of these types of studies or to the other similarly labeled studies in the field. This lack of consistency led to our reliance on the EPOC taxonomy as our primary analytic framework.

Strategies differed considerably in the number of components; the reporting on these components was not always clear enough either to describe the strategy adequately or to let us understand fully the relative importance of component parts. New taxonomies are continually emerging, such as the revised EPOC taxonomy [63] and the refined compilation of implementation strategies from the Expert Recommendations for Implementing Change (ERIC) project [64]. Both might help advance the field by clarifying the conceptual models that underlie this research.

Trying to specify the population and comparison criteria to ensure greater homogeneity of included interventions posed additional challenges. For example, our focus on children and adolescents with existing mental health issues (rather than only the risk of mental health issues) meant we could not examine issues of prevention. In addition, although we included a broad range of eligible comparators in our protocol (usual care, or any other QI, implementation, or dissemination strategy), we did encounter otherwise eligible studies in which the intervention combined both a patient-level intervention and a system-level strategy to implement or disseminate that intervention. Because the use of a “usual-care” arm did not permit the original investigators to draw conclusions about the effect of their implementation or dissemination strategy apart from the underlying intervention, we excluded these studies for having a wrong comparator [65‐72]. In addition, studies often offered limited descriptions of usual-care arms relative to descriptions of experimental arms. Even with incomplete reporting, however, we found wide differences in the number, intensity, and services offered in usual-care arms. These differences sharply restricted our ability to make statements about the overall effectiveness of these strategies as a class.

Reporting issues in the literature also hindered our ability to derive firm conclusions on the effectiveness of included strategies. Authors reported complex analyses but often did not report other aspects of their methods well enough to permit an independent evaluation of the effect size [57], precision of the effect [46‐48, 51], or risk of bias [46, 57].

We did not find evidence on the majority of the outcomes that we had specified a priori. Of particular note, seven strategies (two from a single publication) relied on EBPs and did not report patient health outcomes [50, 54‐56, 58, 59].

When researchers maintain fidelity to the original intervention, assuming that the same level of effectiveness will occur in a new trial is reasonable; adopting this assumption can, therefore, produce efficient use of research funds. Unfortunately, not all studies measured fidelity adequately. New strategies relying on EBPs must, at a minimum, report on fidelity so that practitioners and policymakers can judge whether the strategy is, in fact, a new intervention rather than implementation or dissemination of an existing intervention. Information on pragmatic issues related to implementation (fidelity, adaptation, and minimum elements necessary to achieve change) may not necessarily require new studies on strategies with existing information; support of analyses done with data from existing studies may fill some of the gap.

The sparse evidence base of a set of diverse strategies and outcomes focusing on intermediate and health outcomes and resource use highlights the fact that clinicians and health plan administrators still do not have adequate knowledge of best methods to introduce EBPs successfully into clinical settings for children and adolescent populations. Third-party payers are paying increasing attention to quality metrics, as health care systems move to accountable care models. We found no studies on regulatory components and just one study testing the effectiveness of a financial component, specifically for pay for performance [56]. Future research efforts should evaluate variations of such programs according to patient, provider, organization, systems, and setting characteristics. A better understanding of these variables may help promote the implementation and dissemination of EBPs.

The majority of included studies appropriately used cluster designs. Cluster RCTs, like pragmatic trials, typically need more resources than conventional RCTs. Moreover, they can be harder to analyze than conventional studies. An additional consideration (for all these trials) related to inadequate reporting, a problem often noted in the literature [73, 74]. The studies we found were marked by poor reporting or failure to report key details of the strategy or differences across study arms. A recently developed tool, the StaRI (standards for reporting implementation studies of complex interventions), offers standards for reporting implementation studies [75]. If adopted widely, StaRI may well considerably improve the utility of these studies and the pace of translation of evidence into practice.

Although the failure to use EBPs can lead to gaps between potential and achieved outcomes, closing such holes in the knowledge base requires more than just using an array of EBPs. What continues to be unknown is how to bridge the gap in the context of the finite resource of time allocated for a patient encounter and what implications changes will have on current work flow processes. As expectations increase for documenting or checking off quality metrics for each action within a patient encounter, the risks of errors of both omission and commission rise. For new information to be actionable, more persuasive evidence is needed on the relative merits of each action or strategy.

More research is needed on strategies for the QI, implementation, and dissemination of EBPs relating to both psychological and behavioral treatments and medication interventions for treating youth suffering from mental illness. Other important research targets include developing and testing dissemination strategies for introducing mental health care into geographic areas lacking in mental health care, such as very rural areas with fewer mental health providers than more urban locations have. In these areas especially, focusing more attention on primary care providers may be essential.