Considering Complexity in Systematic Review of Interventions
Assessing the applicability of findings in systematic reviews of complex interventions can enhance the utility of reviews for decision making

https://doi.org/10.1016/j.jclinepi.2013.06.017Get rights and content

Abstract

Assessment of applicability is an essential part of the systematic review process. In the context of systematic reviews of the effects of interventions, applicability is an assessment of whether the findings of a review can be applied in a particular context or population. For more complex interventions, assessing applicability can be challenging because of greater diversity of, and interactions within and between, the intended population, intervention components, comparison conditions, and outcomes as well as a range of further considerations related to intervention context and theoretical basis. We recommend that review authors plan and conduct analyses to explain variations in effect and answer questions about mechanisms of action and influence of different settings, contexts, and populations. We also recommend that review authors provide rich descriptions of the setting, implementation details, resource use, and contexts of included studies and assess applicability for at least one target population, setting, and context. This should facilitate applicability assessments by end users. Consensus on terminology is needed and guidance should be developed for the synthesis of implementation information within reviews as well as the documentation of applicability judgments by review authors.

Introduction

What is new?

Key findings

  1. •

    There are a range of tools available for appraising applicability of findings from systematic reviews; however, there is a lack of consensus regarding terminology and tool content. There are challenges in appraising applicability for systematic reviews of complex interventions and a lack of guidance on how to do this. Review authors can use three main approaches to address these challenges: (1) increased use of both qualitative and quantitative methods to explore causal pathways and assess variations in effect across important characteristics; (2) improved description of studies and their context; and (3) appraisal of applicability for at least one primary target population, setting, and context.

What this adds to what was known?
  1. •

    The study summarizes controversies in existing guidance for assessing applicability in systematic reviews.

  2. •

    It also provides recommendations for assessing applicability in systematic reviews of complex interventions using examples from existing reviews.

What is the implication and what should change now?
  1. •

    There is a need for further guidance to establish consensus on transparent reporting of applicability judgments, improved description of contextual factors, and assessment of causes for variations in effect.

Carefully assessing applicability is important in the use of all research, whether single studies or systematic reviews. Applicability can be considered in terms of individuals and their specific circumstances or can be extended to include populations and settings [1]. In the context of systematic reviews of effects, applicability can be understood as an assessment of whether the findings of a review can be applied in a particular context or population [2]. A number of related terms are used in the literature (e.g., generalizability, external validity, directness, transferability), and there is a lack of standardization in how these terms are used. We have summarized and defined the most widely used terms in Table 1.

There is increasing focus in systematic reviews on judging the applicability of results to a target (or intended) population and setting. For example, The Cochrane Collaboration now requires every review to include a section that discusses the applicability of the evidence and a summary of findings table, which is tailored to a specific population and setting. Although systematic reviews cannot possibly judge applicability to all possible settings and populations, the authors of systematic reviews are well placed to provide information needed for judging applicability, such as characteristics of included populations and settings, and an assessment of both relative and absolute effects.

Assessing the applicability of evidence may be challenging for users of systematic reviews of complex interventions. As described in this series [3] and elsewhere, the features that make an intervention more complex include number of interacting components within experimental and control interventions, number and difficulty of behaviors required by those delivering or receiving the intervention, number of groups or organizational levels targeted by the intervention, number and variability of outcomes, and the degree of flexibility or tailoring of the intervention permitted [4], [5]. Assessing applicability is more challenging as interventions become more complex because of greater diversity of, and interactions within and between, elements of the complex intervention and other characteristics such as the setting and context.

Numerous checklists and items have been developed for appraising applicability and other related concepts such as transferability. These checklists are mostly focused on attributes of patients, providers, and setting, with very little attention to characteristics of the intervention and intervention complexity.

In this article, we summarize the issues faced in assessing the applicability of research findings from systematic reviews of the effects of complex interventions, assess existing guidance on applicability, provide additional guidance for systematic review authors in documenting relevant considerations to inform applicability assessments, and make suggestions regarding where efforts to develop additional guidance could be directed.

Section snippets

What are the issues in assessing applicability for reviews of complex interventions?

Potential sources of complexity in interventions have been published elsewhere [3]. For a complex intervention (or a complex review question), judgments about applicability will often require consideration of the diversity of, and interactions within and between, population, intervention, comparison, and outcome (PICO) elements. Moreover, there are often important differences between the contexts in which the study(ies) were conducted and the context within which the findings of the review

What existing guidance is there?

As described earlier, The Cochrane Collaboration has mandated that all new reviews include “summary of findings” tables that explicitly present the main findings and discuss applicability of the review as they relate to one major setting and population of interest. This encourages systematic reviews to consider applicability for at least one specific setting and population. Additional settings and populations can also be considered; however, one combination must be chosen to be of primary

Where are there controversies or unresolved issues in existing guidance?

Despite widespread acknowledgment that applicability considerations are important, it is only more recently, with increased focus on knowledge translation, that applicability considerations are receiving greater emphasis. Existing reporting guidelines for systematic reviews lack detail about how to transparently report judgments about applicability. There is a lack of consensus on tools for judging applicability (Table 3 for a sample of tools) and in the use of terms to describe applicability

What should systematic review authors do now?

Based on our assessment of challenges and existing guidance, we recommend that review authors assist in considering applicability in three ways: (1) plan and conduct the relevant analyses to answer questions about mechanisms of action, the influence of different settings, contexts and populations on effects and provide details relevant to implementation and monitoring; (2) collect and report transparently information that users of systematic reviews can use to make judgments about applicability

Where is additional guidance needed?

Guidance and methodological studies would be helpful in the following areas:

  • •

    Guidance on how to develop an “implementation table” and synthesize implementation information reported in or alongside evaluations of intervention effects, to assist authors to incorporate expected intervention mechanisms, consider the contribution of various intervention components, and align with review findings.

  • •

    To establish international and crossdisciplinary consensus regarding the range of terms used to describe

Conclusion

Assessment of applicability is an essential part of the systematic review process and is particularly important if reviews of interventions are to inform practice, policy, and research. We propose that review authors can enhance the use of their research syntheses for policy and programmatic decisions by using analytic methods to explain variations in effect, providing rich description of the contexts of included studies, and appraising applicability for at least one target population, setting,

Acknowledgments

E.W. acknowledges financial support from the Jack Brockhoff Child Health and Wellbeing Program. E.R. acknowledges financial support received from the Munich Center of Health Sciences and as part of the European Commission-funded INTEGRATE-HTA project.

References (35)

  • F. Dwamena et al.

    Interventions for providers to promote a patient-centred approach in clinical consultations

    Cochrane Database Syst Rev

    (2012)
  • R. Gaitonde et al.

    Interventions to reduce corruption in the health sector (protocol)

    Cochrane Database Syst Rev

    (2010)
  • L. Cronbach et al.

    Designing evaluations of educational and social programs

    (1982)
  • E.A. Kristjansson et al.

    School feeding for improving the physical and psychosocial health of disadvantaged elementary school children

    Cochrane Database Syst Rev

    (2007)
  • S. Lewin et al.

    Lay health workers in primary and community health care for maternal and child health and the management of infectious diseases

    Cochrane Database Syst Rev

    (2010)
  • R. Turley et al.

    Slum upgrading strategies involving physical environment and infrastructure interventions and their effects on health and socio-economic outcomes

    Cochrane Database Syst Rev

    (2013)
  • Z.S. Lassi et al.

    Community-based intervention packages for reducing maternal and neonatal morbidity and mortality and improving neonatal outcomes

    Cochrane Database Syst Rev

    (2010)

    • Cited by (61)

    • Underreporting contextual factors preclude the applicability appraisal in primary care randomized controlled trials

      2023, Journal of Clinical Epidemiology

    • The prevalence and effects of treatments of rapidly progressive interstitial lung disease of dermatomyositis/polymyositis adults: A systematic review and meta-analysis

      2023, Autoimmunity Reviews

    • A Rapid Review of Interventions to Prevent First Pregnancy among Adolescents and Its Applicability to Latin America

      2021, Journal of Pediatric and Adolescent Gynecology

    • Redesigning the evidence synthesis in palliative care: the realist review

      2019, Medecine Palliative

    • Cochrane Qualitative and Implementation Methods Group guidance series—paper 1: introduction

      2018, Journal of Clinical Epidemiology

    • Ten recommendations for assessing the comparative effectiveness of therapeutic medical devices: a targeted review and adaptation

      2018, Journal of Clinical Epidemiology
      Citation Excerpt :

      Using this framework may help to better identify similar and different properties of health technologies and appropriate evaluation methods. We applied the FSRCIs [15,17,19–22] to TMDs throughout the steps of a SR and, therefore, could make use of its extensive groundwork and guidance to formulate recommendations that systematically allow for the specifics of CE assessment of TMDs. From the literature, we identified the following TMD-specific issues to consider.

    View all citing articles on Scopus

    Conflict of interest: B.B. is an editor for the Cochrane Public Health Group. She has no other conflicts of interest. S.L. is an editor for the Cochrane Effective Practice and Organisation of Care Group and the Cochrane Consumers and Communication Review Group. He has no other conflicts of interest. V.W. is coconvenor of the Campbell and Cochrane Equity Methods group. She has no other conflicts of interest. E.R. has no conflicts of interest. E.W. is the coordinating editor of the Public Health Group and has no other conflicts of interest.

    View full text
    Copyright © 2013 Elsevier Inc. All rights reserved.