Background
Methods
Eligibility criteria
An overview of reviews: | |
---|---|
1. Contains a clearly formulated objective designed to answer a specific research question, typically about a healthcare intervention. 2. Intends to search for and include only systematic reviews (with or without meta-analyses). 3. Uses explicit and reproducible methods to identify multiple systematic reviews that meet the overview of reviews’ inclusion criteria and assess the quality/risk of bias of these systematic reviews. 4. Intends to collect, analyze, and present the following data from included systematic reviews: descriptive characteristics of the systematic reviews and their included primary studies; risk of bias of primary studies; quantitative outcome data; and certainty of evidence for pre-defined, clinical important outcomes. 5. Discusses findings as they relate to the purpose, objective(s), and specific research question(s) of the overview of reviews, including: a summary of main results, overall completeness and applicability of evidence, quality of evidence, potential biases in the overview process, and agreements and/or disagreements with other studies and/or reviews. |
Searches
Document selection
Data extraction and synthesis
Results
Author team or research group | Documents containing explicit methodological guidance for conducting overviews (59 documents produced by 24 research groups) | Documents that describe authors teams’ experience conducting published overviews (20 documents produced by 16 research groups) | ||||
---|---|---|---|---|---|---|
N documents (reference #–Additional file 4) | Years of publication | Document formats | N documents (reference #–Additional file 4) | Years of publication | Document formats | |
Alberta Research Centre for Health Evidence (ARCHE) | 3 (A1-A3) | 2017–2019 | 2 journal articles, 1 associated thesis | 1 (A60) | 2019 | 1 journal article |
Belgian Health Care Knowledge Centre (KCE) | 1 (A4) | 2016 | 1 journal article | – | – | – |
Central South University (CSU) | 1 (A5) | 2020 | 1 journal article | – | – | – |
Cochrane Child Health Field (CHF) | 11 (A6–A16) | 2010–2015 | 8 conference presentations, 2 internal documents, 1 journal article | 2 (A61–A62) | 2011–2013 | 1 journal article, 1 conference presentation |
Cochrane Chile (CChile) | 1 (A17) | 2019 | 1 conference presentation | 1 (A17) | 2019 | 1 conference presentation |
Cochrane Comparing Multiple Interventions Methods Group (CMIMG) | 19a (A6, A13, A14, A18-A36) | 2008–2019 | 8 conference presentations, 6 internal documents, 2 websites, 1 handbook chapter, 1 journal article, 1 interview transcript | – | – | – |
Cochrane Consumers and Communication Review Group (CCRG) | – | – | – | 1 (A63) | 2009 | 1 journal article |
Cochrane Effective Practice and Organization of Care Group (EPOC) | 1 (A37) | 2011 | 1 conference presentation | 3 (A64–A66) | 2011–2019 | 3 conference presentations |
Cochrane Musculoskeletal Group (CM) | – | – | – | 1 (A67) | 2011 | 1 conference presentation |
Cochrane Public Health Group (CPHG) | 1 (A38) | 2014 | 1 journal article | – | – | – |
Cochrane Stroke Group (CSG) | – | – | – | 1 (A68) | 2015 | 1 conference presentation |
Duke University (DukeU) | 1 (A39) | 2012 | 1 journal article | – | – | – |
Dutch Cochrane Centre (DCC) | – | – | – | 1 (A69) | 2009 | 1 conference presentation |
Evidence for Policy and Practice Information and Co-ordinating Centre (EPPI) | 2 (A40–A41) | 2015 | 1 journal article, 1 conference presentation | – | – | – |
Glasgow Caledonian University (GCU) | 1 (A42) | 2019 | 1 journal article | 2 (A70–A71) | 2016–2017 | 2 journal articles |
Harvard University and the Cyprus University of Technology (HarvU) | 1 (A43) | 2019 | 1 journal article | – | – | – |
Joanna Briggs Institute Umbrella Reviews Methodology Group (JBI) | 3 (A44–A46) | 2013-2017 | 1 journal article, 1 internal document, 1 book chapter | – | – | – |
King’s College London (KCL) | 1 (A47) | 2018 | 1 journal article | 1 (A47) | 2018 | 1 journal article |
Ludwig Boltzmann Institute for Health Technology Assessment (LBI) | – | – | – | 1 (A72) | 2015 | 1 journal article |
Northeast Institute of Evidence Synthesis and Translation at Rutgers School of Nursing (NEST) | 1 (A48) | 2016 | 1 book chapter | – | – | – |
Norwegian Knowledge Centre for the Health Services (NOKC) | 1 (A49) | 2013 | 1 book chapter | – | – | – |
Pontifical Xavierian University (PXU) | – | – | – | 1 (A73) | 2011 | 1 conference presentation |
Robinson Research Institute, University of Adelaide (RRI) | – | – | – | 1 (A74) | 2016 | 1 conference presentation |
Sapienza University of Rome (SUR) | 1 (A50) | 2016 | 1 book section | – | – | – |
Trinity College Dublin (TCD) | 1 (A51) | 2011 | 1 journal article | – | – | – |
University of Auckland (UAuck) | – | – | – | 1 (A75) | 2019 | 1 conference presentation |
University of Birmingham (UBirm) | 1 (A52) | 2012 | 1 journal article | – | – | – |
University of Calgary (UCalg) | – | – | – | 1 (A76) | 2017 | 1 case report |
University of Connecticut (UConn) | 2 (A53–54) | 2019 | 2 journal articles | – | – | – |
University of Cyprus (UCyp) | 1 (A55) | 2020 | 1 journal article | – | – | – |
University of Dundee (UDun) | – | – | – | 1 (A77) | 2004 | 1 journal article |
University of Oxford (UOx) | 1 (A56) | 2017 | 1 journal article | – | – | – |
Western Journal of Nursing Research (WJNR) | 1 (A57) | 2014 | 1 editorial | – | – | – |
Witten/Herdecke University (WHU) | 2 (A58–59) | 2014 | 2 journal articles | – | – | – |
Guidance for conducting overviews of reviews
Topic | ARCHE | KCE | CSU | CChile | CHF | CMIMG | EPOC | CPHG | DukeU | EPPI | GCU | HarvU | JBI | KCL | NEST | NOKC | SUR | TCD | UBirm | UConn | UCyp | UOxf | WJNR | WHU | Frequency effect size (of 24) |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Guidance related to the context of conducting an overview of reviews | |||||||||||||||||||||||||
What types of questions can be answered using the overview format? | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 9 | |||||||||||||||
Deciding between conducting an overview and a systematic review | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 7 | |||||||||||||||||
Items to consider before deciding to conduct an overview | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 14 | ||||||||||
Author team composition and roles | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 8 | ||||||||||||||||
Target audience of the overview | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 13 | |||||||||||
Guidance related to the process of conducting an overview of reviews | |||||||||||||||||||||||||
Developing and registering an overview protocol | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 8 | ||||||||||||||||
Specifying the scope | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 13 | |||||||||||
Searching for systematic reviews (and potentially primary studies) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 18 | ||||||
Selecting systematic reviews for inclusion (and potentially primary studies) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 20 | ||||
Should an overview include non-Cochrane systematic reviews? | ✓ | ✓ | ✓ | ✓ | 4 | ||||||||||||||||||||
Assessing the quality of included systematic reviews | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 20 | ||||
Collecting and presenting data on descriptive characteristics of the included systematic reviews | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 11 | |||||||||||||
Collecting and presenting data on the quality of primary studies contained within the systematic reviews | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 10 | ||||||||||||||
Collecting, analyzing, and presenting outcome data | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 21 | |||
Assessing the certainty/quality of the body of evidence | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 13 | |||||||||||
Interpreting outcome data and drawing conclusions | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 8 | ||||||||||||||||
Updating the overview | ✓ | ✓ | ✓ | 3 | |||||||||||||||||||||
Intensity effect size (of 17) | 2 | 9 | 6 | 1 | 13 | 17 | 9 | 6 | 12 | 10 | 6 | 8 | 16 | 4 | 7 | 8 | 15 | 10 | 4 | 10 | 7 | 4 | 11 | 7 |
Types of questions that can be answered using the overview of reviews format
Choosing between conducting an overview of reviews and a systematic review
Items to consider before conducting an overview of reviews
Author team composition and roles
Target audience of the overview of reviews
Developing and registering an overview of reviews protocol
Specifying the scope of the overview of reviews
Searching for systematic reviews (and potentially primary studies)
Selecting systematic reviews for inclusion (and potentially primary studies)
Should an overview of reviews include non-Cochrane systematic reviews?
Assessing the quality or risk of bias of the included systematic reviews
Collecting and presenting data on descriptive characteristics of included systematic reviews (and their primary studies)
Collecting and presenting data on quality of primary studies contained within the included systematic reviews
Collecting, analyzing, and presenting outcome data
Assessing the certainty/quality of the body of evidence
Interpreting outcome data and drawing conclusions
Updating the overview
Challenges
Topic | Number of groups reporting challenges | Summary of reported challenges |
---|---|---|
Challenges related to the context for conducting overviews (i.e., when and why should you conduct an overview? | ||
Choosing between conducting an overview and a systematic review | 2 (CHF, CMIMG) | It is not clear how to decide when it is better to perform an intervention systematic review versus an overview of reviews. It can be difficult to compare multiple interventions in the overview format, and it is often not feasible or appropriate to conduct a network-meta-analysis within an overview of reviews. |
What types of questions about healthcare interventions can be answered using the overview format? | 2 (CCG, CMG) | Methodological approaches may differ depending on the type of question that the overview of reviews aims to answer. |
Questions to consider before deciding to conduct an overview | 8 (CCRG, CHF, CMIMG, EPOC, JBI, UBirm, UCalg, UDun) | Overviews of reviews can be time consuming to produce, so there is a need to think about time and resource limitations and the need to balance flexibility with rigor. Authors need to think about the coverage and up-to-dateness of the available systematic reviews and decide whether an overview of reviews should be conducted if key primary studies or important interventions are missing from the available systematic reviews. Authors need to think about whether it would be feasible within time and resource constraints to update any systematic reviews that are out of date. |
Author team composition and roles | 2 (CMIMG, UCalg) | Authors are challenged with determining the size, composition, and skillset of the team members. A larger team than originally thought might be needed when individual contributors are limited in time. |
Target audience of the overview | 1 (CCG) | Approaches to preparing the overview of reviews may need to be adapted depending on the intended audience. |
Challenges related to the process of conducting overviews (i.e., how do you conduct an overview?) | ||
Specifying the scope | 9 (CHF, DCC, EPPI, LBI, RRI, UCalg, UBirm, UDun, WJNR) | Defining the scope and selecting and prioritizing populations and outcomes of interest can be difficult. The scope of available systematic reviews may be broader or narrower than the scope of the overview of reviews, and the available systematic reviews might not present data that are most relevant to the objective of the overview of reviews. When the scope is broad but time is limited, important outcomes might need to be prioritized. |
Searching for systematic reviews | 7 (CHF, CPHG, DukeU, EPOC, LBI, UBirm, UCalg) | Developing searches and deciding which index terms to use, which sources to search, and what restrictions should be placed on the search (e.g., language, date) can be challenging and need to be well thought out to avoid missing important systematic reviews. There is debate about the need to also search for primary studies that are not contained in any included systematic reviews, or when searches should be updated to find new primary studies. This adds complexity to the search. |
Selecting systematic reviews for inclusion | 15 (ARCHE, CHF, CMIMG, CSG, CSU, DukeU, EPOC, EPPI, GCU, JBI, RRI, UCalg, UBirm, UDun, WHU) | There are many decision points in selecting systematic reviews for inclusion that can be challenging and time-consuming. Authors need to decide how to define a ‘systematic review’, for which there is no single agreed upon definition. Authors then need to plan how they will handle systematic reviews that are out of date. They can update these themselves, add relevant primary studies, or concede that the findings of recent trials will be omitted. This can be a trade-off between amount and quality of evidence included. Finally, authors are challenged with identifying and handling primary study overlap at the selection level when many overlapping systematic reviews may exist. This can be time-intensive and challenging because of variable reporting across the available systematic reviews (e.g., may not transparently report all associated publications, may include different arms of the same trials). |
Should an overview include non-Cochrane systematic reviews | 3 (ARCHE, CHF, CMIMG) | The decision about whether to only include Cochrane systematic reviews or to also include non-Cochrane systematic reviews can be a balance between ensuring quality and coverage of all important interventions. Though non-Cochrane reviews can be of poorer methodological quality and have less detailed reporting, Cochrane reviews alone may not cover all relevant interventions or be adequately up to date. If authors choose to include both Cochrane and non-Cochrane systematic reviews, it is likely that they will need to deal with primary study overlap. However, this may occur even if only Cochrane systematic reviews are included. |
Assessing the quality of included systematic reviews | 10 (ARCHE, CCRG, CHF, CMIMG, EPPI, EPOC, GCU, PXU, RRI, UDun) | There is no agreement on which tool might be best to use (e.g., AMSTAR, AMSTAR 2, or ROBIS) to assess methodological quality, or how to use them in the context of an overview of reviews. It can be difficult to distinguish between methodological quality and the quality of reporting, and poor reporting in the systematic reviews can make assessment challenging. Authors often have difficulty interpreting and coming to agreement with assessments on the available tools. It is unclear whether authors should assess systematic reviews in their entirety or only the components that are relevant to the overview question, and what to do with systematic reviews that include other embedded reviews. When overview quality is being used to choose between overlapping systematic reviews, authors need to be careful to not exclude potentially relevant information. When overlapping systematic reviews use different methodologies and come to discordant conclusions, it can be hard to tell whether their methods are appropriate. |
Collecting and presenting data on descriptive characteristics of included systematic reviews (and primary studies) | 11 (CCRG, CHF, CMIMG, CMG, DCC, DukeU, EPOC, JBI, LBI, NOKC, UCalg) | Overview authors are challenged with data extraction at two levels, first the level of the systematic review, and then potentially the level of the primary study. When relying on the reporting of the included systematic reviews, authors may struggle when these are poorly reported and missing important details. Overview authors need to carefully check systematic reviews for errors in data extraction, as these errors will lead to errors in the overview of reviews. They also need to decide how to deal with systematic reviews with missing information of relevance to the overview of reviews. Going back to the primary studies can be time consuming, but not doing so can lead to a loss of information. |
Collecting and presenting data on quality of primary studies contained within included systematic reviews | 7 (CCRG, CHF, CSG, DCC, EPOC, EPPI, JBI) | Collecting and presenting information on the quality of the primary studies can mean relying on the appraisals of the original systematic review authors, which may be flawed, inconsistent, or poorly reported. Some systematic reviews may only report a summary of appraisals, rather than the risk of bias or quality of individual studies or outcomes of interest. Comparisons across systematic reviews can be difficult if different tools are used in each, because using different methods of assessing risk of bias can lead to disparate judgments. |
Collecting, analyzing, and presenting outcome data | 23 (ARCHE, CChile, CCRG, CMG, CMIMG, CHF, DCC, DukeU, EPOC, EPPI, JBI, KCL, LBI, NOKC, PXU, RRI, TCD, UAuck, UBirm, UCalg, UDun, WHU, WJNR) | Many difficulties may arise when collecting, analyzing, and presenting findings at the overview level, because of inconsistency in methodology and reporting of findings across systematic reviews. For example, the included systematic reviews and their primary studies may use heterogeneous outcome measures. Additionally, the included systematic reviews may be incompletely reported, or may not report data on subgroups of interest. Overlapping systematic reviews might present discordant results or present similar data in different ways (e.g., different summary measures), and it can be complex and time-consuming to ensure that data from single studies are not over-represented. Interpretation of measures of overlap (e.g., matrices and corrected covered area) can be a challenge when the number of primary studies is large. To perform analyses of interest, overview authors might need to go back to individual studies, or concede that the available information is incomplete. It may not always be appropriate or feasible to conduct meta-analyses in overviews, and network meta-analyses and informal indirect comparisons are usually not appropriate. However, narrative synthesis can become complex and open to bias if not adequately described. There is a concern that synthesis errors at the SR level could result in errors at the overview level. |
Assessing quality of evidence of outcome data | 11 (CCRG, CHF, CMG, CMIMG, CSG, DCC, EPOC, Glasgow, PXU, RRI, UDun) | It may not be possible or appropriate to simply extract existing GRADE appraisals from the included systematic reviews. The reviews might not include GRADE appraisals for the outcomes or populations of interest or be missing details on each of the GRADE considerations. Different systematic reviews with the same studies that have made different decisions about handling data (analysis) and appraising study quality may come to different GRADE conclusions, especially related to the study limitations, consistency, and precision domains. Different raters across systematic reviews could come to different conclusions, due to the subjectivity of the GRADE approach. If re-doing the GRADE for each systematic review, authors are likely to encounter difficulty due to an absence of guidance on how to apply GRADE in the context of an overview, incomplete reporting at the level of the systematic review, and a lack of familiarity with the contributing primary studies. |
Interpreting outcome data and drawing conclusions | 9 (CHF, CMIMG, DCC, DukeU, EPOC, GCU, LBI, URRI, UCalg) | Interpreting data and drawing conclusions can be difficult. The included systematic reviews (and their included primary studies) may use heterogeneous outcome measures which can limit the ability to draw useful conclusions. Procedural variation at the systematic review and overview levels (e.g., study selection, data extraction) can lead to different conclusions from the same set of data. It can be difficult to provide interpretation of analyses of multiple interventions; multiple comparisons from different systematic reviews that are included in the same overview; discordant results and conclusions across the included systematic reviews. Authors need to consider the methods used in the systematic reviews and overview, and decide how best to highlight uncertainties and gaps that remain. |