Background
Overviews of reviews of healthcare interventions integrate information from multiple similar systematic reviews (SRs) to provide a single synthesis of relevant evidence for healthcare decision-making [
1]. As the number of published SRs continues to increase [
2], it is becoming more common for overview authors to identify multiple relevant SRs that address the same (or very similar) clinical questions and that include many (but not necessarily all) of the same primary studies [
3,
4]. Including these “overlapping SRs” in overviews can pose methodological challenges for researchers [
3‐
5]. It can be difficult to appropriately extract and analyze outcome data from overlapping SRs that have variable conduct, quality, and reporting. Further, bias may be introduced into the overview if outcome data from primary studies included in multiple SRs contribute to the analyses more than once.
Different researchers use different approaches to manage overlapping SRs in overviews [
3,
4]. Some researchers search for and include all overlapping SRs, but try to extract each primary study’s outcome data only one time, regardless of how many SRs include that primary study [
6,
7]. Other researchers attempt to avoid overlap by searching for and including only Cochrane SRs, as Cochrane attempts to avoid duplication by publishing only one SR per topic of interest [
1,
8]. Yet other researchers search for all overlapping SRs, but avoid overlap by prioritizing inclusion of only one SR per group of overlapping SRs (e.g., by including only the Cochrane, most recent, or highest quality SR) [
6].
In a companion paper by Pollock et al., we examined the impact of the above inclusion decisions on the comprehensiveness and results of overviews, and we documented the challenges associated with these different inclusion decisions [
9]. We found that each inclusion decision presented its own unique advantages, disadvantages, and trade-offs, with respect to comprehensiveness of the overviews and challenges encountered. The purpose of the current study was to practically use the above-described study results to develop an evidence-based decision tool to help researchers make informed inclusion decisions when conducting overviews of reviews of healthcare interventions.
Methods
We used a two-stage process to develop the decision tool. First, we conducted a multiple case study to obtain the empirical evidence upon which the tool is based. We systematically conducted seven overviews five times each, making five different decisions about which SRs to include in the overviews, for a total of 35 overviews. We then examined the impact of the inclusion decisions on the overviews’ comprehensiveness, results, and challenges, both within and across the seven overview cases [
9]. Second, we used a structured and iterative process to transform the evidence obtained from the multiple case study into an empirically based decision tool with accompanying descriptive text. Both stages are described in detail below.
Multiple case study
To develop the decision tool, we first conducted a multiple case study [
10]. Each “case” was an overview of healthcare interventions conducted by the Alberta Research Centre for Health Evidence between 2010 and 2016 that examined the efficacy or effectiveness of multiple healthcare interventions for preventing or treating a clinical condition in pediatrics. The seven cases were acute asthma [
11], acute otitis media [
12], bronchiolitis [
13], croup [
14], eczema [
15], gastroenteritis [
16], and procedural sedation [
17]. Each of the seven cases were conducted using five different inclusion scenarios (described below), for a total of 35 overviews of healthcare interventions.
For each overview, all published, English-language Cochrane and non-Cochrane SRs that met the overview’s inclusion criteria were identified and included. The seven overviews were then conducted using five different inclusion criteria, for a total of 35 overviews. For the
full inclusion scenario, all eligible outcome data were extracted from all eligible SRs; accuracy of effect estimates was ensured by extracting each primary study’s outcome data only once, regardless of how many SRs contained that study’s data. For the
first restricted scenario, all eligible outcome data were extracted from only the Cochrane SRs. For the
last three restricted scenarios, all eligible outcome data were extracted from all non-overlapping SRs, and for groups of overlapping intervention comparisons within SRs, outcome data were extracted from the following: Cochrane SR (if available), most recent SR (by publication date or search date), or highest quality SR (assessed by two independent reviewers using the AMSTAR tool [
18]). Data extraction and analysis for the 35 overviews was conducted by one reviewer using standard methods [
19]. For each overview, we extracted descriptive data (for SRs, and their included primary studies) and outcome data (for all relevant outcomes specified in the corresponding overviews). Outcome data were classified using published criteria as “favorable,” “neutral,” “unfavorable,” or “unknown” [
9,
20,
21]. We also documented the challenges we encountered related to including overlapping SRs in overviews.
For each overview topic, we described the characteristics of the
full inclusion scenario. We then systematically compared the
full inclusion scenario with each of the four
restricted inclusion scenarios by documenting the number and percentage of SRs, intervention comparisons, primary studies, and subjects that were lost in each restricted scenario and the number and percentage of outcomes that were lost or changed (from favorable, neutral, unfavorable, or unknown, to something else) in each restricted scenario. We then identified and narratively described groups of similar and contrasting overviews [
10,
22] and narratively summarized the challenges we encountered when conducting the different inclusion scenarios. The full methods and results are reported in the companion paper by Pollock et al. [
9].
We used a structured and iterative process to transform the results of the multiple case study into an evidence-based decision tool, with accompanying descriptive text, to help researchers make inclusion decisions in overviews. We developed the tool using established principles for designing and populating network displays. This involved reconfiguring the inclusion decisions and study results into an ordered decision model with accompanying descriptive text [
22].
In the multiple case study, we found that certain variables differentially affected the comprehensiveness and results of overviews [
9]. When creating the decision tool, we transformed those variables into yes/no questions, provided guidance to help researchers answer the questions, and structured the tool so that different responses to the questions corresponded to appropriate inclusion decisions. Thus, the decision tool provides a visual display of the decision points and potential decision pathways available to overview authors. In the multiple case study, we also documented the impact of the inclusion decisions on the comprehensiveness and results of overviews and the challenges we encountered when conducting the different inclusion scenarios [
9]. We transformed these results into empirically derived descriptive text explaining the impact, advantages, disadvantages, and potential trade-offs of the different inclusion decisions. Thus, the descriptive text can help authors systematically consider the potential implications of making different inclusion decisions in overviews.
The decision tool was developed iteratively by the same team of researchers who conducted the multiple case study. Thus, all researchers had content and methodological expertise and were familiar with the multiple case study findings. One researcher developed, refined, and finalized the decision tool and its accompanying descriptive text using the above-described process (MP). The additional four researchers oversaw this process and provided several rounds of feedback on the content, placement, and wording of the decision points and pathways in the decision tool, and the content and wording of the descriptive text (RMF, ASN, SDS, LH). All researchers approved the final version of the tool and agreed that it comprehensively reflected the results of the multiple case study. Taken together, the decision tool and its accompanying text can provide overview authors with both the knowledge and the means to make informed inclusion decisions that are specific to the context of their unique overview.
Discussion
In a multiple case study of seven overviews conducted using five different inclusion decisions, we found that different inclusion decisions differentially affected the comprehensiveness and results of overviews and were associated with different types of challenges [
9]. We used these findings to systematically develop an evidence-based decision tool to help researchers make informed inclusion decisions in overviews. Using an evidence-based tool to make inclusion decisions can help promote transparency and rigor and decrease bias. We hope this tool will be a useful resource for researchers conducting overviews, and we welcome discussion, testing, and refinement of the proposed tool.
There are practical considerations involved when using the decision tool in overviews. Specifically, four conditions should be met prior to its use. First, the overview should examine multiple interventions for preventing or treating a health condition, though further testing and real-life application of the tool will help determine whether it can be used with other types of overviews [
1]. Second, the overview format should be more appropriate than the SR format to answer the research question [
3]. Third, researchers should intend to search for and include only SRs in the overview [
1,
3]. Fourth, researchers should be prepared and able to avoid double-counting outcome data from overlapping SRs, either by not including overlapping SRs in the overview or by ensuring that each primary study’s outcome data are extracted from overlapping SRs only once [
3‐
5]. The layout of the tool is based on two additional considerations. First, we acknowledge that researchers may decide to prioritize Cochrane SRs for inclusion in overviews due to both their higher methodological rigor (on average) [
2] and the additional time, skills, resources, and challenges associated with searching for, including, and extracting data from non-Cochrane SRs [
1,
3]. The tool’s layout ensures that researchers consider the potential implications of this inclusion decision upfront, in the context of their specific overview question. Second, we anticipate that some questions in the decision tool may be difficult to answer. To address this, we identified two points in the tool where researchers may wish to gather more information to help inform their decision.
This decision tool can provide practical guidance and support for overview authors by helping them make transparent and informed inclusion decisions when faced with overlapping SRs in overviews. As the number of overlapping (and possibly conflicting) SRs continues to increase [
23], issues surrounding the inclusion of overlapping SRs in overviews will likely become more complex. This decision tool may help address this complexity. It can help authors consider questions that can affect the nature and extent of outcome data included and not included in overviews, as well as the impact, advantages, disadvantages, and potential trade-offs of making different inclusion decisions in overviews. The decision tool may also help researchers structure the decision-making process and appropriately contextualize their overview findings in light of the potential completeness of the overview’s evidence base. Though we did not examine the impact of the different inclusion decisions on overview quality, as there is currently no accepted tool to assess quality of overviews, we hope that using the decision tool can increase the transparency and reproducibility of the methods used and decisions made. Importantly, the decision tool is not designed to tell researchers what to do. Instead, it is designed to provide researchers with both the knowledge and the means to make their own informed inclusion decisions. As such, each author team must consider the unique characteristics of their overview, including its purpose, scope, target audience, and resource availability, when determining the most appropriate course of action.
The inclusion decisions presented in the decision tool and examined in the companion paper by Pollock et al. [
9] are described in the literature as common ways to manage overlapping SRs in overviews while avoiding issues related to double-counting primary studies’ outcome data [
3,
4,
8]. However, researchers may sometimes make other inclusion decisions motivated by reasons other than considerations about overlapping SRs and double-counting data. For example, researchers that suspect that Cochrane SRs are comprehensive may still opt to search for and potentially include non-Cochrane SRs in the overview, while those that suspect that Cochrane SRs are not comprehensive may opt to include only Cochrane SRs and discuss this as a study limitation. Other researchers who are unable to avoid double-counting outcome data may still opt to include all Cochrane and non-Cochrane SRs and discuss this as a study limitation. Yet other researchers may use results of quality assessments as an exclusion criterion. They may also manage groups of overlapping SRs by choosing to include the “most comprehensive SRs” or the “most relevant SRs” (though these subjective assessments may be operationalized in different ways depending on the overview topic and thus were not examined in this methods study). Though outside the scope of the current study, research investigating the implications of these other inclusion decisions in overviews is needed. Further, the seven overviews used in the companion paper by Pollock et al. [
9] all examined interventions in pediatric health, and future research and practical application of the decision tool is also needed to help determine the utility of the tool in a broader sample of overviews of healthcare interventions. Lastly, though we developed the decision tool for use in overviews, future research and practical application of the tool may also find it to be useful for researchers conducting other knowledge syntheses that include SRs.
Conclusions
Different researchers currently make different inclusion decisions in overviews, but may not understand the ways in which these inclusion decisions affect the comprehensiveness and complexity of their overviews. This study used empirical findings to develop an evidence-based decision tool to help researchers make transparent and informed inclusion decisions in overviews. The tool can provide practical guidance and support for overview authors in two ways: by helping them consider questions that can affect the comprehensiveness and complexity of their overviews and by helping them assess the potential impact, advantages, disadvantages, and trade-offs of the different inclusion decisions. The tool may be useful when developing overview protocols and/or when conducting and reporting overviews. This tool would benefit from additional discussion, testing, and refinement. It may also have relevance to other knowledge synthesis products that include SRs.