Use of existing systematic reviews for evidence assessments in infectious disease prevention: a comparative case study

During recent years, the number of published systematic reviews (SR) has increased nearly exponentially [1]. Therefore, scientists experience situations in which one or more SRs for their guideline topic already exist, leading to the question whether and how these reviews can be considered for guideline development. Moreover, limited resources and time constraints are important factors leading to considerations on the use of existing SRs, instead of performing new ones. This goes along with questions regarding the methodological quality of the existing reviews and their comprehensiveness, as well as the potential need for updating. Accordingly, the issue of using existing SRs for new reviews or guidelines has raised a number of methodological concerns [2].

Current evidence assessment frameworks try to provide guidance on how to integrate existing SRs into new reviews. The Project on a Framework for Rating Evidence in Public Health (PRECEPT), which has been initiated and funded by the European Centre for Disease Prevention and Control (ECDC), has developed methodological guidance for evaluating and grading evidence in public health, with a particular focus on infectious disease prevention and control [3]. PRECEPT explicitly recommends the use of existing SRs for evidence assessments, referring to the five-step methodology developed by the US Agency for Healthcare Research and Quality (AHRQ) and published by Robinson et al. [4]. Recently, additional guidance for this approach was published by the same group [5].

Case studies are needed to test whether this approach is useful and to identify challenges and workable solutions during the application process. Therefore, we used two recent evidence assessment projects conducted in our institute to test the applicability of the approach. These projects were related to two domains of studies comprised by PRECEPT, namely those on “risk factors” and “interventions.” In the first project, we aimed at assessing the role of neonatal sepsis as a risk factor for adverse neurodevelopmental outcome (case study A). This assessment was part of a project on the burden of healthcare-associated infections initiated and funded by ECDC. The second project comprised an assessment of the efficacy, effectiveness, and safety of influenza vaccination during pregnancy (case study B) which has been performed to support the decision-making process of the German Standing Committee on Vaccination (STIKO).

We performed a comparative case study on the use of SRs for the development of new SRs. For both case studies, systematic searches and analyses were performed by two teams of reviewers. For case study A (neonatal sepsis), SH and TH performed the searches, analyzed the data and finalized the SR, while TE acted as supervisor. For case study B (influenza vaccination), CR and TH performed the searches, analyzed the data, and finalized the SR, while OW acted as supervisor.

For the conduct of the SR of SRs, we used the approach proposed by Robinson et al. which has been incorporated in the PRECEPT framework [3]. This approach comprises five steps which guide the user through the process of identifying, analyzing, appraising, and using existing SRs. In step 1, it is suggested to locate the existing SRs, using a defined and reproducible approach. In step 2, it is proposed to assess the relevance of the identified existing reviews with regard to the new review/evidence assessment, by comparing the inclusion criteria to those of the new SR. In step 3, the assessment of the methodological quality of the identified SR(s) is suggested. Step 4 proposes four different but not mutually exclusive approaches to determine appropriate use and incorporation of existing SRs into new reviews. This includes

Step 5 suggests to report methods and results from existing SRs. The complete approach is illustrated in Fig. 1.

For each of the five steps of the approach, details on procedures performed for case studies A and B were documented. We considered challenges arising from the application of the approach for each step separately and how they were handled. Finally, a table was constructed that compares the major challenges experienced between both case studies A and B.

Major aim of our work described here was to test the application of the AHQR approach in the context of using systematic reviews for developing evidence-based recommendations in the field of infectious disease prevention and control. Importantly, this might also include research questions which do not come from the field of comparative effectiveness but include topics like risk factors or incidence. Nevertheless, we conclude that the approach is applicable to these scenarios and provides excellent guidance for the use of systematic reviews also in this context.

The use of existing SRs for the development of new SRs has been suggested to be an efficient way to develop evidence-based recommendations in public health, offering a time- and resource-saving alternative to the conduct of a new SR. Our comparative case study shows that applying the five-step approach by Robinson et al. [4] which is integrated in the PRECEPT evidence assessment framework [3] leads to a transparent evidence assessment based on existing SRs. However, depending on the quantity and quality of existing SRs for the given research question, different challenges have to be managed during the evidence assessment process. Furthermore, we experienced that in cases when extensive re-analysis was needed because the existing SRs did not report appropriate data from the primary studies, the advantages of using existing SRs over performing a new SR are likely to be small.

In our experience in conducting systematic reviews “from scratch,” 1/3 of time has to be spent for literature search, 1/3 for data extraction, and 1/3 for data analysis and interpretation of the data, respectively. We think that it is worth using existing reviews, if at least one of these three “steps” can be omitted. We estimated that a full systematic review takes six to 24 months, depending on complexity of question, number of relevant outcomes, relevant publication dates, etc. In our experience, by using existing systematic reviews instead of starting a new one, this process can be shortened by about one to two thirds, depending on topic, comprehensiveness, and quality of the existing reviews.

Overviews of SR are a new approach of evidence synthesis which becomes increasingly popular [18]. However, it has been recognized earlier that a number of methodological challenges exist related to these approaches [19]. So-called umbrella reviews can be considered to be a specific form of SRs of SRs by focusing on meta-synthesis of the results of comparable meta-analyses [20]. However, our case study illustrates that it might be of equal importance to develop methodological guidance for those cases where detailed comparison and dissection of the existing SRs rather than synthesis are needed. For the development of the PRECEPT framework, we decided to use the approach proposed by Robinson et al. [4] which has recently been further elaborated by the same group [5].

Our case studies A and B showed large differences regarding the evidence base. While case study A had to build on only one SR, for case study B four SRs were identified. At first look, these SRs appeared very similar regarding methodological quality and thematic relevance. However, further analysis revealed several challenges resulting from differences, e.g., in included studies, considered outcomes and risk of bias assessments. All reviews included in both case studies had very broad PICO questions, particularly with regard to the outcomes. The systematic reviews on influenza vaccination during pregnancy for example did not focus on one or two outcomes, but used groups of outcomes such as “adverse events during pregnancy” which were presented in an aggregated as well as non-aggregated way. Comparing the systematic reviews on influenza vaccination during pregnancy we realized that the authors transformed the data differently, e.g., regarding outcomes such as “fetal death,” “stillbirth,” “abortion,” or “intrauterine death.”

Interestingly, already 20 years ago, Jadad et al. discussed problems which might arise when SRs on the same topic come to discordant results or conclusions [21]. In our study, one particular challenge arose from the fact that the four SRs had partially overlapping studies. A recent study by Siontis et al. suggested that this problem is not uncommon. Of 73 meta-analyses analyzed in their article, 67 % had at least one other overlapping meta-analysis [22]. Siontis et al. proposed three possible solutions for such a scenario: (i) to select the most appropriate SR, (ii) to perform a new review, or (iii) to synthesize the results of the reviews. Similar challenges in analyzing and using existing SRs with overlapping topics have been recently described by Bolland and Grey [23]. Like in our study, these authors observed important differences in trial selection between seven meta-analyses on vitamin D supplements. Other researchers have developed graphical tools to assist in the assessment of appropriateness of each meta-analysis for a given research question [24]. In our study, we decided to apply the concept of comparative spreadsheets suggested by Whitlock et al. [2] to compare certain features of the included systematic reviews to assess their value for a new evidence synthesis. We thereby identified the most comprehensive review [16] and supplemented this study base with the results of the other three SRs. However, it has to be considered that our approach of selecting the most comprehensive review and supplementing it with studies identified in other reviews does not always work. The most comprehensive review may not be of the highest quality, it ignores the possibility that the selection criteria of the review authors were not entirely transparent, it does not eliminate the need for risk of bias assessment, and it still might require evidence assessments to be done. Furthermore, AMSTAR does not completely separate assessment of bias from reporting quality. It has to be investigated whether the recently developed ROBIS tool [25] performs better in capturing risk of bias in systematic reviews.

In step 4 of the approach, it is suggested to either use the results of the risk of bias assessments from the existing SRs for the new review or to perform a new assessment. In case study A, the latter approach was ultimately necessary since the existing review did not report risk of bias in the included studies. On the contrary, in case study B, all included SRs reported their own risk of bias assessments. However, these were based on four different risk of bias tools. In all SRs, assessments of RCTs were performed by using the Cochrane risk of bias tool [17], whereas for observational studies either the Newcastle Ottawa Scale [10], the Downs and Black Checklist [26], or the JBI-MASTARI tool [16] was used. The authors of these SRs came to differing conclusions regarding the methodological quality of the included original studies. Although a comparison of risk of bias assessments between the reviews would be interesting, reporting quality of these results showed large differences, making a direct comparison impossible: The systematic review by Fell et al. reported numerical quality scores per study, but not per outcome, using the Newcastle Ottawa Scale and the Downs-Black Scale. In the Methods section of the review by Galvao et al., it was declared that the Cochrane risk of bias tool was used, but the authors did not report the results of this assessment. In the systematic review by McMillan et al., the JBI-MASTARI was used, but the results of this assessment were reported in aggregated form only, rather than for individual studies and outcomes. However, one has to take into account that even the use of identical risk of bias tools can lead to different judgements on methodological quality by different authors, since it has been demonstrated that many risk of bias tools show a considerable degree of inter-rater variability [10, 27]. Taking all these uncertainties into account, we decided to perform a new risk of bias assessment for our review, and compared the results to those by the other investigators.

Remarkably, quality of the evidence was not assessed in the original systematic review by Alshaik et al. which served as the base for our case study A. Regarding the systematic reviews used in case study B, only one (Galvao et al.) of four reviews reported a GRADE quality assessment. However, due to differences in the number of included studies, it is difficult to compare the results of the assessment made by those authors to our GRADE assessments. Furthermore, updating of existing systematic review constitutes an important step which should be considered in every situation when the use of existing reviews is discussed as an approach in the development of evidence assessments. Importantly, methods have been developed to evaluate the need for updating, such as the surveillance system proposed by AHRQ [28].

Our comparative case study supports the applicability of the five-step approach [4] included in the PRECEPT framework for the use of existing SRs. However, efforts and challenges depend on the quality and quantity of the evidence in the SRs. Exclusion of non-systematic (or narrative) reviews constitutes a particular challenge since they can often not been identified without reading the full text of the respective article. Important progress in literature searches could be made if this kind of reviews would be adequately termed and indexed in electronic databases. Adherences of all SR authors to the PRISMA guideline [29] would strongly improve the quality of published reviews. Authors of systematic reviews should provide a database (e.g., an Endnote file) alongside with the paper to support replication, re-analysis, and updating of the respective systematic reviews. In situations where more than one SR has to be considered, the development of spreadsheets comparing characteristics, inclusion criteria, risk of bias, included studies, and outcomes can support the development of evidence-based guidance. For such situations, we used our experiences documented in this paper to develop a recommendation on how to proceed (Table 7). Furthermore, our study identified a number of open questions which should be addressed in future studies, such as

The use of a complete SR is often not possible and intensive re-analyses are necessary. Still, even when such reanalyses are needed, the use of existing SRs is a promising alternative to the conduct of a new SR and might support fast and efficient development of evidence-based recommendations in public health.