Toward a comprehensive evidence map of overview of systematic review methods: paper 1—purpose, eligibility, search and data extraction

We retrieved 1850 records through searching databases and methods collections. A further 1384 records were identified through other sources (methods articles identified as part of a related research project [5], reference checking, and forward citation searching). After removal of duplicate records, 1179 records remained (Fig. 3). From screening titles and abstracts, we excluded 1092 records that were ineligible. We assessed 87 full-text reports for eligibility and excluded 21, with reasons noted in Additional file 3. Of the remaining 66, 42 were included in stage I and 24 in stage II.

Our purposive search strategy (dated May 2016) to identify studies evaluating search filters for the retrieval of SRs resulted in the inclusion of three more stage II studies (see Fig. 4 for details), bringing the total number of methods evaluations to 27.

Of the 42 stage I and 27 stage II studies, 29 and 15, respectively, pertained to one or more of the four initial steps in conducting an overview and so are included in this first paper; the remainder will be included in our second companion paper. All 15 stage II studies were evaluations of search filter studies for retrieval of SRs.

We first describe the characteristics of the included articles (see ‘Characteristics of included articles’; Table 2), followed by presentation of the developed methods framework. This presentation is organised into sections representing the broad steps of conducting an overview (sections ‘Specification of purpose, objectives and scope’, ‘Specification of eligibility criteria’, ‘Search methods’ and ‘Data extraction’; Tables 3, 4, 5 and 6). In each section, we orient readers to the structure of the methods framework, which includes a set of steps and sub-steps (e.g. under ‘Search methods’, the steps are ‘plan the sources to search’, ‘plan the search strategy for retrieval of SRs’, and ‘plan how primary studies will be retrieved’). Components within the tables are referred to using labels and numbers (e.g. 2.1.3). We highlight methods/approaches to deal with commonly encountered scenarios for which overview authors need to plan (see ‘Addressing common scenarios unique to overviews’; Table 7). Our description is focused on methods/options that are distinct, have added complexity, compared with SRs of primary studies, or have been proposed to deal with major challenges in undertaking an overview. Importantly, the methods/approaches and options reflect the ideas presented in the literature and should not be interpreted as endorsement for the use of the methods. Reporting considerations for all steps are reported in Additional file 4.

We found no studies that had evaluated methods in the steps of the framework for ‘specification of purpose, objectives and scope’, ‘specification of eligibility criteria’ and ‘data extraction’. Fifteen studies, published between 1998 and 2016, evaluated search filters for the retrieval of SRs (Table 8). One study [12], evaluated the performance of seven bibliographic databases to determine their coverage of SRs. This evaluation mapped to the option ‘select the types of databases to search’ (1.1.1) of the ‘search methods’ step of the framework (Table 5). Of the remaining 14 studies, two compared the performance of multiple published filters [13, 14], four developed new search filters and compared their performance against other published filters [15‐18], and eight developed and evaluated new search filters (but without comparison with other published filters) [19‐26]. These evaluations mapped to the option ‘select a published SR filter’ (2.1.1) of the ‘search methods’ step of the framework (Table 5).

The filters were designed to retrieve SRs across a range of databases (CINAHL, DARE, EMBASE, PsycINFO, Epistemonikos, MEDLINE, Simple Web Indexing System for Humans (SWISH) and TRIP). Seven studies developed the gold standard by handsearching journals, three used a combination of handsearching journals and database searches and five used only database searches. The performance measures used included sensitivity/recall, specificity, precision, accuracy and the number needed to read. In terms of risk of bias, none of the evaluation studies referred to a study protocol or noted the existence of one, and only three validated their search filter on a data set distinct from the derivation set [13, 16, 17].

Our analysis aligns with findings of other recent reviews in identifying important gaps in guidance on the conduct of overviews [27, 28]. These gaps include patchy coverage of methods, wherein guidance covers selected options but not alternatives and insufficient description to operationalise many methods (Table 2). While others have concluded there is a lack of consensus over many methods [28], overviews serve many purposes, and different approaches are needed for different purposes. Recognising this, the framework attempts to capture the spectrum of options available to overview authors, providing a tool for systematic consideration of alternative approaches. We highlight the scenarios for which overview authors need to plan and identify methods proposed to tackle each scenario (Table 7). While these contributions help to address the patchy coverage of methods, the framework cannot address the lack of operational detail in current guidance. A forthcoming update of the Cochrane Handbook should help [28], but other guidance will be needed to cover the many methods not applicable to Cochrane overviews. For authors writing guidance, the framework could serve as a checklist to ensure comprehensive coverage of the methods proposed in the literature.

The lack of evaluation studies identified in stage II indicates that there is limited evidence to inform methods decision-making in overviews. For each of the steps in the framework, there is often a range of different methods to use, which could conceivably impact on the results and conclusions of the overview, their utility for decision-makers, and the time/resources required to complete the overview. This lack of evaluation of methods means [28] there may be inappropriate variability in the methods employed across overviews (as has been observed [6]). Further, overviews that seek to address the same research question, but which are undertaken using different methods, may reach discordant conclusions.

The framework may be useful to researchers conducting overviews and methodologists. As highlighted above, the framework is useful for making explicit the decisions overview authors need to make when planning an overview. Using the framework as a checklist to plan methods for dealing with common scenarios should lessen the challenges that arise when conducting an overview. Using the framework during protocol development may also lead to less post-hoc decision-making that can arise from not being aware of the decisions that need to be made before commencing the overview. Less post hoc decision-making may limit potential bias in the process of undertaking the overview. For overview methodologists, the provision of comparative options for each of the steps of the framework facilitates identification (and prioritisation) of methods evaluations that might be undertaken. For example, examining those steps where selection of a different option is hypothesised to importantly impact on the results and conclusions of the overview (discussed below under ‘Future research to refine and populate the framework and evidence map’).

To our knowledge, this is the first attempt to create a comprehensive framework of the many methods proposed for use in overviews. It is also the first study, of which we are aware, that has used evidence mapping in the context of methods research. A protocol of this investigation has been published [10] and any post hoc decisions have been documented. During our analysis, we developed an organising structure to group-related methods and used consistent language to synthesise the varied descriptions encountered in the literature. We also made inferences to ensure that where a clear alternative to a described method existed, it was captured in the framework. Both steps helped generate a more uniform and complete inventory of methods than would have been possible through simply collating methods as described.

Methods studies related to overviews are challenging to find other than in specialist methodology registers, such as the Cochrane Methodology Register and the Meth4ReSyn library, meaning that some methods articles may have been missed. We conducted reference checking and forward citation searching in three databases to minimise the number of missed articles. Further, we focused our search on locating articles that used the term ‘overview’ (or related terminology). However, methods that may be applicable to overviews, such as those used in clinical practice guidelines, may not have been located. We did not broaden our search, or specifically examine guidance documents for producing guidelines, to keep the project containable. Our analysis involved piecing together information spread across multiple sources, and ‘translating’ varied descriptions of methods into a common language. This process, and the many decisions involved in structuring our framework required considerable judgement. While the process led to a more complete and uniform description of methods than we identified in any other source, the subjective nature of this analysis means that other researchers may have made different decisions.

Future research will involve seeking input on the framework from methodologists and researchers conducting overviews in terms of their face validity, that is, the structure of the framework and the comprehensiveness of the steps and identified methods. Hence, the framework will likely be refined and evolve over time. Further, as methods for overviews are evaluated, the evidence map can be further populated. While there is currently too little methods evaluation for a visual representation (or map) of the evidence to be useful, the framework provides the structure for creating this map. Some priority areas requiring evaluation, which we encourage methodologists to consider, relate to the decisions around eligibility and data extraction. For example, what is the effect of selecting one SR from multiple SRs addressing the same topic versus including all SRs? Outcomes of interest may include proximal measures, such as whether eligible primary studies or important data are missed. More distal measures include time taken to complete the overview, utility for decision-makers, and whether the findings and conclusions of the review change. Additionally, researchers could examine whether observed effects vary when different eligibility criteria are used to select one SR from a multiple. Similar questions can be posed about the effects of extracting data from one versus multiple SRs, from primary studies to SRs only, and so on. Evidence arising from these evaluations should lead to further refinement of the framework and, more importantly, empirical data about the trade-offs associated with alternative methodological approaches.