Defining the process to literature searching in systematic reviews: a literature review of guidance and supporting studies

Eight documents provided guidance on who should undertake literature searching in systematic reviews [2, 4, 6‐11]. The guidance affirms that people with relevant expertise of literature searching should ‘ideally’ be included within the review team [6]. Information specialists (or information scientists), librarians or trial search co-ordinators (TSCs) are indicated as appropriate researchers in six guidance documents [2, 7‐11].

The guidance is consistent with studies that call for the involvement of information specialists and librarians in systematic reviews [12‐26] and which demonstrate how their training as ‘expert searchers’ and ‘analysers and organisers of data’ can be put to good use [13] in a variety of roles [12, 16, 20, 21, 24‐26]. These arguments make sense in the context of the aims and purposes of literature searching in systematic reviews, explored below. The need for ‘thorough’ and ‘replicable’ literature searches was fundamental to the guidance and recurs in key stage two. Studies have found poor reporting, and a lack of replicable literature searches, to be a weakness in systematic reviews [17, 18, 27, 28] and they argue that involvement of information specialists/ librarians would be associated with better reporting and better quality literature searching. Indeed, Meert et al. [29] demonstrated that involving a librarian as a co-author to a systematic review correlated with a higher score in the literature searching component of a systematic review [29]. As ‘new styles’ of rapid and scoping reviews emerge, where decisions on how to search are more iterative and creative, a clear role is made here too [30].

Knowing where to search for studies was noted as important in the guidance, with no agreement as to the appropriate number of databases to be searched [2, 6]. Database (and resource selection more broadly) is acknowledged as a relevant key skill of information specialists and librarians [12, 15, 16, 31].

Whilst arguments for including information specialists and librarians in the process of systematic review might be considered self-evident, Koffel and Rethlefsen [31] have questioned if the necessary involvement is actually happening [31].

The aim: Five of the nine guidance documents use adjectives such as ‘thorough’, ‘comprehensive’, ‘transparent’ and ‘reproducible’ to define the aim of literature searching [6‐10]. Analogous phrases were present in a further three guidance documents, namely: ‘to identify the best available evidence’ [4] or ‘the aim of the literature search is not to retrieve everything. It is to retrieve everything of relevance’ [2] or ‘A systematic literature search aims to identify all publications relevant to the particular research question’ [3]. The Joanna Briggs Institute reviewers’ manual was the only guidance document where a clear statement on the aim of literature searching could not be identified. The purpose of literature searching was defined in three guidance documents, namely to minimise bias in the resultant review [6, 8, 10]. Accordingly, eight of nine documents clearly asserted that thorough and comprehensive literature searches are required as a potential mechanism for minimising bias.

Whilst the guidance calls for thorough and comprehensive literature searches, it lacks clarity on what constitutes a thorough and comprehensive literature search, beyond the implication that all of the literature search methods in Table 2 should be used to identify studies. Egger et al. [34], in an empirical study evaluating the importance of comprehensive literature searches for trials in systematic reviews, defined a comprehensive search for trials as:

Tricco et al. (2008) used a similar threshold of bibliographic database searching AND a supplementary search method in a review when examining the risk of bias in systematic reviews. Their criteria were: one database (limited using the Cochrane Highly Sensitive Search Strategy (HSSS)) and handsearching [35].

Together with the guidance, this would suggest that comprehensive literature searching requires the use of BOTH bibliographic database searching AND supplementary search methods.

Comprehensiveness in literature searching, in the sense of how much searching should be undertaken, remains unclear. Egger et al. recommend that ‘investigators should consider the type of literature search and degree of comprehension that is appropriate for the review in question, taking into account budget and time constraints’ [34]. This view tallies with the Cochrane Handbook, which stipulates clearly, that study identification should be undertaken ‘within resource limits’ [9]. This would suggest that the limitations to comprehension are recognised but it raises questions on how this is decided and reported [36].

Six documents provided guidance on preparing for a literature search [2, 3, 6, 7, 9, 10]. The Cochrane Handbook clearly stated that Cochrane authors (i.e. researchers) should seek advice from a trial search co-ordinator (i.e. a person with specific skills in literature searching) ‘before’ starting a literature search [9].

Two key tasks were perceptible in preparing for a literature searching [2, 6, 7, 10, 11]. First, to determine if there are any existing or on-going reviews, or if a new review is justified [6, 11]; and, secondly, to develop an initial literature search strategy to estimate the volume of relevant literature (and quality of a small sample of relevant studies [10]) and indicate the resources required for literature searching and the review of the studies that follows [7, 10].

Three documents summarised guidance on where to search to determine if a new review was justified [2, 6, 11]. These focused on searching databases of systematic reviews (The Cochrane Database of Systematic Reviews (CDSR) and the Database of Abstracts of Reviews of Effects (DARE)), institutional registries (including PROSPERO), and MEDLINE [6, 11]. It is worth noting, however, that as of 2015, DARE (and NHS EEDs) are no longer being updated and so the relevance of this (these) resource(s) will diminish over-time [64]. One guidance document, ‘Systematic reviews in the Social Sciences’, noted, however, that databases are not the only source of information and unpublished reports, conference proceeding and grey literature may also be required, depending on the nature of the review question [2].

Two documents reported clearly that this preparation (or ‘scoping’) exercise should be undertaken before the actual search strategy is developed [7, 10]).

The guidance offers the best available source on preparing the literature search with the published studies not typically reporting how their scoping informed the development of their search strategies nor how their search approaches were developed. Text mining has been proposed as a technique to develop search strategies in the scoping stages of a review although this work is still exploratory [65]. ‘Clustering documents’ and word frequency analysis have also been tested to identify search terms and studies for review [66, 67]. Preparing for literature searches and scoping constitutes an area for future research.

The Population, Intervention, Comparator, Outcome (PICO) structure was the commonly reported structure promoted to design a literature search strategy. Five documents suggested that the eligibility criteria or review question will determine which concepts of PICO will be populated to develop the search strategy [1, 4, 7‐9]. The NICE handbook promoted multiple structures, namely PICO, SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) and multi-stranded approaches [4].

With the exclusion of The Joanna Briggs Institute reviewers’ manual, the guidance offered detail on selecting key search terms, synonyms, Boolean language, selecting database indexing terms and combining search terms. The CEE handbook suggested that ‘search terms may be compiled with the help of the commissioning organisation and stakeholders’ [10].

The use of limits, such as language or date limits, were discussed in all documents [2‐4, 6‐11].

The guidance typically relates to reviews of intervention effectiveness so PICO – with its focus on intervention and comparator - is the dominant model used to structure literature search strategies [68]. PICOs – where the S denotes study design - is also commonly used in effectiveness reviews [6, 68]. As the NICE handbook notes, alternative models to structure literature search strategies have been developed and tested. Booth provides an overview on formulating questions for evidence based practice [69] and has developed a number of alternatives to the PICO structure, namely: BeHEMoTh (Behaviour of interest; Health context; Exclusions; Models or Theories) for use when systematically identifying theory [55]; SPICE (Setting, Perspective, Intervention, Comparison, Evaluation) for identification of social science and evaluation studies [69] and, working with Cooke and colleagues, SPIDER (Sample, Phenomenon of Interest, Design, Evaluation, Research type) [70]. SPIDER has been compared to PICO and PICOs in a study by Methley et al. [68].

The NICE handbook also suggests the use of multi-stranded approaches to developing literature search strategies [4]. Glanville developed this idea in a study by Whitting et al. [71] and a worked example of this approach is included in the development of a search filter by Cooper et al. [72].

Hausner et al. [73] provide guidance on writing literature search strategies, delineating between conceptually and objectively derived approaches. The conceptual approach, advocated by and explained in the guidance documents, relies on the expertise of the literature searcher to identify key search terms and then develop key terms to include synonyms and controlled syntax. Hausner and colleagues set out the objective approach [73] and describe what may be done to validate it [74].

The guidance documents offer direction on the use of limits within a literature search. Limits can be used to focus literature searching to specific study designs or by other markers (such as by date) which limits the number of studies returned by a literature search. The use of limits should be described and the implications explored [34] since limiting literature searching can introduce bias (explored above). Craven et al. have suggested the use of a supporting narrative to explain decisions made in the process of developing literature searches and this advice would usefully capture decisions on the use of search limits [75].

Table 2 summarises the process of literature searching as reported in each guidance document. Searching bibliographic databases was consistently reported as the ‘first step’ to literature searching in all nine guidance documents.

Three documents reported specific guidance on where to search, in each case specific to the type of review their guidance informed, and as a minimum requirement [4, 9, 11]. Seven of the key guidance documents suggest that the selection of bibliographic databases depends on the topic of review [2‐4, 6‐8, 10], with two documents noting the absence of an agreed standard on what constitutes an acceptable number of databases searched [2, 6].

The guidance documents summarise ‘how to’ search bibliographic databases in detail and this guidance is further contextualised above in terms of developing the search strategy. The documents provide guidance of selecting bibliographic databases, in some cases stating acceptable minima (i.e. The Cochrane Handbook states Cochrane CENTRAL, MEDLINE and EMBASE), and in other cases simply listing bibliographic database available to search. Studies have explored the value in searching specific bibliographic databases, with Wright et al. (2015) noting the contribution of CINAHL in identifying qualitative studies [76], Beckles et al. (2013) questioning the contribution of CINAHL to identifying clinical studies for guideline development [77], and Cooper et al. (2015) exploring the role of UK-focused bibliographic databases to identify UK-relevant studies [78]. The host of the database (e.g. OVID or ProQuest) has been shown to alter the search returns offered. Younger and Boddy [79] report differing search returns from the same database (AMED) but where the ‘host’ was different [79].

The average number of bibliographic database searched in systematic reviews has risen in the period 1994–2014 (from 1 to 4) [80] but there remains (as attested to by the guidance) no consensus on what constitutes an acceptable number of databases searched [48]. This is perhaps because thinking about the number of databases searched is the wrong question, researchers should be focused on which databases were searched and why, and which databases were not searched and why. The discussion should re-orientate to the differential value of sources but researchers need to think about how to report this in studies to allow findings to be generalised. Bethel (2017) has proposed ‘search summaries’, completed by the literature searcher, to record where included studies were identified, whether from database (and which databases specifically) or supplementary search methods [81]. Search summaries document both yield and accuracy of searches, which could prospectively inform resource use and decisions to search or not to search specific databases in topic areas. The prospective use of such data presupposes, however, that past searches are a potential predictor of future search performance (i.e. that each topic is to be considered representative and not unique). In offering a body of practice, this data would be of greater practicable use than current studies which are considered as little more than individual case studies [82‐90].

When to database search is another question posed in the literature. Beyer et al. [91] report that databases can be prioritised for literature searching which, whilst not addressing the question of which databases to search, may at least bring clarity as to which databases to search first [91]. Paradoxically, this links to studies that suggest PubMed should be searched in addition to MEDLINE (OVID interface) since this improves the currency of systematic reviews [92, 93]. Cooper et al. (2017) have tested the idea of database searching not as a primary search method (as suggested in the guidance) but as a supplementary search method in order to manage the volume of studies identified for an environmental effectiveness systematic review. Their case study compared the effectiveness of database searching versus a protocol using supplementary search methods and found that the latter identified more relevant studies for review than searching bibliographic databases [94].

Table 2 also summaries the process of literature searching which follows bibliographic database searching. As Table 2 sets out, guidance that supplementary literature search methods should be used in systematic reviews recurs across documents, but the order in which these methods are used, and the extent to which they are used, varies. We noted inconsistency in the labelling of supplementary search methods between guidance documents.

Rather than focus on the guidance on how to use the methods (which has been summarised in a recent review [95]), we focus on the aim or purpose of supplementary search methods.

The Cochrane Handbook reported that ‘efforts’ to identify unpublished studies should be made [9]. Four guidance documents [2, 3, 6, 9] acknowledged that searching beyond bibliographic databases was necessary since ‘databases are not the only source of literature’ [2]. Only one document reported any guidance on determining when to use supplementary methods. The IQWiG handbook reported that the use of handsearching (in their example) could be determined on a ‘case-by-case basis’ which implies that the use of these methods is optional rather than mandatory. This is in contrast to the guidance (above) on bibliographic database searching.

The issue for supplementary search methods is similar in many ways to the issue of searching bibliographic databases: demonstrating value. The purpose and contribution of supplementary search methods in systematic reviews is increasingly acknowledged [37, 61, 62, 96‐101] but understanding the value of the search methods to identify studies and data is unclear. In a recently published review, Cooper et al. (2017) reviewed the literature on supplementary search methods looking to determine the advantages, disadvantages and resource implications of using supplementary search methods [95]. This review also summarises the key guidance and empirical studies and seeks to address the question on when to use these search methods and when not to [95]. The guidance is limited in this regard and, as Table 2 demonstrates, offers conflicting advice on the order of searching, and the extent to which these search methods should be used in systematic reviews.

Five of the documents provided guidance on managing references, for example downloading, de-duplicating and managing the output of literature searches [2, 4, 6, 8, 10]. This guidance typically itemised available bibliographic management tools rather than offering guidance on how to use them specifically [2, 4, 6, 8]. The CEE handbook provided guidance on importing data where no direct export option is available (e.g. web-searching) [10].

The literature on using bibliographic management tools is not large relative to the number of ‘how to’ videos on platforms such as YouTube (see for example [102]). These YouTube videos confirm the overall lack of ‘how to’ guidance identified in this study and offer useful instruction on managing references. Bramer et al. set out methods for de-duplicating data and reviewing references in Endnote [103, 104] and Gall tests the direct search function within Endnote to access databases such as PubMed, finding a number of limitations [105]. Coar et al. and Ahmed et al. consider the role of the free-source tool, Zotero [106, 107]. Managing references is a key administrative function in the process of review particularly for documenting searches in PRISMA guidance.

The Cochrane Handbook was the only guidance document to recommend a specific reporting guideline: Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [9]. Six documents provided guidance on reporting the process of literature searching with specific criteria to report [3, 4, 6, 8‐10]. There was consensus on reporting: the databases searched (and the host searched by), the search strategies used, and any use of limits (e.g. date, language, search filters (The CRD handbook called for these limits to be justified [6])). Three guidance documents reported that the number of studies identified should be recorded [3, 6, 10]. The number of duplicates identified [10], the screening decisions [3], a comprehensive list of grey literature sources searched (and full detail for other supplementary search methods) [8], and an annotation of search terms tested but not used [4] were identified as unique items in four documents.

The Cochrane Handbook was the only guidance document to note that the full search strategies for each database should be included in the Additional file 1 of the review [9].

All guidance documents should ultimately deliver completed systematic reviews that fulfil the requirements of the PRISMA reporting guidelines [108]. The guidance broadly requires the reporting of data that corresponds with the requirements of the PRISMA statement although documents typically ask for diverse and additional items [108]. In 2008, Sampson et al. observed a lack of consensus on reporting search methods in systematic reviews [109] and this remains the case as of 2017, as evidenced in the guidance documents, and in spite of the publication of the PRISMA guidelines in 2009 [110]. It is unclear why the collective guidance does not more explicitly endorse adherence to the PRISMA guidance.

Reporting of literature searching is a key area in systematic reviews since it sets out clearly what was done and how the conclusions of the review can be believed [52, 109]. Despite strong endorsement in the guidance documents, specifically supported in PRISMA guidance, and other related reporting standards too (such as ENTREQ for qualitative evidence synthesis, STROBE for reviews of observational studies), authors still highlight the prevalence of poor standards of literature search reporting [31, 110‐119]. To explore issues experienced by authors in reporting literature searches, and look at uptake of PRISMA, Radar et al. [120] surveyed over 260 review authors to determine common problems and their work summaries the practical aspects of reporting literature searching [120]. Atkinson et al. [121] have also analysed reporting standards for literature searching, summarising recommendations and gaps for reporting search strategies [121].

One area that is less well covered by the guidance, but nevertheless appears in this literature, is the quality appraisal or peer review of literature search strategies. The PRESS checklist is the most prominent and it aims to develop evidence-based guidelines to peer review of electronic search strategies [5, 122, 123]. A corresponding guideline for documentation of supplementary search methods does not yet exist although this idea is currently being explored.

How the reporting of the literature searching process corresponds to critical appraisal tools is an area for further research. In the survey undertaken by Radar et al. (2014), 86% of survey respondents (153/178) identified a need for further guidance on what aspects of the literature search process to report [120]. The PRISMA statement offers a brief summary of what to report but little practical guidance on how to report it [108]. Critical appraisal tools for systematic reviews, such as AMSTAR 2 (Shea et al. [124]) and ROBIS (Whiting et al. [125]), can usefully be read alongside PRISMA guidance, since they offer greater detail on how the reporting of the literature search will be appraised and, therefore, they offer a proxy on what to report [124, 125]. Further research in the form of a study which undertakes a comparison between PRISMA and quality appraisal checklists for systematic reviews would seem to begin addressing the call, identified by Radar et al., for further guidance on what to report [120].