Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review

The electronic search strategy was developed by an experienced information specialist (SW). We searched Medline (Ovid), all PubMed databases, and the Cochrane Methodology Register (see Additional file 1: Appendix A); the last bibliographic search was conducted in October 2018. We also applied the “similar articles” function in PubMed with 4 known key publications to identify additional relevant articles (applied for the first 20 entries). Furthermore, in June 2018 we searched all Cochrane Colloquium abstracts (since 2009) as well as the Cochrane database of oral, poster and workshop presentations (since 1994). We also cross-checked reference lists of all articles included. In addition, we screened guidelines known to us on the conduct of systematic reviews.

We included all evaluations comparing single with double screening (i.e. including at least 2 reviewers screening independently of one another). We did not limit the evaluations to a certain number of screening steps, i.e. evaluations dealing only with one screening step (e.g. title/abstract screening) were eligible for inclusion. Evaluations involving students or persons without screening experience were excluded.. The rationale behind this decision was that we were only interested in testing standards for a highly professional environment (e.g. an HTA agency) and not in whether or how inexperienced researchers could be involved in the screening process. No text mining or automation tools were allowed. No restrictions to the type of studies to be screened in the evaluations (e.g. therapeutic) were applied. Only evaluations published in English and German were included.

As a minimum requirement, each evaluation had to report at least one quantitative measure for missing studies. No data on the agreement between reviewers (e.g. calculation of Cohen’s kappa) were considered, as they were not the focus of our study.

We expected the most frequent comparison to be single versus double screening (i.e. the gold standard applied in the evaluations; see Table 1 for definitions). Additional analyses could include an assessment of the impact of the non-detection of relevant studies, for example, by investigating whether this would have led to changes in the results of a meta-analysis of the study pool originally included. Additional file 2: Appendix B outlines the eligibility criteria in detail.

We used an online screening tool for the screening process (an internal tool called web Trial Selection Database, webTSDB). All titles/abstracts identified in the electronic databases were screened by 2 authors (DP, SB) independently of one another. Discrepancies were resolved by discussion. Cochrane Colloquium abstracts and the database of oral poster and workshop presentations were screened by one author (SB), which is in line with recommendations for the screening of supplementary information sources [3]. All potentially relevant full texts were screened by 2 authors (DP, SB) independently of one another. Discrepancies were resolved by discussion. In the case of discrepant judgements, a third author (SW) was involved.

Data extraction was conducted by one author (SW) and checked by another (DP). Data were summarized in a structured narrative way. The narrative synthesis included information on the sample (evaluations, sets for screening, and studies included), reviewers, screening methods, the gold standard as well as results. In addition, we performed a post-hoc subgroup analysis to investigate the impact of reviewer experience. No assessment of risk of bias or methodological quality was performed due to the nature of our review and the wide range of study designs included in the evaluations analysed.

We calculated the median proportion of missed studies with respect to all screenings. As there were great variations in the number of missed studies between reviewers, we conducted post-hoc subgroup analyses based on reviewer experience.

We did not register our review in the International Prospective Register of Systematic Reviews (PROSPERO), as it did not meet the eligibility criteria (inclusion of at least one outcome of direct patient or clinical relevance). The current systematic review was conducted in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (Additional file 3: Appendix C).

Only 2 of the 4 evaluations included re-analysed data without the missing studies. In 3 of the 7 single screenings, the studies missed would have led to a substantial change in the findings of the meta-analyses. Even though the reviewer responsible was less experienced than the other reviewer involved, the number of studies missed was surprising. For example, he or she missed 11 of 19 studies in the Wilhelm 2011 review ([14] in Pham 2016). In comparison with the results of the other evaluations, this is a major outlier (the results for the other inexperienced reviewer ranged from 3 to 24% missed studies).

Pham did not provide an explanation for this, but even if study inclusion criteria had been applied inconsistently or random errors had occurred, this high number of studies missed is unusual. As this evaluation was the only one with pre-test screening, any topic-related systematic error should have been eliminated.

It has also been shown that reviewer experience has an impact on other tasks in systematic reviews, such as assessing their methodological quality or extracting data [15].

One explanation why studies were missed might be that the research question was too vague and largely depended on the reviewer’s interpretation. Pham stated “the specificity of the review question may have made identification of relevant studies more straightforward for reviewers” [11]. This is supported by the fact that the Bucher 2015 review, which yielded the best results for single screening, had the most narrowly defined research question of all 3 reviews included by Pham, with just one population group, one intervention and one pathogen. Patients, interventions, comparisons and outcomes (PICO) should therefore be defined as exactly as possible in order to avoid dependence on the reviewer’s interpretation of exactly what study types, interventions or, in this example, pathogens might be eligible.

The question remains as to whether it is an appropriate decision to apply single screening of titles and abstracts as a methodological shortcut for rapid reviews. As Shemilt 2016 concluded, such a decision depends on “the willingness of review teams and funders to sacrifice recall in order to substantively reduce the overall workload and total costs of systematic review production”. In our opinion, the reduction in recall is marginal and the results are robust enough to establish this approach as a methodological shortcut, as long as it is applied by an experienced reviewer.

A further aspect should also be considered in future research: all results of the 4 evaluations included refer to the screening of citations retrieved from bibliographic databases as the only information source. However, systematic reviews generally consider several other sources (e.g. clinical study reports provided by regulatory agencies or manufacturers, study registries, scanning reference lists etc.), so that the identification of the relevant study pool does not rely solely on the screening approach for the results of the bibliographic search. The impact of these additional searches on the number of missing studies is not mentioned in the evaluations analysed. However, there is evidence that these different search approaches (e.g. citation searching) could represent useful supplementary alternatives [16]. It should also be noted that in the assessment of drugs, bibliographic databases provide insufficient information to enable the assessment of a primary study and should therefore not be the main information source [17]. None of the 4 evaluations we included mentions this aspect, even though other information sources had also been considered (e.g. Pham 2016 evaluated the screening in Greig 2012, in which reference lists and conference proceedings had also been screened). The impact of studies missed in the screening of results of bibliographic searches may thus be lower than expected when other information sources, which may contain the missing studies, are considered.

We could compare our results only with one other systematic review. Recently, Robson 2018 summarized evaluations of methods for systematic reviews, including study selection. According to their results on screening, the evidence supported the involvement of 2 independent experienced reviewers. Robson 2018 included 4 studies to investigate the question as to whether 2 independent reviewers are required for study selection. They included Yip 2013 [25], which we excluded due to the lack of a quantitative measure for missing studies. In addition, we included one further evaluation (Pham 2016) not included in Robson. Robson summarized the conclusions of the evaluations included, whereas we extracted and analysed the actual data. Our findings may thus potentially provide a more accurate picture of the current evidence. However, we emphasize that our findings can only indicate certain tendencies or be used to help create hypotheses for future research to test when a single screening approach might be applicable.

Evidence is still lacking on the issue as to whether the number of missed studies would change if full-text screening were also performed by a single reviewer.

A further important issue is the technical aspect of screening. Except for Shemilt 2016, none of the evaluations reported whether they had used a screening tool, reference management software, or hardcopies for screening. It can be assumed that 15 to 20 years ago (applies to Edwards 2002 and Doust 2005) screening was conducted using hardcopies, an approach that might be more error prone than the use of a screening tool. Edwards 2002 noted that aspects of electronic records could influence their ease of identification for systematic reviews.

Our work has some limitations: firstly, searching for evaluations on screening approaches is challenging. We tried to identify all relevant sources; however, we cannot exclude that we missed some relevant evaluations. Secondly, we had to rely on the information provided in the evaluations included; re-analyses were not possible due to the way results were reported. Thirdly, we could only roughly classify reviewer experience, as the information provided in the evaluations was inconsistent and incomplete: for instance, only one evaluation reported the extent of screening experience in years and none reported the number of systematic reviews previously conducted.