Single screen of citations with excluded terms: an approach to citation screening in systematic reviews

Systematic reviews aim to identify all empirical evidence to answer a specific research question [1]. As part of the systematic review process, comprehensive literature searches of multiple databases are performed to identify all potentially relevant studies. The number of citations identified for a systematic review varies greatly based on the complexity of the systematic review and research questions (e.g., number of questions, inclusion criteria), the number of literature databases searched, the search strategies employed, and the range of search dates used. For some systematic reviews, thousands (or tens of thousands) of citations of potentially relevant studies are identified [2, 3]. As a result, systematic reviewers are exploring methods to improve the accuracy and efficiency of the citation screening process without compromising the quality of the systematic review [4‐9]. This article describes an approach to expediting the citation screening process by single screening citations that include terms in the abstract and/or keywords related to the exclusion criteria of the systematic review to quickly reject studies that have a high likelihood of being excluded from the systematic review.

An initial literature search designed by a research librarian was conducted for a U.S. Preventive Services Task Force systematic review on the use of non-traditional risk factors in cardiovascular disease risk assessment [10]. The initial search yielded 18,360 unique citations; citations were managed in EndNote™ version 7.3.1 (Thomson Reuters, New York, N.Y.). The final research questions, protocol, and systematic evidence review [10], including the literature search strategies and study selection criteria, can be found at https://​www.​uspreventiveserv​icestaskforce.​org.

After receiving the results of the initial search, but prior to initiating citation screening, a list of 83 terms (50 truncated terms) related to an excluded setting, population, or condition was developed based on the exclusion criteria of the systematic review (Table 1); from here on, these terms are labeled as “excluded terms.” In EndNote, these excluded terms were searched for in the abstract and/or keyword fields (using an asterisk for truncation) in groups of 10 (the maximum allowable number of terms in a search group) using the Boolean operator “OR.” Due to the limited number of groups of terms that can be combined into a “group of groups” (10 groups), one group of 10 excluded terms was only searched for in the abstract field. The nine groups of excluded terms were batched together using the Boolean operator “OR” and then separated from the group of citations that did not have these excluded terms in their abstract and/or keywords using the Boolean operator of “NOT” (reaching the limit of 10 groups in a “group of groups”). These citations were single-screened by the author for relevancy using the pre-specified inclusion and exclusion criteria of the systematic review, with single citations being either rejected or moved forward for dual abstract screening. Investigators screened citations in DistillerSR (Evidence Partners, Ottawa, Canada), an online-based systematic review software program.

A total of 6503 (35.4%) of the 18,360 citations from the initial search contained excluded terms in the abstract and/or keywords; these citations were screened by a single reviewer. The remaining 11,847 citations underwent traditional dual screening. Only 246 (3.8%) of 6503 citations containing excluded terms were moved forward for dual screening as they were deemed potentially relevant for inclusion, of which 39 (0.6% of 6503 citations with excluded terms) underwent dual full-text review. Among 39 articles reviewed at full-text, only 2 (0.03% of the 6503 citations with excluded terms) were included in the systematic review. The remaining 37 articles were excluded due to reporting on the wrong outcomes, being the wrong study design, or evaluating the wrong risk prediction base model. The two citations included in the systematic review did not influence the conclusions of the review. None of the 6257 citations with excluded terms—which did not undergo a second review—that were immediately rejected at single screening were brought forward for dual screening or full-text review based on checking the reference lists of included studies or existing systematic reviews. In total, 99.9% of the 6503 citations with excluded terms were excluded from the systematic review. Although these results only describe the initial search of the systematic review, these percentages are likely to not change with the inclusion of citations identified from bridge searches or after expert review and public comment.

Single screening of citations with terms in the abstract and/or keywords that are related to the exclusion criteria of the systematic review is a potential method to increase the efficiency of the citation screening process as it reduces the number of citations requiring dual screening. It also has the potential to be reliable method as 99.9% of the 6503 citations with excluded terms were excluded from the systematic review. This approach would need to be reproduced in other systematic reviews to determine if it is a valid, reliable, accurate, and efficient method to citation screening. Single screening of citations with excluded terms was appropriate for this topic given its high literature yield and the a priori knowledge that most of the citations would not be relevant. This approach, however, may only work well (or be necessary) in topics that have a high literature yield and not those that have less restrictive inclusion criteria (e.g., any country). The amount of time spent on single and dual screening would also need to be compared to truly evaluate efficiency, which was not captured in this study. An optimal method for identifying and batching excluded terms is also needed. In this study, the selection of excluded terms was not systematic and could have been better informed by screening a few hundred citations to identify excluded terms that were more frequent or to omit those that might lead to false excludes. For example, “Kenya, Kenyan” and “Libya, Libyan” yielded zero citations, while “stent” (not an excluded term used) yielded 1480 citations. More excluded terms could also have been used if EndNote was not used to search for citations with excluded terms, as EndNote’s search function limited the number of terms to be searched and grouped. Excluded terms could be integrated into the original literature search strategies and imported separately into EndNote from those that did not have an excluded term in the abstract and/or keywords. And finally, an evaluation of whether the precautions to ensure all relevant studies are identified (i.e., examining reference lists of included studies and existing systematic reviews) is needed to determine that eligible studies with excluded terms were not wrongfully excluded during the single screen. There is still a possibility—even with reference mining—of a false negative as a result of excluding a relevant citation published since included studies or systematic reviews were completed.

Citations identified for a systematic review should at least be single-screened for eligibility, and citations with an excluded term should not be automatically rejected as the excluded term might be in the introduction or background section of the abstract. Due diligence should be taken to ensure no studies were wrongfully excluded by the single screener such as examining references lists of included studies and existing systematic reviews and querying experts for relevant citations. While more efficient processes are being developed and tested, such as machine learning, single screening citations that have a high likelihood of being excluded based on terms in the abstract and/or keywords related to the exclusion criteria of the systematic review can help reduce the number of citations requiring dual screening and thus improve efficiency while maintaining the quality of the systematic review. Additional research is needed to validate this approach.