Background
Systematic reviews have been integral to the evidence-informed practice movement [
1‐
5] in the field of public health [
6‐
9]. A systematic review consists of an examination of all of the primary studies on a topic, which includes searching for, collating, and assessing the studies, to establish conclusive evidence about a topic [
10]. Systematic reviews present a more consistent and conservative estimate of the effect of interventions across a body of literature and as such, can have an important impact on program planning decisions in public health. However, public health decision makers state that finding and accessing systematic reviews related to public health continues to be a barrier to evidence-informed public health practice [
11‐
16]. The field of public health can be defined as a combination of sciences, skills, and values that function through collective societal, legislative, and political activities. It involves both public and private programs, services, and institutions aimed at protecting and improving the health of all people, including preventing disease, promoting health and wellbeing, and prolonging life. When necessary, public health also engages in restoring the health of individuals, specified groups, populations or communities through mobilizing and engaging local, state, national, and international resources to assure the conditions in which people can be healthy [
17‐
19]. In short, the field of public health is broad, and decision makers wear many hats, requiring evidence on a wide range of topics.
Public health practitioners have expressed a need for a single place where they can access reviews evaluating the effectiveness of interventions, have confidence in the methodological quality of the evidence, and access plain language review summaries with corresponding implications for policy and practice [
20]. Health-evidence.ca is a free, searchable online registry of systematic reviews and meta-analyses evaluating the effectiveness of public health and health promotion interventions. This registry represents one component of a larger knowledge translation and exchange (KTE) [
21] strategy that supports users in accessing and interpreting research evidence. KTE is a two-way process involving dialogue, interaction, and the sharing of knowledge and evidence between and among the producers and users of knowledge and research evidence. It is a broad term that is often used to include knowledge transfer, exchange, translation, dissemination, and diffusion. The target audience for health-evidence.ca is decision makers working in public health and health promotion at all levels (front line practitioners to senior management and policy makers in government). Public health decision makers need to find, assess and interpret research evidence quickly and easily if it is to inform program and policy decisions. Health-evidence.ca provides decision makers with easy access to public health-relevant, quality-appraised systematic reviews evaluating the effectiveness of public health interventions. The site is freely accessible and can be searched by selecting common public health indexing terms. Search results include links to published review abstracts and a rating of the methodological quality of each review. In addition, health-evidence.ca team members write evidence summaries for reviews of good methodological quality to summarize key findings and provide recommendations for policy and practice. A more complete description of this online resource has been published and is accessible at
http://www.biomedcentral.com/1471-2458/10/496.
Health-evidence.ca was updated quarterly until 2012 and is now updated on a monthly basis. Updates consist of conducting monthly searches of relevant electronic databases, importing results into a bibliographic database management program, screening titles to identify relevant articles, retrieving potentially relevant articles and screening full document versions for inclusion. Included reviews must meet relevance criteria and must be systematic reviews that focus on public health, provide outcome data on the effectiveness of interventions, and include a documented search strategy.
As of February 2012, over 1,017,500 titles had been screened, yielding 2,450 relevant reviews. The large number of titles screened to reach the final, relevant set reflects the challenges of searching bibliographic databases for public health and health promotion literature. These challenges stem from the lack of a single database dedicated exclusively to public health and health promotion literature, requiring searches in multiple health (MEDLINE, EMBASE, CINAHL), science, and social science databases (BIOSIS, PsycINFO, SPORTDiscus, Sociological Abstracts). There are also several limitations inherent in searching these databases. For example, 33-44% of the journals identified by experts in the field as public health journals are not indexed in MEDLINE. These challenges are not limited to public health as others have encountered similar difficulties in searching for mental health content [
23] and health services research literature [
24]. A further challenge is identifying what is relevant to public health and health promotion practitioners, given that it is a dynamic field characterized by a wide scope of practice, defined regionally and changing constantly.
Along with the challenges of searching for public health and health promotion content, review literature, though rapidly growing, remains limited in volume when compared to primary studies. For example, over 700,000 articles were indexed in MEDLINE in 2010, of which approximately 2500 (0.36%) were health-related systematic reviews [
25]. Currently, there is no single MEDLINE subject heading term for ‘systematic review’; this lack of an indexing term requires the end user to employ a Clinical Query developed to locate systematic reviews, or to screen very large sets of irrelevant articles in order to retrieve systematic reviews. MEDLINE does have an indexing term for ‘review’ however its application is very broad. Of the 19,430,768 articles currently indexed in MEDLINE as of February 13, 2012, 8.5% (1,656,583) [
26] were indexed as reviews. Upon screening a small portion of this results set, it was evident that the majority were not systematic reviews, but rather literature reviews and overviews. While the MEDLINE indexing term ‘meta-analysis’ is useful for identifying systematic reviews, it only captures systematic reviews that use statistical software to combine the results of the included primary studies in a single pooled estimate of effect. However, meta-analyses represent a small portion of all reviews evaluating the effectiveness of public health interventions. For example, fewer than half of public health intervention reviews indexed on health-evidence.ca are meta-analyses, thus reliance on this text word to identify reviews is not sufficient. A combination of indexing terms is required to detect relevant reviews that can be captured in online databases such as MEDLINE. Thus, although it has been time-consuming, screening a high number of irrelevant articles has been necessary. Search filters, also referred to as “search hedges”, are “collections of search terms intended to capture frequently sought research methods such as randomized controlled trials, or other aspects of health care” [
27]. While search filters for the retrieval of systematic reviews were being used by others for searching MEDLINE [
19‐
31], EMBASE [
32], and CINAHL [
33], none had been used and tested for locating public health and health promotion reviews that we were aware of at the time of this project. These filters, including those targeting content-specific literature relevant to the subject of interest [
24,
25], provided guidance as we developed a systematic review filter for health-evidence.ca.
Prior to 2008, we used a Public Health (PH) search filter that was developed in collaboration with health science librarians at McMaster University. The Head of Public Services worked with one of the authors (KD) to systematically run and informally evaluate the results of various search strategies for retrieving systematic reviews and meta-analyses evaluating the effectiveness of public interventions in MEDLINE, EMBASE, CINAHL, PsycINFO, and Sociological Abstracts. Search strategies were assessed and improvements made based on findings. The resulting PH search filter consisted of two distinct components: 1) indexing terms and keywords referring to systematic review methods, combined with the Boolean ‘OR’ operator (systematic, meta analysis, review); and 2) indexing terms and keywords referring to public health content areas, combined with the Boolean ‘OR’ operator (community health services, education, health education, health promotion, prevention, preventive). The content and methods components were then combined using the Boolean ‘AND’ operator. Seventeen topic areas were included in the content component: addiction, adult health, chronic diseases, communicable disease and infection, community health, dental health, environmental health, food safety and inspection, injury prevention and safety, mental health, nutrition, parenting, physical activity, pregnancy, sexual education, sexually transmitted infections, and women’s health. This search strategy also made it more likely that we would capture articles for which established indexing terms did not exist such as social determinants of health and healthy communities.
Our PH search filter typically yielded a very high volume of results with very low precision. For example, between January 2006 and December 2007, of the 136,427 titles screened, 409 were relevant for the health-evidence.ca registry, or in other words, precision was 0.3%. In addition to using the PH search filter, more than 40 public health-relevant journals were hand searched annually, as well as the reference lists of all relevant reviews. Given this systematic search of the published review literature, we were reasonably confident that our retrieval methods were capturing a near complete set of relevant articles. We considered this set (the electronic database searches plus additional search strategies), the ‘gold standard’ for health-evidence.ca. A gold standard is “a set of relevant records against which a new search filter is tested and validated to determine how effective it is at retrieving particular types of records” [
34]. While it is impossible to prove that the gold standard for health-evidence.ca identified all public health relevant systematic reviews, we are confident that this approach captured the vast majority of relevant reviews.
Given that the precision of the PH search filter was so low, we began to create an effective search filter that would decrease the total number of results retrieved, while maximizing the number of relevant results. The health-evidence.ca Systematic Review (SR) search filter we developed in 2008 was adapted from a previously-validated filter [
30], which included the terms: MEDLINE.tw, systematic review.tw, meta-analysis.pt, combined with the Boolean OR operator. While this filter was highly specific, it captured less than 82% of articles identified by our gold standard set. To customize this filter to retrieve only those systematic reviews of interventions, the term ‘intervention’ was added as an indexing term. This is referred to as the development data set.
The MEDLINE version of our health-evidence.ca SR search filter included the following indexing terms, combined with the Boolean ‘OR’ operator: MEDLINE.tw, systematic review.tw, meta-analysis.pt, intervention$.ti. We slightly modified the filter for use in EMBASE and CINAHL due to differences in indexing terms between the various databases. The indexing terms systematic review.tw and intervention$.ti are viable in both EMBASE and CINAHL, therefore these terms were consistent across all three databases. However, in both EMBASE and CINAHL, meta-analysis was not an indexed publication type, and therefore the term meta-analysis was included as a keyword in the search filter for these two databases. Each database employs a unique controlled vocabulary, thus the search strategy is tailored to the database. For example, MEDLINE does not have a preferred search term for systematic review so that concept must be searched as a text word. EMBASE and CINAHL, however, do have a specific indexing term for systematic review, so that term is used when tailoring the search to those databases.
The objective of this paper is to report the results of our efforts to evaluate and validate the health-evidence.ca SR search filter for retrieving systematic reviews and meta-analyses that evaluate the effectiveness of interventions. First, we compared the performance of the health-evidence.ca SR search filter to the PH search filter. We then compared the health-evidence.ca SR search filter to other known search filters targeted at capturing systematic reviews in existence at the time (Tables
1,
2 and
3).
Table 1
Performance of search terms and filters designed for retrieving systematic reviews in MEDLINE
health-evidence.ca
|
health-evidence.ca SR search filter | 89.9 (85.0, 93.3) | 98.9 (98.9, 98.9) | 1.4 (1.3, 1.5) | 71.4 (68.7, 75.5) |
Montori, et. al (2005) |
Sensitive query | 99.0 (96.5, 99.7) | 62.0 (62.0, 62.0) | 0 (0, 0) | 2191.2 (2166.3, 2284.3) |
‘Balanced query’ (sensitivity > specificity) | 99.0 (96.5, 99.7) | 87.6 (87.6, 87.6) | 0.1 (0.1, 0.1) | 712.4 (706.7, 733.4) |
Balanced query (specificity > sensitivity) | 87.9 (82.8, 91.7) | 98.5 (98.5, 98.5) | 1.1 (1.0, 1.1) | 94.9 (90.9, 100.9) |
Specific query | 81.6 (75.8, 86.3) | 99.3 (99.3, 99.3) | 2.0 (1.9, 2.3) | 49.4 (46.7, 53.2) |
Shojania and Bero (2001) | 85.5 (80.1, 89.7) | 99.1 (99.1, 99.1) | 1.7 (1.6, 1.8) | 57.8 (55.1, 61.8) |
Hunt and McKibbon (1997)4 terms | 69.6 (63.0, 75.4) | 99.4 (99.4, 99.4) | 1.9 (1.7, 2.0) | 53.9 (49.7, 59.6) |
Hunt and McKibbon (1997)8 terms | 85.5 (80.1, 89.7) | 99.2 (99.2, 99.2) | 1.9 (1.8, 2.0) | 53.4 (50.9, 57.0) |
Boynton, et. al (1998) |
Sensitivity maximiser | 99.5 (97.3, 99.9) | 75.6 (75.6, 75.6) | 0.1 (0.1, 0.1) | 1395.1 (1387.7, 1437.2) |
Precision query (> 70%) | 47.8 (41.2, 54.6) | 99.6 (99.6, 99.6) | 2.1 (1.8, 2.5) | 46.7 (40.9, 54.4) |
BMJ Clinical Evidence
| 88.9 (83.9, 92.5) | 99.0 (99.0, 99.0) | 1.6 (1.5, 1.7) | 61.7 (59.3, 65.5) |
Centre for Reviews and Dissemination
|
For inclusion in DARE | 92.8 (88.4, 95.6) | 95.7 (95.7, 95.7) | 0.4 (0.4, 0.4) | 262.2 (254.2, 275.8) |
Strategy 1 | 99.0 (96.5, 99.7) | 71.2 (71.2, 71.2) | 0.1 (0.1, 0.1) | 1693.1 (1659.8, 1773.3) |
Strategy 2.1 | 99.5 (97.3, 99.9) | 87.4 (87.4, 87.4) | 0.1 (0.1, 0.1) | 717.5 (714.2, 736.1) |
Strategy 2.2 | 99.0 (96.5, 99.7) | 88.9 (88.9, 88.9) | 0.2 (0.2, 0.2) | 636.0 (631.0, 654.5) |
Scottish Intercollegiate Guidelines Network Filter
| 87.0 (81.7, 90.9) | 99.2 (99.2, 99.2) | 1.9 (1.8, 2.0) | 52.0 (49.7, 55.4) |
Table 2
Performance of search terms and filters designed for retrieving systematic reviews in EMBASE
health-evidence.ca
|
health-evidence.ca SR search filter | 87.9 (80.3, 92.8) | 98.2 (98.2, 98.2) | 0.5 (0.5, 0.6) | 186.0 (176.0, 208.9) |
Wilcynski and Haynes (2007) |
Sensitive query | 96.3 (90.8, 98.5) | 72.3 (72.3, 72.3) | 0 (0, 0) | 2709.5 (2622.5, 2945.2) |
‘Small drop in specificity, substantive gain in sensitivity’ query | 75.7 (66.7, 82.8) | 99.3 (99.3, 99.3) | 1.1 (1, 1.2) | 88.2 (80.5, 100.1) |
Best optimization query | 96.3 (90.8, 98.5) | 85.5 (85.5, 85.5) | 0.1 (0.1, 0.1) | 1403.4 (1363.4, 1502.0) |
Specific query | 63.4 (28.0, 45.9) | 99.5 (99.5, 99.5) | 0.9 (0.7, 1.1) | 117.8 (93.4, 154.2) |
BMJ Clinical Evidence filter
| 84.1 (76.0, 89.8) | 98.5 (98.5, 98.5) | 0.6 (0.5, 0.6) | 167.9 (157.0, 186.1) |
Centre for Reviews and Dissemination filter
| 66.4 (57.0, 74.6) | 97.6 (97.6, 97.6) | 0.3 (0.3, 0.3) | 341.0 (302.0, 400.0) |
Scottish Intercollegiate Guidelines Network filter
| 81.3 (72.9, 87.6) | 99.0 (99.0, 99.0) | 0.8 (0.8, 0.8) | 118.6 (110.1, 132.5) |
Table 3
Performance of search terms and filters designed for retrieving systematic reviews in CINAHL
health-evidence.ca
|
health-evidence.ca SR search filter | 89.9 (93.5, 94.0) | 97.6 (97.6, 97.6) | 1.8 (1.6, 1.8) | 57.2 (54.7, 61.7) |
Wong, et. al (2006) |
Best sensitivity query | 96.1 (91.2, 98.3) | 94.6 (94.6, 94.6) | 0.8 (0.8, 0.8) | 120.8 (118, 127.7) |
‘Small drop in sp, substantive gain in sensitivity’ query | 45 (36.7, 53.6) | 95.3 (95.3, 95.3) | 0.5 (0.4, 0.5) | 235.3 (193.9, 296.4) |
Best optimization (sensitivity > specificity) query | 50.4 (42, 58.8) | 99.4 (99.4, 99.4) | 3.8 (3.2, 4.5) | 26.3 (22.5, 31.6) |
Best specificity query | 47.3 (38.9, 55.8) | 99.4 (99.4, 99.4) | 3.5 (2.8, 4.1) | 29.1 (24.7, 35.5) |
Centre for Reviews and Dissemination Filter
| 98.4 (94.5, 99.6) | 94.0 (94.0, 94.0) | 0.8 (0.7, 0.8) | 130.4 (128.9, 136.2) |
McKibbon (1998) | 78.3 (70.5, 84.5) | 98.9 (98.9, 98.9) | 3.2 (2.9, 3.4) | 31.7 (29.3, 35.2) |
Our intent was to identify a search filter that resulted in the optimal use of time and resources in updating the health-evidence.ca registry. Specifically, this paper reports the performance of each filter with respect to sensitivity, specificity, precision, and the number needed to read. The best option for our purposes is one that achieves high precision while not compromising sensitivity.
Discussion
The objective of health-evidence.ca is to contribute to evidence-informed decision making in public health by facilitating access to published systematic reviews evaluating the effectiveness of public health and health promotion interventions. An optimal search filter for health-evidence.ca is one that has high sensitivity, specificity, and precision and a relatively low NNR. However, any reduction in NNR was desirable. A filter such as this allows us to have confidence that all relevant articles will be identified (sensitivity), fewer non-relevant articles will be retrieved (specificity), most of the identified articles will be relevant (precision), and the NNR will be reduced. Reducing the NNR is of great importance since screening is a resource- and time-intensive process.
Although a search filter may perform exceptionally well on any single outcome, it is the balance of performance across these four domains – sensitivity, specificity, precision, NNR – that distinguishes the best filter for our purposes. By replacing the PH search filter with the health-evidence.ca SR search filter, the overall number of articles retrieved from health-evidence.ca electronic searches was greatly reduced without losing relevant content. The balance struck by the SR search filter means that this filter would be useful to those wishing to retrieve systematic reviews related to health care, with wider application than that of our own database of reviews on the effectiveness of interventions. The desired benefit of filters is that they save time both in search strategy development and screening. One study demonstrated how filters reduce the number of results needed to screen [
37], while another found that saving time both in search strategy development and screening of results was the most common benefit reported by librarians [
38]. For our purposes, the health-evidence.ca SR search filter offered overall improvements in specificity and precision, with the associated decrease in the NNR, substantially decreasing screening time. The desired improvement in precision was feasible while only minimally impacting the sensitivity of the search strategy. The results of this study illustrate that for the most part, the health-evidence.ca SR search filter outperformed the PH search filter with respect to sensitivity, specificity, precision and NNR in all three databases. However, it was the overall balance among these variables and the fact that high precision could be combined with high sensitivity that made the health-evidence.ca SR search filter the optimal choice for identifying systematic reviews evaluating the effectiveness of interventions.
When compared to other filters in MEDLINE, EMBASE and CINAHL, overall, the health-evidence.ca SR search filter offered the right balance of sensitivity, specificity, precision, and NNR. Although other filters had higher sensitivity scores than the health-evidence.ca SR search filter in MEDLINE, these higher sensitivity scores were generally accompanied by poorer precision and NNR performance. In EMBASE, the health-evidence.ca SR and Scottish Intercollegiate Guidelines Network search filters performed the best overall and were comparable in terms of performance across all of the outcome measures. Likewise in CINAHL, though the health-evidence.ca SR search filter did not outperform other filters on any single outcome, it offered the most robust overall result of high sensitivity and specificity with a reasonably low NNR in comparison to other filters.
The health-evidence.ca SR search filter streamlines the process of locating and screening relevant reviews by allowing us to effectively search health databases with a simpler strategy that maintains a high level of both sensitivity and precision. The task of searching the health databases for every relevant systematic review evaluating effectiveness of public health interventions is a challenging one that requires balance. Because of the growth of the literature in the area of systematic reviews, highly sensitive searches often come up with result sets that are unmanageably large. However, if a search is too specific, then it has the risk of missing relevant articles. It is important to establish the right balance in the trade-off between sensitivity and specificity depending on what will best serve the purpose at hand [
39,
40]. Using the health-evidence.ca SR search filter has allowed us to achieve the right balance in our searches by retaining greater than 85% sensitivity across all three databases, while reducing the NNR by two thirds. We estimate that this has translated into a savings of 384 hours of staff time per quarterly update of health-evidence.ca by reducing the hours required to execute database searches, screen results, retrieve full-text versions of potentially relevant reviews, and test reviews for relevance. The reduction has meant that resources are available for the exploration and development of new protocols for searching other relevant but previously unexplored electronic databases covering areas such as environmental health, social welfare, and veterinary sciences for relevant public health content.
The health-evidence.ca SR search filter is an easy-to-use tool. It can be entered into the OVID interface for searching in MEDLINE and EMBASE. Compared to other more complex filters, the health-evidence.ca SR search filter is easily entered. A survey of librarians revealed that users find search strings too long [
38,
40]. The SR search filter used by health-evidence.ca is a relatively short search filter, with other authors also finding that the brief search filters work well. Our results, which are similar to those of others [
39,
38], indicate that methodological search filters can be as or more effective than content filters for retrieving relevant systematic reviews [
27‐
35,
39]. Using a methodological filter allows us to circumvent the need to generate an accurate and all encompassing definition of public health that can be translated and applied across indexing systems within different databases. However, if desired, the search strategy can be combined (using Boolean logic, e.g. AND) with topic-specific search terms to reduce the number of articles retrieved, if applied for a specific topic area (e.g. influenza).
Limitations
Searching was conducted in OVID’s search interface for all three databases; other search interfaces for these databases (e.g. PubMed) may handle the searches somewhat differently. As of August 30, 2008, CINAHL moved from OVID Technologies to be hosted by EBSCO, exclusively. Unfortunately, this change to EBSCO renders the CINAHL filters included in this paper, including our filters, out of date. The performance of these filters would require reevaluating them in the EBSCO platform before their application. This brings light to a key limitation of search filters – creation dates must always be considered before using a filter as changes to indexing terms and hosting platforms can impact filter function.
The sensitivity scores calculated for each search filter can be applied to broader searches for systematic reviews evaluating various interventions and are not necessarily applicable only to public health interventions. However, precision and NNR scores were calculated specifically for public health content and cannot be generalized to topic areas outside of public health. The low precision scores yielded across all search filters were expected, since precision is generally low when searching large databases [
39,
40]. Lastly, our group’s own manual screening set was used as the gold standard. Although a consistent set of relevance criteria were applied to generate this results set, screening was shared between two authors (MD, KD), and several other members of the health-evidence.ca team. Although either MD or KD acted as second reviewer on each article, there was still potential for reviewer bias through the involvement of a small number of reviewers. Additionally, having a combination of both systematic review methodology indexing terms and public health indexing terms in our PH search filter dually limited our results sets, retrieving only content which met all requirements for both methodology and public health content.
Authors’ contributions
EL conceived of the study, participated in the analysis and wrote the first draft of the manuscript. KD and MD completed subsequent drafts and the final version of the manuscript. LM, DT, and HH consulted on the analysis. All authors read and approved the final manuscript.