Summary of major findings
The results of this study indicate that PubMed is the most fruitful source of scholarly literature relevant to systematic reviews of obesity prevention policy. The databases that index the greatest number of relevant articles not included in PubMed are the large multidisciplinary databases Google Scholar, Scopus, and Web of Science; and databases specializing in economics, namely, EconLit and Business Source Premier. EconLit and Business Source Premier appear especially useful for reviews of economic interventions such as food and beverage taxes or subsidies. Larger databases like Google Scholar, Scopus, and Web of Science cover an expansive body of literature across many disciplines; however, this means that searching these resources may produce unwieldy search results, with potentially large numbers of irrelevant citations.
Scholarly literature searches
When developing search strategies, review authors are not necessarily selecting databases in a manner that maximizes search retrieval. For example, 10 systematic reviews searched CINAHL, making it the second most-searched resource. However, CINAHL only indexed 4 of the 82 relevant articles not indexed in PubMed. By contrast, Scopus, which indexed 66 of the 82 articles, was searched in only 1 review. This implies that the majority of relevant articles identified through CINAHL searches could also be located with a comprehensive search of PubMed. It should be noted that if such a PubMed search were not adequately comprehensive, i.e., if it were not a highly sensitive search, it would be problematic to assume that all relevant articles would be retrieved. This has implications not only for the question of whether to search additional databases, but also for how systematic review searching is approached more broadly speaking. When designing the search methodology for a systematic review, researchers must decide for themselves which balance of specificity vs. sensitivity best suits the aims and parameters of their project.
The fact that searches of Google Scholar retrieved all non-PubMed articles implies that one could potentially locate all relevant scholarly literature by searching these two databases. However, the two processes of searching for a known item by title and discovering that same article within a large list of results when searching a database with keywords are profoundly different [
32]. This fact applies to all databases searched in the present study. We assessed each database’s ability to retrieve relevant studies by searching for those studies by title; it is entirely possible that a keyword search of the same database may not retrieve the same studies, especially if the search strategy were not sufficiently broad.
Google Scholar in particular has been found to be problematic when searched for systematic reviews, for a few reasons. These range from the inability to create a structured and reproducible search strategy to the limits Google Scholar places on search syntax and viewing results [
33,
34]. While all relevant non-PubMed primary studies were indexed by Google Scholar, they may not be retrievable by keyword search in Google Scholar given these limitations. Therefore, it would be imprudent to assume that all relevant literature could be identified through Google Scholar alone in a systematic review. However, it may prove useful in identifying gray and scholarly literature beyond that which can be identified using traditional search methods [
35,
36] and should therefore be considered as one of multiple resources to search.
Gray literature searches
Of the 10 reviews that reported searching gray resources, all but one ultimately included gray literature of some kind in their analyses. An additional three reviews that did not report searching gray resources nonetheless included relevant gray literature in their analyses. Similar to what has been found elsewhere [
37‐
39], this indicates that searching gray resources, e.g., organizational websites or conference proceedings, in addition to traditional bibliographic databases, may likely produce relevant data for research syntheses. Despite the additional time this requires, searching the gray literature broadens the scope of a review and helps researchers avoid potential publication bias.
Some of the challenges inherent to including gray literature can be mitigated by using a systematic method for searching and identifying evidence outside of traditional databases. For example, developing a plan for the search in advance, including the sources to search and terms to use, can help keep this process manageable within the desired timeframe [
39]. Stansfield, Dickson, and Bangpan [
40] propose a three-stage process that allows the flexibility necessary to adapt website searches to various research topics while preserving the systematic review principles of transparency, accountability, and reproducibility.
Limitations
These conclusions are applicable to systematic reviews of obesity prevention policy and are not necessarily generalizable to other health policy or public health topics. Systematic reviews in other health policy areas may require the use of resources other than those found to be most effective in this example, due to the fact that there may be different, discipline-specific databases more appropriate for a given topic. Further, these results pertain to research on “big P” policies, comprising actions by governmental bodies; conclusions may not transfer to studies on “small p” policies, such as organizational policies or interventions. While our search was comprehensive and included several databases, we did not search any resources that specifically focus on evidence syntheses, such as Epistemonikos, Trip, and Health Evidence. There may be additional systematic reviews of obesity prevention policy that we did not identify, either in these resources or elsewhere.
The data collected on search methods and citations were limited to what review authors provided in the articles. When possible, we contacted authors for missing or unclear information. Only 5 of the 21 reviews reported a complete, line-by-line search strategy, including all search terms and indication of what fields were searched in which database. Consequently, we were unable to assess the quality of the included reviews’ search strategies. A review with a demonstrably superior search strategy would likely produce different results from a review with a flawed search strategy, especially if a search erred in favor of specificity rather than sensitivity. Reviews with insufficiently broad search strategies may fail to identify additional relevant primary studies. If this were the case with any of the reviews included in this study, it could affect the conclusions we have drawn; without complete search strategies, it is not possible to know. The incompleteness of published search strategies in the majority of systematic reviews analyzed here underlines the need for increased scrutiny at the editorial level, in order to ensure that all published systematic reviews contain truly reproducible search strategies.
No published Cochrane reviews currently exist on this topic. Given the rigor of Cochrane review methodology, it may have strengthened or otherwise affected our analysis to have included one or more of these. While no Cochrane reviews were available for inclusion in our study, as of June 2017, several protocols for proposed Cochrane reviews of food and beverage taxation that appear potentially eligible for inclusion have been published.
We analyzed 23 of the 30 databases that review authors reported searching but were unable to access the remaining 7 databases. It is possible that these databases may include many of the articles not included in PubMed, and possibly more of them than the databases we searched. While authors reported which databases they searched, it was not possible to identify the origin of articles indexed in multiple databases. Therefore, while 76% of the scholarly articles were indexed in PubMed, we cannot say for certain that this is where the authors identified all of them. If an article is indexed, for example, in both PubMed and CINAHL, the CINAHL search may have identified this article if different keywords and controlled vocabulary were used than in the PubMed search.
We located these articles individually in PubMed; as mentioned above, this is a very different process than searching PubMed using keywords. As such, searching additional databases with significant MEDLINE overlap, such as CINAHL, PsycINFO, and Embase, may be useful in that they can help “fill in the gaps” missed by a PubMed search. The data presented in Table
2 should therefore be interpreted with caution. However, these data indicate that if a highly sensitive search is run in PubMed, diminishing returns may be seen in the number of unique relevant articles added to the review by these additional databases. Some researchers may still wish to search these databases in order to ensure that nothing was missed in PubMed.
Finally, while all systematic reviews addressed policies related to obesity prevention, they were nevertheless a heterogeneous collection, including topics from transportation and physical activity to food and beverage taxes and school-based policies. Further research may include a closer analysis of the reviews that had equivalent objectives, for example, those that examined economic interventions, allowing for a more apples-to-apples comparison.