Searching the scientific literature: Implications for quantitative and qualitative reviews

Abstract

Literature reviews are an essential step in the research process and are included in all empirical and review articles. Electronic databases are commonly used to gather this literature. However, several factors can affect the extent to which relevant articles are retrieved, influencing future research and conclusions drawn. The current project examined articles obtained by comparable search strategies in two electronic archives using an exemplar search to illustrate factors that authors should consider when designing their own search strategies. Specifically, literature searches were conducted in PsycINFO and PubMed targeting review articles on two exemplar disorders (bipolar disorder and attention deficit/hyperactivity disorder) and issues of classification and/or differential diagnosis. Articles were coded for relevance and characteristics of article content. The two search engines yielded significantly different proportions of relevant articles overall and by disorder. Keywords differed across search engines for the relevant articles identified. Based on these results, it is recommended that when gathering literature for review papers, multiple search engines should be used, and search syntax and strategies be tailored to the unique capabilities of particular engines. For meta-analyses and systematic reviews, authors may consider reporting the extent to which different archives or sources yielded relevant articles for their particular review.

Introduction

All empirical publications and review papers, including those in Clinical Psychology Review discuss and rely on previously published work. Researchers and clinicians alike routinely use search engines such as PsycINFO and PubMed to obtain scholarly articles that inform reviews of the literature (e.g., meta-analyses, introduction sections of articles), interpretation of results, future research directions, and clinical practice. Among the various methods that can be used to identify relevant literature, web-based search engines (e.g., PubMed) are perhaps the quickest and most accessible (see Arnold et al., 2006, Falagas et al., 2008, for reviews). In fact, the vast majority of reviews and meta-analyses typically use electronic archives as a key source for identifying relevant literature, in addition to other methods of identifying relevant literature (Lipsey & Wilson, 2001). In the years 2007–2009, for example, 65 meta-analyses were published in three representative journals (Health Psychology, Psychological Bulletin, Journal of Consulting and Clinical Psychology). Of these articles, the search engines PubMed/Medline and/or PsycINFO were used in almost all (n = 61; 93.8%) studies to obtain relevant articles.

Although searching of electronic databases has many benefits, including being able to search large archives in a short time, there are a variety of challenges that arise when using these engines. Importantly, these challenges may affect the results of searches if users are not aware of potential differences between search engines and the archives with which they are connected. First, the archives that search engines access may differ in the journals that are included and in the proportion of literature that is relevant for particular research or clinical areas (Gavel and Iselid, 2008, Lohonen et al., 2010, Watson and Richardson, 1999a, Watson and Richardson, 1999b). For instance, McDonald, Taylor, and Adams (1999) demonstrated that commonly used electronic databases differ in the psychiatry journals indexed, such that approximately one-third of psychiatry journals are indexed in only one database. Second, journals and their respective fields differ in the terminology used to represent the same idea (e.g., “pediatric” versus “child”) and using the same search terms across engines may lead to disparate results simply because one term is preferred over another within particular literatures. Third, search engines can have different structures, such as unique search capabilities, which can lead to different results. In PsycINFO, for example, users can filter search results by “quantitative study” whereas users cannot apply this filter in PubMed. Fourth, search engines and archives use different index terms that can affect search results. Two of the most commonly used resources in clinical psychology are PubMed and PsycINFO. PubMed, a search engine that draws from MEDLINE in addition to other sources uses the U.S. National Library of Medicine (2011) Medical Subject Headings, (MeSH) which are assigned to individual articles. In contrast, PsycINFO, an archive administered by the American Psychological Association (APA), uses index terms from the Thesaurus of Psychological Index Terms (Tuleya, 2007). Although some index terms are common to both MeSH and the Thesaurus of Psychological Index Terms, there are numerous terms unique to each system. In addition, there are terms that describe the same construct but that differ between systems (e.g., “Attention Deficit Hyperactivity Disorder” [ADHD] in the Thesaurus and “Hyperkinetic Disorder” in MeSH). Indeed, these differences between databases and their respective search engines lead to varying results when parallel searches are run (Arnold et al., 2006, Brettle and Long, 2001, Conn et al., 2003). Due to these differences in archive content, terminology, index terms, and search capabilities between particular databases, some studies have examined ways to tailor and optimize electronic search strategies (Eady et al., 2008, Jenuwine and Floyd, 2004).

In sum, the existing literature indicates that search results can differ between databases and that search strategies should account for the capabilities and structures of individual search engines (Kelly & St. Pierre-Hansen, 2008). However, this literature has focused primarily on the search results obtained when the same search terms are used across different databases, or the results of tailored search strategies within single databases (e.g., PsycINFO). Few studies have examined the extent to which search results differ when search strategies are tailored to the index terms and search capabilities of particular engines. This issue is an important concern to authors of systematic reviews and meta-analyses who want to ensure that the whole literature is comprehensively searched so that accurate conclusions and interpretations can be drawn. Researchers may overlook or underestimate the extent to which archives and search engines function differently, use different algorithms, and include different parts of related literatures. However, it is unknown to what extent the two major search engines for clinical psychology literature (i.e., PsycINFO, PubMed) provide unique or duplicate returns. In addition, no studies to our knowledge have examined potential differences in terms of the content of articles (e.g., whether articles address biological correlates, treatment issues) retrieved from different archives. An examination of these issues would inform both the search strategies that authors use and understanding of potential differences in article content between archives. Furthermore, these issues are particularly relevant for intersecting research fields (e.g., medicine and psychology) and fields contributing to the application of quantitative methods of research in psychology (e.g., mathematics, statistics).

Consistent with prior literature examining broad methodological issues affecting study design and result presentation (e.g., Kratochwill and Levin, 2009, Lane and Sandor, 2009), the primary aim of this review was to investigate a methodology (i.e., literature searching) which can affect research design, results (e.g., meta-analytic results), and interpretation of results. Specifically, we examined potential differences in literature search results when comparing search strategies and archives. Using exemplar topics, comparable searches were conducted in PsycINFO and PubMed to identify systematic differences in the results obtained by the two search engines. The results were examined for (a) differences in the relevance of the articles returned with regard to the topic of investigation (see Procedure section), (b) differences in select features of article content, and (c) the extent to which the search engines returned unique versus duplicate results.

Previously published studies on literature search methodology have typically focused on specific content areas (e.g., health care worker burnout, rehabilitation services for individuals with severe mental illness; Arnold et al., 2006, Brettle and Long, 2001) and have used the results of these topic-focused searches to illustrate larger literature search issues. Consistent with this methodology, we chose a particular content area on which to focus in our searches. Specifically, we selected ADHD and bipolar disorder because of the growing literature regarding diagnostic and comorbidity issues for these disorders (e.g., Carlson, 1998, Geller et al., 2002), as well as the substantial literatures addressing broader facets of each disorder. These disorders are also of immediate relevance to the current revision process for the International Classification of Diseases (ICD) and the Diagnostic and Statistical Manual (DSM; First, 2010, Reed, 2010). We therefore focused our searches on ADHD and bipolar disorder and issues of classification and diagnosis. In addition, we chose to focus our literature searches on review and meta-analytic articles, because researchers and practitioners alike rely on reviews to provide summaries of the extant literature and to identify primary sources which may be of interest (Lipsey & Wilson, 2001). Thus, similar to the previously published literature in this area, although our search speaks to a specialty question for a particular domain, the aim of our analysis is to produce information with search implications for other domains and subfields within clinical psychology.