Database coverage and their use in systematic reviews regarding spinal manipulative therapy: an exploratory study

Systematic reviews (SRs) of randomized controlled trials (RCTs) are widely accepted to be on top of the evidence hierarchy [1, 2]. They are cornerstones in evidence-based healthcare [3] and evidence-based research [4]. This comes to fruition by condensing all relevant and available evidence on a topic and drawing a general conclusion from a broader population by combining sample sizes and thereby reducing biases [5]. In order to collect all relevant studies, a comprehensive literature search must be conducted, and researchers are generally advised to search multiple databases and use additional methods such as citation tracking, contacting experts in the field, and searching gray literature [6‐12]. As the Cochrane Handbook for Systematic Reviews of Interventions highlights, leaving out relevant evidence can lead to selection bias. Cochrane thereby recommends searching at least the following three databases: The Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, and EMBASE [11]. However, these recommendations may not sufficiently cover all relevant aspects of the research question. Some types of research or research topics may only be found in specialty journals that are not indexed in all databases [13]. An example of such could be literature related to chiropractic and, more specifically, spinal manipulative therapy (SMT) [14]. SMT is a guideline-recommended conservative therapy used by various practitioners, including chiropractors, osteopaths, and physiotherapists worldwide, typically to treat low back pain, neck pain, and headache [15, 16]. Furthermore, the procedures and theoretical frameworks have developed quite substantially over the last century [17]. It is not unlikely that specific papers are only published in journals related to those professions and thereby only found in the corresponding database.

In contrast, searching too many databases has clear disadvantages, as the search strategy must be translated to fit different databases using different interfaces and search syntaxes, and the time spent screening more, likely irrelevant, titles and abstracts is not insignificant [18]. Which and how many databases are necessary to be searched and the added value of select databases has been the topic of many previous studies, and the main takeaway seems that it, as expected, heavily depends on the topic of interest [13, 14, 19‐30]. No research has looked systematically at retrieving relevant SMT papers. However, in the broader field of musculoskeletal disorders, Aagaard et al. [31] found MEDLINE, EMBASE, and CENTRAL to be insufficient at identifying all effect studies based on achieving a combined coverage rate of 88.9%. In an attempt to make a more generalized recommendation across all biomedical fields, Bramer et al. [32] found that searches should include EMBASE, MEDLINE, Web of Science, and Google Scholar as minimum requirements.

The Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) tool was developed in 2009 to standardizing reporting in SRs, ensuring transparency and minimizing biases [33]. PRISMA and the use of an information specialist have become imperative when conducting a high-quality SR [34‐36].

Hence, conducting a SR on a specific intervention such as SMT is not without challenge, and the selection of databases has not yet been explored sufficiently. This study will examine the frequency and choice of databases used by researchers in SRs of SMT. Secondly, to analyze the RCTs included in the SRs to determine the optimal combination of databases to conduct efficient literature searches for SRs of SMT. Finally, to examine whether the year of publication or the use of an information specialist influenced the number of investigated databases and how the use of PRISMA has changed over time.

The research protocol for this study was registered at the Open Science Framework (protocol: https://​osf.​io/​6ezxn/​?​view_​only=​3a750c9d398e4afa​895c5a5d53346aa4​).

The initial searches in PubMed and Epistemonikos yielded 1,256 results, of which 128 were duplicates. After title and abstract screening, 314 SRs were eligible for full text review. Eighty-five SRs ended up being included. The 36 SRs where we could not access full-texts were excluded as this was an exploratory study, and resources were limited for acquiring additional materials. The full process can be seen in Fig. 1.

From the 85 included SRs, 1227 RCTs were collected, and after removing 785 duplicates, 442 (36%) unique titles were manually searched in the nine databases, the five previous stated and CINAHL (via EBSCOhost), Index to Chiropractic Literature (ICL) (Chiroindex.org – Index To Chiropractic Literature), PEDro (English – PEDro), and AMED – The Allied and Complementary Medicine Database (via EBSCOhost). MANTIS was also searched by more than 20% of the SRs, but despite multiple attempts, we could not gain access to MANTIS. From a sample of 50 random RCTs, an ICC of 0.97 (95% confidence interval = 0.96–0.97) showed excellent agreement between the two assessors without the need for further training.

On average, the SRs searched 5.8 databases, commonly corresponding to the Cochrane Handbook for Systematic Reviews of Interventions (i.e., MEDLINE/PubMed, Cochrane Library, and EMBASE) [11]. The SRs contained 14.4 RCTs on average, with an IQR of 15, indicating a large variation in research available depending on the topic within SMT. The large proportion of duplicate RCTs (64%) within all the included SRs, reflect a considerable overlap with many similar SRs on SMT in general.

The single database with the highest overall coverage rate was Cochrane Library (91.6%). It also outperformed the other databases on mean, median, and 100% coverage per SR, retrieving all RCTs in 75.3% of the included SRs. Adding Google Scholar, the coverage rate increased to 94.3%, only seven short of detecting the 424 possible RCTs. Excluding Google Scholar, the combination of Cochrane Library and PEDro retrieved 93.7% of all RCTs. The best combination of three databases, Cochrane Library, Google Scholar, and PEDro, was able to retrieve all possible RCTs with a coverage rate of 95.9%. When excluding Google Scholar, the best combination was Cochrane Library, PEDro, and ICL or EMBASE, with a coverage rate of 94.3%, retrieving eight more RCTs than Cochrane Library, MEDLINE/PubMed, and EMBASE combined, as recommended by the Cochrane Handbook. Although CINAHL was used more frequently than PEDro and performed better on its own than ICL, we suggest using PEDro or ICL over CINAHL when searching multiple databases. This is mainly due to fact that PEDro and ICL performed better than CINAHL when combined with Cochrane Library or both Cochrane Library and MEDLINE/PubMed. Furthermore, CINAHL did not retrieve any unique RCTs, while PEDro and ICL retrieved five and one unique RCTs, respectively, when excluding Google Scholar from the analysis.

Bramer et al. [32] suggested that an acceptable literature search for a SR should cover at least 95% of all possible studies. This was possible using any combination of Cochrane Library, Google Scholar, and PEDro/EMBASE/ICL. However, 18 RCTs were not found in any of the nine databases investigated in this study, resulting in the highest possible coverage rate being 95.9% (94.6% when excluding Google Scholar). However, we still find our results representative for conducting a thorough search as the same 18 RCTs limited our findings. Two of the included SRs contained 12 of the 18 RCTs not found, and for these apply, that they searched in either Chinese databases or databases explicitly related to osteopathy. The rest were found in six different SRs. The major challenge was Chinese literature (n = 11). Most likely because they are only indexed in databases other than the ones we searched, although issues relating to translation cannot be ruled out. The large diversity in databases searched by the SRs, especially Asian databases, and the amount of Chinese studies not found might suggest that a wide diversity of electronic databases is required to find all relevant materials. Our findings underline this, where PEDro found most unique references when ignoring Google scholar. Further research should aim to determine the role of Asian databases when performing SRs of SMT. Moreover, authors should remember the importance of a wide diversity of electronic databases combined with additional methods than electronic databases when searching for literature. These methods include hand searching journals, conference proceedings, searching reference lists of previously conducted systematic and narrative reviews, contacting experts in the field, and searching databases related to theses and dissertations [11, 12]. Searching ongoing and unpublished studies (often referred to as gray literature) also make up an important part of a systematic literature search, but since unpublished literature was excluded from this study, we cannot provide any specific considerations.

Overall, our results suggest that, in theory, using Cochrane's recommended databases along with PEDro and ICL appears sufficient to capture more than 95% of all SMT RCTs. Supporting the results of Aagaard et al. [31], who concluded that searching MEDLINE, EMBASE, and CENTRAL were insufficient when searching for musculoskeletal disorders. In their study, adding PEDro or ICL did not improve their search. However, their scope was much broader than ours. It is not unlikely that our findings can be extrapolated into manual therapy in general, as different types of interventions typically share (1) journals, (2) keywords, and (3) professions who administrate them. However, this is entirely speculative. Given our findings, we suggest that when performing reviews related to specific professions (e.g., chiropractic), selecting profession-specific databases (e.g., PEDro or ICL) in addition to Cochrane's recommended databases may provide more unique RCTs. Likewise, a review with another profession-oriented approach than chiropractic or physiotherapy could arguably exchange the ICL/PEDro for another profession-related database (e.g., Osteopathic Research Web for osteopaths).

Ranking databases based on coverage rates presents some challenges. The presence of a relevant study in a database does not automatically correspond to that study being found by the search strategy used (e.g., the selected keywords). This limitation becomes evident in the case of Google Scholar. Google Scholar achieved the second-highest single database coverage rate of 88.2% and was a part of the combinations with the highest coverage rates. Also, Google Scholar was able to identify the most unique RCTs (six) of all databases. Despite these impressive results, it has previously been reported that the precision of Google Scholar is low [40, 41]. Because of this and other limitations in its search functions, it has been assessed to be inadequate as a standalone database and should rather be used in addition to traditional databases [43]. An example could be to search for gray literature and quality assurance.

Only 16 (19%) of the SRs reported the use of an information specialist contradicting general guidelines [11] and suggestions from previous studies [34, 35]. However, this may be the result of under-reporting [35]. We highly suggest using an information specialist when conducting a SR since it enhances the quality of the SR [35, 44]. We would also remind researchers to report the use of information specialists when used, and acknowledge their work, either as an author, if they qualify for that according to the Vancouver guidelines, or in acknowledgments [45].

The increased number of databases searched and the increased use of PRISMA in recent years may reflect a tendency towards more emphasis on thorough methodology and transparency. This may explain the large number of duplicates found. Earlier SRs may be of such low quality that the information disserted is inapplicable to clinical practice, and newer SRs provide a more thorough and detailed dissertation. While this is speculative, some evidence suggests that the manual therapy professions have provided higher quality research in recent years [46, 47].

Cochrane Library had the single highest overall coverage rate and contained the third most unique RCTs of the nine databases investigated. The combination which performed best excluding Google Scholar, was Cochrane Library, PEDro, Index to Chiropractic Literature and either EMBASE, MEDLINE/PubMed, or CINAHL, with a coverage of 94.6%.

For studies related to SMT, we suggest following the recommendations by the Cochrane Handbook searching Cochrane Library, MEDLINE, and EMBASE and adding PEDro and Index to Chiropractic Literature. In addition, Google Scholar might also be used to search gray literature and quality assurance or can be included in the search strategy depending on authors’ available research time and ambition.

Researchers should apply these results to select the most relevant databases for future SMT reviews. Furthermore, our findings should be translated to other areas of manual therapy.