Figures
Abstract
Background
Publication and selective outcome reporting bias are a threat to the validity of systematic reviews. Extensive searching for additional trials in prospective trial registers could reduce this problem. We have evaluated how authors of Cochrane systematic reviews currently make use of trial registers as an additional source for the identification of potentially eligible trials.
Methodology/Principal Findings
We included 210 systematic Cochrane reviews of interventions published between 2008 and 2010 of which the protocol was first published in 2008. When prospective trial registers were searched we recorded the names of the register(s), the authors' motive(s) and if they yielded any extra trials.
In 80 reviews (38.1%) the authors had searched in one or more prospective trial register(s) of which 55% had searched in overlapping search portals and individual registers. Most frequently assessed were the MetaRegister (66.3%) and Clinicaltrials.gov (60%) which is in sharp contrast of other registers or portals like the WHO ICTRP Search Portal (20%). Reported motives to use registers were to identify ongoing trials (83.3%), to identify unpublished outcomes or trials (23.5%), to identify recently published trials (11.8%), or to identify any relevant trial (3.9%).In 28 reviews (35%) the authors had selected (ongoing) trials identified in trial registers as potentially eligible.
Discussion
Trial registers as an additional source of information are gaining acknowledgement amongst Cochrane reviewers. Nevertheless, searches seem to be inefficient as overlapping databases are frequently consulted, while the WHO ICTRP Search Portal that includes the data from all approved registers worldwide is being underused. Moreover, the emphasis is now on the identification of ongoing trials, although the prospective registers offer a broader potential. Further familiarity of registers and guidance how to search and to report will help to implement this as a common method and utilize the full potential of prospective trial registers for systematic reviews.
Citation: van Enst WA, Scholten RJPM, Hooft L (2012) Identification of Additional Trials in Prospective Trial Registers for Cochrane Systematic Reviews. PLoS ONE 7(8): e42812. https://doi.org/10.1371/journal.pone.0042812
Editor: Roberta W. Scherer, Johns Hopkins Bloomberg School of Public Health, United States of America
Received: April 10, 2012; Accepted: July 11, 2012; Published: August 15, 2012
Copyright: © van Enst et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The Dutch Cochrane Centre had received a grant of ZonMw (The Netherlands Organisation for Health Research and Development) in 2003 to establish the Netherlands Trial Register. This grant has not been used to finance the current study. Therefore, the funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: L. Hooft is the director of the Netherlands Trial Register (NTR). This does not alter the authors' adherence to all the PLoS ONE policies on sharing data and materials. The other authors have declared that no competing interests exist.
Introduction
Systematic reviews of randomized controlled trials (RCTs) are regarded as the highest level of evidence to guide decisions in healthcare. Cochrane reviews in particular are of high quality because these reviews follow explicit, transparent and systematic methods [1], [2]. The results of systematic reviews, however, can be biased when the included evidence does not offer a fair representation of all existing evidence. Empirical evidence consistently suggests that statistically significant and positive findings are more likely to be published than non-significant or negative findings [3], [4] and will take shorter time to be submitted and to get published after completion of the study [5]–[8]. When publication of trials or outcomes depends on the results, publication bias and selective outcome reporting bias may arise. This can affect the results from the meta-analysis of the review and possibly also the results of a review without any meta-analysis [9].
To minimize the effects of publication bias and outcome reporting bias review authors should perform a comprehensive search to identify all relevant trials [10], [11]. Most trials can be identified in well known biomedical databases like MEDLINE or EMBASE. Nevertheless, some trials can only be identified by the use of additional strategies like contacting experts, checking the reference lists of eligible trials, handsearching of conference proceedings, searching the Internet with web search engines like Google or searching the websites of relevant organisations [12]. These strategies may be very time consuming and still do not guarantee that all relevant trials will be found.
Recently searching in prospective trial registers can be added as another strategy to identify relevant trials. Already in 1986 it was suggested that prospective registration of trials could reduce or even resolve the problems resulting from publication bias and outcome reporting bias [11], [13], [14]. However, for a long period of time trials were not systematically registered. In September 2004, however, the International Committee of Medical Journal Editors (ICMJE) announced that they would only accept manuscripts for publication as of September 2005 if essential information about the underlying trial design had been deposited into an accepted prospective trial register before enrolment of the first patient. In November 2004, the World Health Organization (WHO) was asked by the international scientific and political community to facilitate the establishment of a network of these national clinical trials registers and to develop strict criteria for ‘registry approval’ concerning the content, quality and accessibility [15]. Currently, there are 15 registries that meet these strict international requirements [16].
The prospective registration policy of the ICMJE was adopted by many biomedical journals. Trial registration has become common and the number of registered trials has grown considerably [17]. Authors can search in the prospective trial registers for ongoing trials, for completed trials that have not published the results (yet) or to check whether the primary outcome has changed or if all outcomes have been reported. The various national or regional trial registers can be searched individually or simultaneously through search portals that include various other registers e.g. the WHO International Clinical Trial Registry Platform (ICTRP) and the MetaRegister of Current Controlled Trials.
Our objective was to evaluate how authors of Cochrane systematic reviews make use of trial registers as an additional source for the identification of potentially eligible trials.
Methods
Selection of reviews
For this study, we included reviews with a protocol published in 2008 that had been converted into a full Cochrane Review by February 2010. The Cochrane Collaboration's Information Management System (ARCHIE at archie.cochrane.org) was searched to identify all Cochrane protocols that were published in 2008 and the Database of Systematic Reviews was searched for full Cochrane reviews in February 2010. Cochrane Diagnostic Test Accuracy reviews, Cochrane Methodology reviews and Cochrane Overviews of reviews were excluded.
The publication date of protocols (2008) was chosen, because the Cochrane Handbook for Systematic Reviews of Interventions which authors are required to read and follow as a guide had added the statement that: ‘Trial registers, are the best solution to unpublished trials and the conduct of all systematic reviews should be much simplified when the use of registers becomes widespread’ at the end of 2006 [18].
Data extraction
We extracted information on three subjects. First, of each included review we extracted the applied strategies for the identification of additional potentially eligible trials and emphasized on the use of prospective trial registers. We distinguished between three methods to assess trials in a prospective trial registers:
- search portals with which one can search in various trial registers,
- national or regional registers that are approved by the ICMJE or WHO,
- non-approved registers (e.g. registers of the pharmaceutical industry, non-approved national registers or registers of specialized foundations).
Second, in case any of these three methods was applied, we searched the review for a particular motive the author had for searching in the prospective trial registers. These motives were retrospectively classified as identification of ongoing trials, identification of unpublished trials or outcomes, identification of recently completed yet unpublished trials, or identification of any relevant trial.
Third, for every review for which prospective trial registers were searched, we registered if the searches had yielded trials from prospective trial registers. We classified trials as identified in a prospective trial register when the trial identification number was reported or when the reference of the trial included a link or a reference to a prospective trial register without any other reference to a publication in a journal. Cochrane reviews distinguish among four types of references: included studies, excluded studies, ongoing studies and studies awaiting assessment. The number of reviews that had identified trials from prospective trial registers was registered for each type of reference.
Two reviewers independently extracted data. Disagreements were resolved by discussion and in case of persistent disagreement a third expert was asked to make a decision.
Results
We identified 519 protocols for Cochrane reviews that were published in 2008. Of these protocols 212 were converted into a full systematic review in the Cochrane Database of Systematic Reviews by February 2010. Two reviews were excluded because they were Diagnostic Test Accuracy reviews. The final set of systematic reviews consisted of 210 Cochrane reviews. These reviews were published in 2008 (n = 7), 2009 (n = 147) and 2010 (n = 56).
Most applied strategies to identify additional potentially eligible trials were checking the reference lists (83.3%) and contacting experts (49.0%) (table 1). In 80 of the included reviews (38.1%), the authors had searched in at least one prospective trial register either by using a search portal, a national or regional register approved by the ICMJE or WHO, or a non-approved register. Of those 80 reviews the MetaRegister of Current Controlled Trials was the most frequently used search portal (66.3%) and the WHO ICTRP search portal was used in only 20.0% (table 2). Clinicaltrials.gov was the most searched individual register (60.0%) which is in sharp contrast with other registers (table 2). In 75 reviews (93.8%) the authors had searched in a search portal or register that is approved by the ICMJE or WHO, leaving 5 reviews in which only non-approved registers were assessed.
The combinations of usage are presented in table 3. In 44 reviews (55%) both a search portal and one or more individual trial registers that are already included in the search portal had been consulted, ignoring the overlap.
In 51 of the 80 reviews (63.8%) one or more motives for searching in prospective trial registers were reported. Of the 51 reviews the motives were to identify ongoing trials (83.3%), to identify unpublished outcomes or trials (23.5%) (they either searched for unpublished outcomes (9.8%), unpublished trials (11.8%), or both (1.9%)), to identify recently published trials (11.8%), and to identify any relevant trial (3.9%).
In 28 of the 80 in which a search portal or register was used reviews (35.0%) the authors had yielded potentially eligible trials from a prospective trial register: in 4 reviews (14.3%) trials were actually included in the review, in 8 reviews (28.6%) the potentially eligible trials ended in the excluded category, in 20 reviews (71.4%) in the ongoing studies category and in 4 reviews (14.3%) in the category of studies awaiting classification (the total percentage exceeds 100% because some reviews found trials for multiple categories). In 34 there were no trials from prospective trial registers mentioned in the reference lists. Additionally, in 18 of the 80 reviews (22.5%) the results from extended strategies were some what confusingly documented such that we were not sure whether the reviewer had or had not identified the trial in a prospective register. None of the reviews explored the possible impact of publication bias.
Discussion
This study indicates that the majority of Cochrane authors tried to identify additional trials through extended search strategies. In 38.1% of the reviews this extended search involves consulting any prospective trial registers. This number is a good start but should be improved in the coming years. The emphasis of the use of prospective trial registers is now on the identification of ongoing studies, but this could be much more extensive, for example to compare the outcomes of the protocol to the outcomes in the publication.
The proportion of authors that had searched in prospective trial registers to identify additional trials is promising since trial registration and its possibility to minimize bias in systematic reviews has only received major attention since 2005. In this year the ICMJE required trials to be registered in publicly accessible databases. However, compared to other strategies, like contacting experts or checking the reference lists of eligible trials, this source seems to be underused and there still seems to be room for improvement. First, search portals can be seen as the most efficient way to identify trials as these searches in multiple register at once. In our evaluation we found that most authors searched the MetaRegister of Current Controlled Trials although the WHO ICTRP Search Portal has more underlying registers it was only consulted by 20% of the review authors. Furthermore, we found that many authors consult Clinicaltrials.gov which is also accessible through both the WHO ICTRP Search Portal and the MetaRegister. The popularity of the MetaRegister and Clinicaltrials.gov may be the consequence of the guidance of the Cochrane Handbook of 2006 which emphasized on consulting this portal and register [18]. Currently, many more registers are mentioned in the Handbook, but clear guidance on which search methods are most efficient is still lacking [19]. Second, in 55% of the searches in prospective registers, redundant work was performed by searching a portal and a register that is already incorporated by the portal. This could be the results of a lack of knowledge amongst review authors but it could also be an indication that authors have doubts about the sensitivity of a search in a search portal. Especially in more extensive and complex search strategies a search in a portal could miss studies from underlying registers and the reverse [20]. Moreover, some search portals update the registered trials only weekly or monthly [21]. To ensure completeness of the search review authors might have decided to search all sources and ignore their overlap. Future improvement of the search options and more frequent updating of the various registers could make the search portals more efficient. Finally, approved registers were used in 52 reviews, whereof only three non-western registers (not American, Australian or European). Authors should actively try to search in all registers to prevent a geographical skewed distribution of trials. The WHO ICTRP Search Portal is helpful for this purpose as it searches in western and non-western registers.
The main motive for searching in prospective trial registers was to identify ongoing trials. This is not surprising because the Cochrane Handbook for Systematic Reviews of Interventions recommends consulting prospective registers for the identification of ongoing trials [22]. However, some authors seem to work ahead of guidance and had broader purposes like identification of unpublished trials or unpublished outcomes which enable controlling for or assessing publication bias or selective outcome reporting bias. This strong feature of trial registers seems to be used only occasionally and deserves more emphasis and guidance. In addition to this subject, prospective trial registers can also be consulted to compare the primary outcomes as stated in the register to the actual published primary outcomes [23]. However, this strong feature of trial registers seems to be used only occasionally and deserves more emphasis and guidance. On the other hand, to improve this feature, the quality of trial data provided in trial registers has deficiencies and needs to be improved [24]. The approved registers are most appropriate to compare the outcomes as they fulfil strict criteria on reporting. The approved registers can easily be assessed by using the WHO ICTRP Search Portal that incorporates all approved registers.
Searching in prospective trial registers seems to be worthwhile. In 35 reviews (43.8%), at least one or more trials had been identified in a trial register as potentially eligible for the review. Most of those were included in the ongoing trials section. This may alert readers and enable them to track down such trials and update the results of the reviews for their own purposes. This also applies to trials identified in trial registers that were listed in the excluded studies section. Those trials might still be important for the reader if their study question differs from the study questions of the review. Therefore, searching prospective trial registers can help to identify relevant outcomes or trials for the reviews and contribute to the completeness of the evidence and quality of the review.
Our study has some limitations. First, we studied a cohort of Cochrane reviews which apply uniform methods and include detailed reports of the results. We assume that our results do not apply to non-Cochrane reviews. Secondly, although Cochrane authors follow strict methodological and reporting criteria, it could be that not all our items of interest were transparently reported in the review. For example, according to our data, checking the reference list occurred in 83% of the reviews. This seems low for Cochrane reviews where it is standard methodology. This implies that also other items, for example the use of prospective trial registers, could have been not transparently reported, thereby underestimating the results. Incomplete reporting can also apply for the reported motives. Finally the yield from searches in prospective trial registers was poorly and inconsistent documented in almost a quarter (22.5%) of the reviews that had consulted prospective registers or search portals. Therefore, the yield of trials retrieved from prospective trial registers in Cochrane reviews is possibly underestimated. Third, it would be very interesting to measure the effect of the inclusion of trials from prospective trial registers on the results of the review but unfortunately we had too low power (n = 4) to perform sensible analysis [25], [26]. Future research should try to measure this effect.
Our study indicates that many Cochrane authors did search in prospective trial registers which has led to the identification of relevant trials for the review. However, there seems to be room for improvement. More reviewers should search prospective trial registers and search more efficiently utilizing the full potential of prospective registers instead of focussing on identification of ongoing trials. The Cochrane Collaboration should promote the use of prospective trial registers more intensively and give more guidance to authors to increase the frequency of using prospective registers. This especially applies to the usefulness of trial registers beyond the identification of ongoing trials and to the efficiency to search the WHO ICTRP Search Portal that includes all approved national or regional registers. Coordinators of prospective trial registers and search portals could help authors and trials search coordinators of Cochrane Review Groups to make their search portals more user-friendly. These measures may ensure more frequent and efficient use of current search portals and prospective trial registers using all its potential with the ultimate goal restricting biased trial results and thereby improving evidence-based decisions in healthcare.
Acknowledgments
We would like to thank Cynthia Ho for her contribution to the data-extraction and René Spijker for his contribution to develop a search strategy for this study.
Author Contributions
Conceived and designed the experiments: WE RS LH. Performed the experiments: WE. Analyzed the data: WE LH. Contributed reagents/materials/analysis tools: WE RS LH. Wrote the paper: WE RS LH.
References
- 1. Fleming PS, Seehra J, Polychronopoulou A, Fedorowicz Z, Pandis N (2012) Cochrane and non-Cochrane systematic reviews in leading orthodontic journals: a quality paradigm? European Journal of Orthodontics doi: 10.1093/ejo/cjs016..
- 2.
Higgins JPT, Green S (2008) Cochrane Handbook for Systematic Reviews of Interventions. Chichester, England: John Wiley & Sons Ltd.
- 3. Dickersin K (1990) The existence of publication bias and risk factors for its occurrence. JAMA 263: 1385–1389.
- 4. Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, et al. (2008) Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS One 3: e3081 doi: 10.1371/journal.pone.0003081..
- 5. Hopewell S, McDonald S, Clarke M, Egger M (2007) Grey literature in meta-analyses of randomized trials of health care interventions. Cochrane Database of Systematic Reviews MR000010 doi: 10.1002/14651858.MR000010.pub3..
- 6. Hutton JL, Williamson PR (2000) Bias in meta-analysis due to outcome variable selection within studies. Journal of Applied Statistics 49: 359–370.
- 7. Ioannidis JP (1998) Effect of the statistical significance of results on the time to completion and publication of randomized efficacy trials. JAMA 279: 281–286.
- 8. Williamson PR, Gamble C, Altman DG, Hutton JL (2005) Outcome selection bias in meta-analysis. Statistical Methods in Medical Research 14: 515–524.
- 9. Kirkham JJ, Dwan KM, Altman DG, Gamble C, Dodd S, et al. (2010) The impact of outcome reporting bias in randomised controlled trials on a cohort of systematic reviews. BMJ 340: c365.
- 10. Egger M, Juni P, Bartlett C, Holenstein F, Sterne J (2003) How important are comprehensive searches and the assessment of trial quality in systematic reviews? Health Technology Assessment 7: 1–76.
- 11. Song F, Parekh S, Hooper L, Loke YK, Ryder A, et al. (2010) Dissemination and publication of research findings: an updated review of related bias. Health Technology Assessment 14: 1–193.
- 12. Blackhall K (2007) Finding studies for inclusion in systematic reviews of interventions for injury prevention - the importance of grey and unpublished literature. Injury Prevention 13: 359.
- 13. Simes RJ (1986) Publication bias: the case of an international registry of clincial trials. Journal of Clinical Oncology 4: 1529–1541.
- 14. Dickersin K (1990) The existence of publication bias and the risk factors of its occurence. JAMA 263: 1385–1389.
- 15. DeAngelis CD, Drazen JM, Frizelle FA, Haug C, Hoey J, et al. (2004) Clinical trial registration: a statement from the International Committee of Medical Journal Editors. JAMA 292: 1363–1364 doi: 10.1001/jama.292.11.1363..
- 16.
World Health Organisation (2012) International Clinical Trials Registry Platform (ICTRP) website. How to cite a record on a clinical trials register. Available: http://www.who.int/ictrp/faq/en/ Accessed 2012 July 23.
- 17. Ghersi D, Pang T (2009) From Mexico to Mali: four years in the history of clincal trial regsitration. Journal of Evidence Based Medicine 2: 1–7.
- 18.
Higgins JPT, Green S (2006) Cochrane Handbook for Systematic Reviews of Interventions. Chichester, England: John Wiley & Sons Ltd.
- 19.
Higgings JPT, Green S (2011) The Cochrane Handbook for Systematic Reviews of Interventions. Chichester, England: John Wiley & Sons Ltd.
- 20.
Duffy S, Glanville J, McCool R, Varley D, Blackhall K (2012) Clinical trial registers as a source of trials for HTAs. In press.
- 21.
World Health Organisation (2011) International Clinical Trials Registry Platform. Available: http://apps.who.int/trialsearch/ Accessed 2012 July 23.
- 22.
Lefebvre C, Manheimer E, Glanville J (2008) Searching for studies. In: Higgins JPT, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. Chichester; England: John Wiley & Sons Ltd. pp. 95–150.
- 23. Mathieu S, Boutron I, Moher D, Altman DG, Ravaud P (2009) Comparison of registered and published primary outcomes in randomized controlled trials. JAMA 302: 977–984 doi: 10.1001/jama.2009.1242..
- 24. Viergever RF, Ghersi D (2011) The quality of registration of clinical trials. PLoS One 6: e14701 doi: 10.1371/journal.pone.0014701..
- 25. Hart B, Lundh A, Bero L (2012) Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ 344: d7202.
- 26. Trinquart L, Abbe A, Ravaud P (2012) Impact of reporting bias in network meta-analysis of antidepressant placebo-controlled trials. PLoS One 7: e35219 doi: 10.1371/journal.pone.0035219..