Intended for healthcare professionals

  1. Research
  2. Overlapping meta...
  3. Overlapping meta-analyses on the same topic: survey of published studies
CCBYNC Open access
Research

Overlapping meta-analyses on the same topic: survey of published studies

BMJ 2013; 347 doi: https://doi.org/10.1136/bmj.f4501 (Published 19 July 2013) Cite this as: BMJ 2013;347:f4501
  1. Konstantinos C Siontis, resident physician1,
  2. Tina Hernandez-Boussard, assistant professor2,
  3. John P A Ioannidis, professor34
  1. 1Department of Medicine and Mayo School of Graduate Medical Education, College of Medicine, Mayo Clinic, Rochester, MN, USA
  2. 2Department of Surgery, Stanford University School of Medicine, Stanford, CA, USA
  3. 3Stanford Prevention Research Center, Department of Medicine and Department of Health Research and Policy, Stanford University School of Medicine, Stanford, CA, USA
  4. 4Department of Statistics, Stanford University School of Humanities and Sciences, Stanford, CA, USA
  1. Correspondence to: J P A Ioannidis jioannid{at}stanford.edu
  • Accepted 9 July 2013

Abstract

Objective To assess how common it is to have multiple overlapping meta-analyses of randomized trials published on the same topic.

Design Survey of published meta-analyses.

Data sources PubMed.

Study selection and methods Meta-analyses published in 2010 were identified, and 5% of them were randomly selected. We further selected those that included randomized trials and examined effectiveness of any medical intervention. For eligible meta-analyses, we searched for other meta-analyses on the same topic (covering the same comparisons, indications/settings, and outcomes or overlapping subsets of them) published until February 2013.

Results Of 73 eligible meta-analyses published in 2010, 49 (67%) had at least one other overlapping meta-analysis (median two meta-analyses per topic, interquartile range 1-4, maximum 13). In 17 topics at least one author was involved in at least two of the overlapping meta-analyses. No characteristics of the index meta-analyses were associated with the potential for overlapping meta-analyses. Among pairs of overlapping meta-analyses in 20 randomly selected topics, 13 of the more recent meta-analyses did not include any additional outcomes. In three of the four topics with eight or more published meta-analyses, many meta-analyses examined only a subset of the eligible interventions or indications/settings covered by the index meta-analysis. Conversely, for statins in the prevention of atrial fibrillation after cardiac surgery, 11 meta-analyses were published with similar eligibility criteria for interventions and setting: there was still variability on which studies were included, but the results were always similar or even identical across meta-analyses.

Conclusions While some independent replication of meta-analyses by different teams is possibly useful, the overall picture suggests that there is a waste of efforts with many topics covered by multiple overlapping meta-analyses.

Introduction

Systematic reviews and meta-analyses are often considered to be the highest level in the hierarchy of evidence,1 2 and justifiably these designs have become increasingly popular.3 Meta-analysis in particular requires some extra quantitative skills in synthesizing data with sophisticated statistical methods. The ready availability of multiple meta-analysis software that can be used even by minimally trained people, however, allows such analyses to be conducted on a massive scale. A search on 29 January 2013 with the “meta-analysis” tag for “type of publication” in PubMed showed a 17-fold acceleration in the annual number of meta-analyses published between 1991 (n=334) and 2011 (n=5861). Meta-analyses of randomized trials are a large share of this literature. It is unknown whether these meta-analyses are totally different and done on different topics; represent serial updates of the same topic done by the same team of authors who want to bring their data up to date; or are on some topics that attract attention of multiple different teams of systematic reviewers who independently perform and publish their meta-analyses. Multiple independent meta-analyses on the same topic have been identified in the past for diverse topics such as gastric ulcer prophylaxis, dosing of aminoglycosides, selective decontamination of the digestive tract, orthopedic procedures, and wound healing, among others.4 5 6 7 8 9 10 11 12 13 14 Usually multiple reviews on the same topic would find the same results, but discrepancies can ensue either at the level of the objective results or, more commonly, at the level of their interpretation,12 and potential discrepancies can cause endless debates. Some meta-analyses might need updating when new evidence emerges (especially if this evidence is likely to change the conclusions),15 and independent replication by different teams is also welcome as in any field of research. It would be concerning, however, if many overlapping meta-analyses were published on the same topic by different teams. This could cause confusion and duplication in the vast data space of meta-analyses16 and would suggest potentially wasted effort. Indeed, there is some concern that such redundancy and inefficiency might be a real problem, and efforts are underway to encourage the registration of meta-analysis protocols, as in the PROSPERO initiative.17

We assessed how common it is for a published meta-analysis of randomized trials to have other published overlapping meta-analyses and investigated the characteristics of these potential redundancies.

Methods

Index meta-analyses: search and eligibility criteria

For the selection of index meta-analyses we searched PubMed for meta-analyses of randomized controlled trials published in 2010 using the search terms “randomi* AND 2010[dp] AND Meta-Analysis[ptyp]” without language restrictions. With the sample 5 command in Stata software version 11.0 (StataCorp, College Station, TX, USA), we selected a 5% random sample of the resulting items to assess for eligibility at the abstract level. Eligible for inclusion were meta-analyses of randomized trials evaluating the effectiveness of diagnostic, preventive, or therapeutic interventions for any condition. Meta-analyses including both randomized trials and observational studies were also eligible. We excluded studies pertaining to safety of interventions, prognostic associations, those with observational studies only, qualitative reviews without meta-analysis, meta-analyses pertaining to dentistry, and diagnostic accuracy meta-analyses.

Additional meta-analyses on same topic: search and eligibility criteria

Using individualized search algorithms for the topic described in each index meta-analysis we searched PubMed for overlapping meta-analyses of randomized controlled trials indexed until February 2013. Meta-analyses including both randomized trials and observational studies were also eligible. Potential redundancy was defined as overlap in terms of comparisons of interventions, type of populations, and outcomes between meta-analyses. There had to be at least one overlapping analysis (same comparison, type of population/indication, and outcome) for them to be considered as overlapping. When the index study was a Cochrane review, its updates were not considered as overlapping. When an original Cochrane review and its update were identified as overlapping, we considered only the update.

Data extraction

For each index meta-analysis, we extracted information on the intervention(s) tested, the diseases/indications assessed, the number of additional meta-analyses, and whether an author was involved in more than one meta-analysis on the same topic. For each of the index meta-analyses and the additional ones, we also noted whether it was a Cochrane review and the country of affiliation of the first author, as listed in PubMed.

We randomly selected 20 topics with overlapping meta-analyses publications and a pair of meta-analyses from each (40 articles total) and examined the extent of overlap in the outcomes and whether the more recently published meta-analysis (the one with the higher PubMed ID number) had any additional outcomes evaluated with quantitative synthesis compared with the older meta-analysis. When there were more than two overlapping meta-analyses in a topic, the pair of meta-analyses was selected randomly.

Evaluation of overlapping meta-analyses

We extracted characteristics of the index and additional meta-analyses including the publication venue (Cochrane versus other), type of condition (cardiovascular diseases, gastroenterology, neurology, hematology/oncology, psychiatry, pulmonary diseases, surgery, obstetrics and gynecology, other), type of intervention (drugs/biologics, surgical/interventional, behavioral/psychological, physical therapy, other), and countries of origin. Index meta-analyses with and without additional meta-analyses were compared by exact test on the publication venues, types of conditions, types of interventions, and countries of origin.

Topics with large number of overlapping meta-analyses

For topics for which we could identify eight or more meta-analyses, we also evaluated whether the overlapping meta-analyses differed in their eligibility criteria regarding the breadth of interventions being assessed and the eligible settings.

Use of statins for atrial fibrillation after cardiac surgery had the largest number of overlapping meta-analyses with similar eligibility criteria for type of intervention and indication/setting, and we analyzed each of the published meta-analyses in more depth. We captured information on and compared these meta-analyses regarding their publication date, date of last literature search, type of studies considered eligible (randomized trials or also observational studies), summary metric used for the effect on the main outcome (postoperative atrial fibrillation), summary treatment effect and 95% confidence interval, and which studies had been included in the calculations. For each study that had been included in the calculations of at least one meta-analysis, we evaluated whether it had been included in each meta-analysis; we also determined whether each non-inclusion was because of ineligibility of the design type, publication after the search date of the meta-analysis, or neither of these reasons. Finally, we noted whether the meta-analyses published in 2012 cited those published in previous years.

Results

Index meta-analyses

We screened 95 meta-analyses published in 2010 and excluded 22 that did not meet eligibility criteria: safety of interventions only (n=7), prognostic associations (n=5), observational studies only (n=4), qualitative reviews without meta-analysis (n=3), dentistry topics (n=2), diagnostic accuracy meta-analysis (n=1). After these exclusions, 73 index meta-analyses were eligible (see appendix). Twenty seven (37%) of those were Cochrane reviews, and the rest were meta-analyses published in peer reviewed journals. Cardiovascular diseases was the most common topic (19%), followed by gastroenterology (12%) and neurology (11%). The interventions evaluated in our sample of meta-analyses most commonly pertained to drugs or other biologics (67%). Table 1 gives details of characteristics.⇓

Table 1

 Characteristics of eligible index meta-analyses in study

View this table:

Topics with overlapping meta-analyses

There were 138 overlapping meta-analyses (see appendix), corresponding to 49 (67%) of the 73 index meta-analyses. Among these 49 topics, the median number of overlapping meta-analyses per topic was two (interquartile range 1-4), including the index meta-analysis.

Table 2 lists the outcomes evaluated in pairs of overlapping meta-analyses in 20 randomly selected topics.⇓ As shown, most of the outcomes were evaluated with quantitative synthesis in both articles of each pair. When we examined only the 27 outcomes that were stated to be the primary ones or the unique outcome assessed in a meta-analysis, 22 of them (81%) were evaluated by both articles of each pair. In 13 of the 20 pairs, the more recent meta-analysis did not include any additional clearly different outcomes beyond those already assessed in the older meta-analysis. In 17 of the 20 pairs, the more recent meta-analysis did not include any additional primary outcome that had not been evaluated in the older meta-analysis.

Table 2

 Evaluated outcomes in pairs of overlapping meta-analyses in 20 randomly selected topics

View this table:

Among the 187 overlapping meta-analyses (49 index meta-analyses published in 2010 plus their 138 overlapping counterparts), the United Kingdom was the most common country of origin (n=47), followed by the United States (n=38) and Canada (n=19). Of the 138 overlapping meta-analyses, 17 were published before 2005, 60 in 2005-09, and 61 in or after 2010.

For 17 (23%) topics at least one author was involved in at least two of the overlapping meta-analyses. In seven of the 17 topics these overlapping meta-analyses were presented as updates. Another seven pairs of meta-analyses had partially overlapping interventions, comparators, and/or indications/settings. One was a pair of a meta-analysis of published literature versus one with patient level data contributed by primary study investigators. One pair pertained to a meta-analysis of only randomized controlled trials and a meta-analysis of both randomized controlled trials and observational studies. Finally, in one pair it was unclear whether there were any differentiating characteristics.

Comparison of index meta-analyses with v without additional meta-analyses

There was no significant association between the evaluated characteristics of index meta-analyses and the potential for redundancy, including venue of publication, type of condition, type of intervention, and country of origin (table 3).⇓

Table 3

 Association between characteristics of index meta-analysis and possibility for redundancy in a topic

View this table:

Topics with many overlapping meta-analyses

There were four topics with eight or more overlapping meta-analyses (table 4).⇓ For statins for atrial fibrillation after cardiac surgery, all meta-analyses considered the same interventions (all available statins were eligible) and the same setting (cardiac surgery), except for one early meta-analysis that also included a single trial in a different setting (acute coronary syndrome). Conversely, for three other topics (chemoprevention for colorectal neoplasia, pharmacotherapy for fibromyalgia, antiepileptics for refractory epilepsy), the index meta-analysis had typically considered several drugs or interventions, while some of the other meta-analyses on the same topic considered only one of these interventions. For fibromyalgia there were also two meta-analyses that considered a wider range of causes of pain (besides fibromyalgia) than the index meta-analysis.

Table 4

 Topics with eight or more overlapping meta-analyses*

View this table:

Evaluation of 11 meta-analyses on statins for atrial fibrillation after cardiac surgery

Table 5 provides detailed information on overlapping meta-analyses of statins for the prevention of atrial fibrillation after cardiac surgery.⇓ We identified 11 overlapping meta-analyses, which were published over 57 months with a relatively steady appearance of new meta-analyses every few months. Their search dates differed over a span of 46 months. Eight of the 11 included only randomized trials, while three also included observational studies. With the exception of the first meta-analysis, all meta-analyses consistently showed a significant benefit of statins on the occurrence of postoperative atrial fibrillation, and the treatment effect was consistently large with summary risk ratios ranging between 0.54 and 0.57 and summary odds ratios ranging between 0.40 and 0.78.

Table 5

 Mapping of potential redundancy in 11 meta-analyses on use of statins for prevention of atrial fibrillation after cardiac surgery

View this table:

The main differences in included studies on statins were from eligibility criteria regarding observational studies and, to a lesser extent, the non-consideration of trials published after the search date of each meta-analysis (table 5⇑). Nevertheless, no meta-analysis included all randomized trials available during their study period: two meta-analyses did not include one trial, four others missed two trials, and five missed three or more such trials. Three meta-analyses published in 2011-12 included exactly the same eight trials and derived an identical summary effect. The three meta-analyses published in 2012 cited four, two, and two of the eight previously published meta-analyses on the same topic.

Discussion

Our empirical evaluation shows that for two thirds of the meta-analyses published in 2010 there was at least one more additional meta-analysis on the same topic that did not represent an update by the same authors in most cases. Typically more recent meta-analyses did not evaluate important additional outcomes beyond those already assessed by an older one on the same topic. Perusal of topics with eight or more meta-analyses suggested that these often differed (but still overlapped) in their inclusion criteria regarding eligible interventions, settings, and types of studies. Thus it was common for some meta-analyses to cover subsets of the evidence covered by other meta-analyses on the same topic. In depth perusal of a topic with 11 meta-analyses with similar eligible interventions and similar eligible settings showed some variability regarding which studies were included, but the results were similar. Thus some of the observed overlap in our sample seems unnecessary and can reflect wasted efforts and inefficiency in the process of summarizing evidence.

Implications of redundancy

Some potential overlap of meta-analyses is justified or even of value on grounds of necessary updating and even independent replication. There is currently no firm consensus on how and when updating should occur, but several methods have been proposed and piloted.18 19 20 21 22 Prior surveys have shown that most updated reviews are Cochrane reviews—for example, nearly 38% of 125 Cochrane reviews indexed in PubMed in November 2004 were updates, while this was true for only 2.3% of 88 reviews published in journals in the same year.3 In our analysis, we specifically excluded past versions of Cochrane updates. Nearly all overlapping meta-analyses were done by entirely independent teams. While some can claim that they also serve an updating need, there is no continuity of effort by the same team. Assuming that each independent team has to start from scratch to set the protocol, perform searches, extract data, analyze results, and write the paper, it is likely that there is a lot of duplicated effort that would be avoided if the same team were responsible for the updating. This continuity would not prohibit some partial renewal and recycling in the authors involved.

Replication of systematic reviews by entirely independent teams, however, could be useful. Replication is useful in any scientific field. Examples of discordant meta-analyses on the same topic abound in the literature.4 5 6 7 8 9 10 11 12 They often reflect differences in eligibility criteria, types of studies selected, statistical methods, occasional errors, or even diverse subjective interpretation of otherwise similar results.4 5 6 7 8 9 10 11 12 The same issues also apply in meta-analyses of non-randomized studies23 24 and their subjective interpretation.25 Most systematic reviews and meta-analyses to date are retrospective, and the data to be synthesized are already available to view informally or even scrutinize before a protocol is drafted. Therefore, replication efforts cannot be independent from each other. For many protocol decisions for a meta-analysis where different reviewer teams have acted differently, there is no clearly discernible correct versus incorrect choices. We worry that meta-analysts might sometimes try to make a case that their meta-analysis is different than others on the same topic to help publish their work. This can lead to choices on the eligibility criteria, outcomes, or methods that are not rational but are driven by the need to show some kind of novelty. Then the purpose of replication is not well served.

In our empirical evaluation, in most topics with a large number of meta-analyses, these meta-analyses differed on how broad their eligibility criteria were in terms of what interventions and settings to include. Some publications seemed to be slices of more comprehensive reviews. More inclusive publications that consider multiple, if not all, available treatment options for the same condition or even for multiple similar conditions can offer more complete pictures of the evidence and the available treatment options.26 Most meta-analyses evaluate small fragments of the evidence on a clinical question of interest.27 More inclusive designs, such as umbrella reviews28 and networks,29 might become more popular in the future. Even then, however, the problem of redundancy and inefficiency might remain—for example, there is already evidence that for some topics—such as anti-tumor necrosis factor (TNF) agents for rheumatoid arthritis—there are already multiple network meta-analyses published by independent teams.30

The topic of statins for atrial fibrillation after cardiac surgery provides an example where redundancy is most clear. Despite some differences on whether observational studies should be included or not, all meta-analyses (with the exception of the first one, which was inconclusive) showed a large effect for the intervention. More recent meta-analyses might have included some more recently published trials, but their incremental value was uncertain. Three recent meta-analyses from different authors were even identical in the studies included and the obtained summary results. One wonders whether in some cases, newer meta-analyses actually build silently on other preceding meta-analyses on the same topic—for example, by using already extracted data and results. In our example of statins for postoperative atrial fibrillation, meta-analyses published in 2012 cited only two to four of the eight previous meta-analyses.

Limitations

Our work has some limitations. First, we did not try to determine for each meta-analysis whether it was truly unnecessary or not. This would have been a subjective decision, and there might be differences in opinion about the exact utility of so many overlapping meta-analyses. Their high prevalence, however, suggests that there is substantial inefficiency in the process. Second, overlap was not always absolute, and occasionally additional outcomes were assessed. Overlap for primary outcomes, however, was high and most of the newer meta-analyses did not study any more clearly different outcomes than those already covered by older ones. Some of the overlapping meta-analyses were slices of the evidence focusing on narrow outcomes. Understanding the real merits of interventions, however, requires the full picture about the outcomes they induce. Thus, meta-analyses with limited coverage of outcomes are suboptimal, and consideration of all core outcomes is preferable.31 32

Conclusions

Despite these caveats, the high prevalence of overlapping meta-analyses on the same topic suggests that there is room for improving the efficiency of evidence synthesis worldwide. There are still many important topics for which no systematic review and meta-analysis has been performed.33 34 Conversely, for others there are already more than 10 published meta-analyses. Better coordination, communication between reviewers, and potentially registration of protocols for systematic reviews17 are options to consider. Previous meta-analyses on the same topic should also be properly acknowledged and placed into appropriate context versus a new overlapping effort. This should be part of standardized methods of meta-analysis and reporting and could be included as one of the PRISMA checklist items.35

What is already known on this topic

  • Systematic reviews and meta-analyses are popular study designs that are often considered to offer the highest level of evidence

  • The number of meta-analyses published annually is increasing steadily, and the degree of redundancy among published meta-analyses is unknown

What this study adds

  • Two thirds of published meta-analyses have at least one partially or completely overlapping meta-analysis that in most cases does not represent an update

Notes

Cite this as: BMJ 2013;347:f4501

Footnotes

  • Contributors: JPAI had the original idea, and all three authors conceived and designed the study. KCS and TH identified the eligible meta-analyses and extracted the relevant data. KCS and JPAI performed the statistical analyses, and all authors interpreted the data. KCS and JPAI wrote the manuscript, and all authors revised it critically for content and approved the final version. JPAI is guarantor.

  • Funding: This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.

  • Competing interests: All authors have completed the ICMJE uniform disclosure form at www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author) and declare: no support from any organisation for the submitted work; no financial relationships with any organisations that might have an interest in the submitted work in the previous three years; no other relationships or activities that could appear to have influenced the submitted work.

  • Ethical approval: Not required.

  • Data sharing: Datasets are available from the corresponding author.

This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/.

References

  1. ↵
    Greenhalgh T. How to read a paper: getting your bearings (deciding what the paper is about). BMJ1997;315:243.
    OpenUrlFREE Full Text
  2. ↵
    Davidoff F, Haynes B, Sackett D, Smith R: Evidence based medicine. BMJ1995;310:1085-6.
    OpenUrlFREE Full Text
  3. ↵
    Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med2007;4:e78.
    OpenUrlCrossRefPubMed
  4. ↵
    Cook DJ, Reeve BK, Guyatt GH, Heyland DK, Griffith LE, Buckingham L, et al. Stress ulcer prophylaxis in critically ill patients. Resolving discordant meta-analyses. JAMA1996;275:308-14.
    OpenUrlCrossRefPubMedWeb of Science
  5. ↵
    Jadad AR, Cook DJ, Browman GP. A guide to interpreting discordant systematic reviews. CMAJ1997;156:1411-6.
    OpenUrlAbstract/FREE Full Text
  6. ↵
    Ioannidis JP, Lau J. On meta-analyses of meta-analyses. Lancet1996;348:756.
    OpenUrlPubMedWeb of Science
  7. ↵
    Hernandez AV, Walker E, Ioannidis JP, Kattan MW. Challenges in meta-analysis of randomized clinical trials for rare harmful cardiovascular events: the case of rosiglitazone. Am Heart J2008;156:23-30.
    OpenUrlCrossRefPubMedWeb of Science
  8. ↵
    Van Nieuwenhoven CA, Buskens E, van Tiel FH, Bonten MJ. Relationship between methodological trial quality and the effects of selective digestive decontamination on pneumonia and mortality in critically ill patients. JAMA2001;286:335-40.
    OpenUrlCrossRefPubMedWeb of Science
  9. ↵
    Linde K, Willich SN. How objective are systematic reviews? Differences between reviews on complementary medicine. J Royal Soc Med2003;96:17-22.
    OpenUrlAbstract/FREE Full Text
  10. ↵
    Poolman RW, Abouali JAK, Conter HJ, Bhandari M. Overlapping systematic reviews of anterior cruciate ligament reconstruction comparing hamstring autograft with bone-patellar tendon-bone autograft: Why are they different? J Bone Joint Surg Am2007;89:1542-52.
    OpenUrlCrossRef
  11. ↵
    Campbell J, Bellamy N, Gee T. Differences between systematic reviews/meta-analyses of hyaluronic acid/hyaluronan/hylan in osteoarthritis of the knee. Osteoarthritis Cartilage2007;15:1424-36.
    OpenUrlCrossRefPubMedWeb of Science
  12. ↵
    Shrier I, Boivin JF, Platt RW, Steele RJ, Brophy JM, Carnevale F, et al. The interpretation of systematic reviews with meta-analyses: an objective or subjective process? BMC Med Informatics Dec Making2008;8:19.
    OpenUrlCrossRef
  13. ↵
    Peinemann F, McGauran N, Sauerland S, Lange S. Disagreement in primary study selection between systematic reviews on negative pressure wound therapy. BMC Med Res Methodol2008;8:41.
    OpenUrlCrossRefPubMed
  14. ↵
    Chou R. Same trials, different conclusions: sorting out discrepancies between reviews on interventional procedures of the spine. Spine J2009;9:679-89.
    OpenUrlCrossRefPubMedWeb of Science
  15. ↵
    Garritty C, Tsertsvadze A, Tricco AC, Sampson M, Moher D. Updating systematic reviews: an international survey. PLoS One2010;5:e9914.
    OpenUrlCrossRefPubMed
  16. ↵
    Bastian H, Glasziou P, Chalmers I: Seventy-five trials and eleven systematic reviews a day: how will we ever keep up? PLoS Med2010;7:e1000326.
    OpenUrlCrossRefPubMed
  17. ↵
    Booth A, Clarke M, Ghersi D, Moher D, Petticrew M, Stewart L. An international registry of systematic-review protocols. Lancet2011;377:108-9.
    OpenUrlCrossRefPubMedWeb of Science
  18. ↵
    Tugwell P, Knottnerus A, Idzerda L. Updating systematic reviews—when and how? J Clin Epidemiol2011;64:933-5.
    OpenUrlCrossRefPubMed
  19. ↵
    Tsertsvadze A, Maglione M, Chou R, Garritty C, Coleman C, Lux L, et al. Updating comparative effectiveness reviews: current efforts in AHRQ’s Effective Health Care Program. J Clin Epidemiol2011;64:1208-15.
    OpenUrlCrossRefPubMed
  20. ↵
    Sutton AJ, Donegan S, Takwoingi Y, Garner P, Gamble C, Donald A. An encouraging assessment of methods to inform priorities for updating systematic reviews. J Clin Epidemiol2009;62:241-51.
    OpenUrlCrossRefPubMed
  21. ↵
    Chalmers I, Haynes B. Reporting, updating, and correcting systematic reviews of the effects of health care. BMJ1994;309:862-5.
    OpenUrlFREE Full Text
  22. ↵
    Moher D, Tsertsvadze A, Tricco AC, Eccles M, Grimshaw J, Sampson M, et al. When and how to update systematic reviews. Cochrane Database Syst Rev2008;1:MR000023.
    OpenUrlPubMed
  23. ↵
    Minozzi S, Davoli M, Bargagli AM, Amato L, Vecchi S, Perucci CA. An overview of systematic reviews on cannabis and psychosis: discussing apparently conflicting results. Drug Alcohol Rev2010;29:304-17.
    OpenUrlCrossRefPubMedWeb of Science
  24. ↵
    Thompson RL, Bandera EV, Burley VJ, Cade JE, Forman D, Freudenheim JL, et al. Reproducibility of systematic literature reviews on food, nutrition, physical activity and endometrial cancer. Public Health Nutr2008;11:10006-14.
    OpenUrl
  25. ↵
    Panagiotou OA, Ioannidis JP. Primary study authors of significant studies are more likely to believe that a strong association exists in a heterogeneous meta-analysis compared with methodologists. J Clin Epidemiol2012;65:740-7.
    OpenUrlCrossRefPubMed
  26. ↵
    Haidich AB, Pilalas D, Contopoulos-Ioannidis DG, Ioannidis JP. Most meta-analyses of drug interventions have narrow scopes and many focus on specific agents. J Clin Epidemiol2013;66:371-8.
    OpenUrlCrossRefPubMed
  27. ↵
    Ioannidis JP, Karassa FB. The need to consider the wider agenda in systematic reviews and meta-analyses: breadth, timing, and depth of the evidence. BMJ2010;341:c4875.
    OpenUrlFREE Full Text
  28. ↵
    Ioannidis JP. Integration of evidence from multiple meta-analyses: a primer on umbrella reviews, treatment networks and multiple treatments meta-analyses. CMAJ2009;181:488-93.
    OpenUrlFREE Full Text
  29. ↵
    Mills EJ, Ioannidis JP, Thorlund K, Schünemann HJ, Puhan MA, Guyatt GH. How to use an article reporting a multiple treatment comparison meta-analysis. JAMA2012;308:1246-53.
    OpenUrlCrossRefPubMedWeb of Science
  30. ↵
    Thorlund K, Druyts E, Aviña-Zubieta JA, Wu P, Mills EJ. Why the findings of published multiple treatment comparison meta-analyses of biologic treatments for rheumatoid arthritis are different: an overview of recurrent methodological shortcomings. Ann Rheum Dis2012 Oct 20 epub ahead of print.
  31. ↵
    Kirkham JJ, Gargon E, Clarke M, Williamson PR. Can a core outcome set improve the quality of systematic reviews? A survey of the Co-ordinating Editors of Cochrane review group. Trials2013;14:21.
    OpenUrlCrossRefPubMedWeb of Science
  32. ↵
    Williamson P, Clarke M. The COMET (Core Outcome Measures in Effectiveness Trials) Initiative: its role in improving Cochrane reviews. Cochrane Database Syst Rev2012;5:ED000041.
    OpenUrlPubMed
  33. ↵
    Starr M, Chalmers I, Clarke M, Oxman AD. The origins, evolution, and future of the Cochrane Database of Systematic Reviews. Int J Technol Assess Health Care2009;25(suppl 1):182-95.
    OpenUrlCrossRefPubMed
  34. ↵
    Mallett S, Clarke M. How many Cochrane reviews are needed to cover existing evidence on the effects of health care interventions? ACP J Club2003;139:A11.
    OpenUrlPubMed
  35. ↵
    Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ2009;339:b2535.
    OpenUrlFREE Full Text
View Abstract
Back to top