Download PDF
Semin Reprod Med 2003; 21(1): 095-106
DOI: 10.1055/s-2003-39999
Copyright © 2003 by Thieme Medical Publishers, Inc., 333 Seventh Avenue, New York, NY 10001, USA. Tel.: +1(212) 584-4662

Evaluating Systematic Reviews and Meta-Analyses

James J. Schlesselman1,2 , John. A. Collins3
  • 1Department of Epidemiology and Public Health, University of Miami School of Medicine
  • 2Division of Biostatistics, Sylvester Comprehensive Cancer Center, Miami, Florida
  • 3Department of Obstetrics and Gynecology, McMaster University, Hamilton, Ontario, and Department of Obstetrics and Gynaecology, Dalhousie University, Halifax, Nova Scotia, Canada
Further Information

Publication History

Publication Date:
12 June 2003 (online)

    Permissions and Reprints

ABSTRACT

Systematic review and meta-analysis procedures make use of explicit methods to methodically search and critically appraise and synthesize the medical care research literature. The methods involve refining a clinical question, designing a search procedure to find eligible studies, and determining the validity of the eligible studies. Independent data extraction by two or more reviewers is preferred. Agreement between the reviewers with respect to relevance and validity should be measured. Meta-analysis procedures estimate an overall average effect from the individual study effects and determine whether these effects appear to measure the same relationship (that is, the studies are not heterogeneous). In the inverse variance method, which is most frequently applied, the overall effect is a weighted average of the individual study effects, where each weight is the inverse of the study variance. To evaluate a systematic review, first determine whether it addresses a question that is relevant to the patients, treatments, and outcomes that are usual in your clinical practice. Then assess the validity of the systematic review, which is reflected by quality of the individual studies, the rigor with which the systematic methods were applied, and the extent of heterogeneity. If the results of the systematic review are valid, then is the effect important enough to make a difference in your clinical practice? Applying the results to an individual patient involves the absolute treatment effect or the number needed to treat, and an awareness of the patient's specific level of risk and personal preferences.

KEYWORDS

Systematic review - meta-analysis - randomized controlled trials - risk difference - number needed to treat

REFERENCES

  • 1 Sackett D L, Strauss S E, Richardson W S, Rosenberg W, Haynes R B. Evidence-Based Medicine: How to Practice and Teach EBM. Edinburgh: Churchill Livingstone 2000
    Google Scholar
  • 2 Cook D J, Mulrow C D, Haynes R B. Systematic reviews: synthesis of best evidence for clinical decisions.  Ann Intern Med . 1997;  126 376-380
    CrossrefPubMedGoogle Scholar
  • 3 Egger M, Smith G D, Phillips A. Meta-analysis: principles and procedures.  BMJ . 1997;  315 1533-1537
    CrossrefPubMedGoogle Scholar
  • 4 Light R J, Pillemer D B. Summing Up: The Science of Reviewing Research. Boston: Harvard University Press 1984: 62-74
    Google Scholar
  • 5 Kemmeren J M, Algra A, Grobbee D E. Third-generation oral contraceptive and risk of venous thrombosis: meta-analysis.  BMJ . 2001;  323 1-9
    CrossrefPubMedGoogle Scholar
  • 6 Hennessey S, Berlin J A, Kinman J L, Margolis D J, Marcu S M, Strom B L. Risk of venous thromboembolism from oral contraceptives containing gestodene and desogestrel versus levonorgestrel: a meta-analysis and formal sensitivity analysis.  Contraception . 2001;  64 125-133
    CrossrefPubMedGoogle Scholar
  • 7 Schlesselman J J. Risk of endometrial cancer in relation to use of combined oral contraceptives: a practitioner's guide to meta-analysis.  Human Reprod . 1997;  12 1851-1863
    CrossrefPubMedGoogle Scholar
  • 8 Chalmers T C, Smith H, Blackburn B. et al . A method of assessing the quality of a randomized control trial.  Control Clin Trials . 1981;  2 31-49
    CrossrefPubMedGoogle Scholar
  • 9 Detsky A S, Naylor C D, O'Rourke K, McGeer A J, L'Abbe K A. Incorporating variations in the quality of individual randomized trials into meta-analysis.  J Clin Epidemiol . 1992;  45 255-265
    CrossrefPubMedGoogle Scholar
  • 10 Schulz K F, Chalmers I, Hayes R J, Altman D G. Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials.  JAMA . 1995;  273 408-412
    CrossrefPubMedGoogle Scholar
  • 11 Sackett D, Gent M. Controversy in counting and attributing events in clinical trials.  N Engl J Med . 1979;  301 1410-1412
    PubMedGoogle Scholar
  • 12 Moher D, Cook D J, Eastwood S, Olkin I, Rennie D, Stroup D F. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement.  Lancet . 1999;  354 1896-1900
    CrossrefPubMedGoogle Scholar
  • 13 Thompson W D, Walter S D. A reappraisal of the kappa coefficient.  J Clin Epidemiol . 1988;  41 949-958
    CrossrefPubMedGoogle Scholar
  • 14 Thompson J R. Estimating equations for kappa statistics.  Stat Med . 2001;  20 2895-2906
    CrossrefPubMedGoogle Scholar
  • 15 Fleiss J L. Statistical methods for rates and proportions. Toronto: John Wiley 1981: 212-236
    Google Scholar
  • 16 Sackett D L, Haynes R B, Tugwell P. Clinical Epidemiology: A Basic Science for Clinical Medicine. Boston: Little, Brown 1985
    Google Scholar
  • 17 LeBlanc E S, Janowsky J, Chan B KS, Nelson H D. Hormone replacement therapy and cognition: systematic review and meta-analysis.  JAMA . 2001;  285 1489-1499
    CrossrefPubMedGoogle Scholar
  • 18 Deeks J J, Altman D G, Bradburn M J. Statistical methods for examining heterogeneity and combining results from several studies in meta-analysis. In: Egger M, Davey Smith G, Altman DG, eds. Systematic Reviews in Healthcare: Meta-analysis in Context London: BMJ Publishing Group 2001: 285-312
    Google Scholar
  • 19 Greenland S. Quantitative methods in the review of epidemiologic literature.  Epidemiol Rev . 1987;  9 1-30
    PubMedGoogle Scholar
  • 20 Mantel N, Haenszel W. Statistical aspects of the analysis of data from retrospective studies of disease.  J Natl Cancer Inst . 1959;  22 719-748
    PubMedGoogle Scholar
  • 21 Breslow N E, Day N E. Statistical Methods in Cancer Research. Vol. 1: The Analysis of Case-Control Studies Lyon: International Agency for Research on Cancer 1980: 136-157
    Google Scholar
  • 22 Yusuf S, Peto R, Lewis J, Collins R, Sleight P. Beta blockade during and after myocardial infarction: an overview of the randomized trials.  Prog Cardiovasc Dis . 1985;  27 335-371
    PubMedGoogle Scholar
  • 23 Halperin M, Byar D P, Mantel N. et al . Testing for interaction in an I × J × K contingency table.  Biometrika . 1977;  64 271-275
    PubMedGoogle Scholar
  • 24 Greenland S, Salvan A. Bias in the one-step method for pooling study results.  Stat Med . 1990;  9 247-252
    PubMedGoogle Scholar
  • 25 Petitti D. Meta-analysis, Decision Analysis and Cost-Effectiveness Analysis. New York: Oxford 2000
    Google Scholar
  • 26 Thompson S G, Pocock S J. Can meta-analyses be trusted? . Lancet 1991: 338:1127-1130
    Google Scholar
  • 27 DerSimonian R, Laird N. Meta-analysis in clinical trials.  Control Clin Trials . 1986;  7 177-188
    CrossrefPubMedGoogle Scholar
  • 28 Poole C, Greenland S. Random-effects meta-analyses are not always conservative.  Am J Epidemiol . 1999;  150 469-475
    PubMedGoogle Scholar
  • 29 Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple graphical test.  BMJ . 1997;  315 629-634
    CrossrefPubMedGoogle Scholar
  • 30 Sterne J AC, Egger M, Sutton A J. Meta-analysis software. In: Egger M, Davey Smith G, Altman DG, eds. Systematic Reviews in Healthcare: Meta-analysis in Context London: BMJ Publishing Group 2001: 336-346
    Google Scholar
  • 31 Shapiro S. Meta-analysis/shmeta-analysis.  Am J Epidemiol . 1994;  140 771-778
    PubMedGoogle Scholar
  • 32 Eddy D M, Hasselblad V, Shachter R. Meta-Analysis by the Confidence Profile Method. Boston: Academic Press 1992
    Google Scholar
  • 33 Grady D, Rubin S M, Petitti D B. et al . Hormone therapy to prevent disease and prolong life in postmenopausal women.  Ann Intern Med . 1992;  117 1016-1037
    PubMedGoogle Scholar
  • 34 Torgerson D J, Bell-Syer S EM. Hormone replacement therapy and prevention of vertebral fractures: a meta-analysis of randomised trials.  BMC Musculoskelet Disord . 2001;  2 7
    CrossrefPubMedGoogle Scholar
  • 35 Torgerson D J, Bell-Syer S EM. Hormone replacement therapy and prevention of non-vertebral fractures: a meta-analysis of randomized trials.  JAMA . 2001;  285 2891-2897
    CrossrefPubMedGoogle Scholar
  • 36 Hulley S, Grady D, Bush T. et al . Randomized trial of estrogen plus progestin for secondary prevention of coronary heart disease in postmenopausal women. Heart and Estrogen/progestin Replacement Study (HERS) Research Group.  JAMA . 1998;  280 605-613
    CrossrefPubMedGoogle Scholar
  • 37 Komulainen M H, Kroger H, Tuppurainen M T. et al . HRT and Vit D in prevention of non-vertebral fractures in postmenopausal women: a 5 year randomized trial.  Maturitas . 1998;  31 45-54
    CrossrefPubMedGoogle Scholar
  • 38 Oxman A D, Cook D J, Guyatt G H. Users' guide to the medical literature VI: how to use an overview.  JAMA . 1994;  272 1367-1371
    PubMedGoogle Scholar
  • 39 Cook D J, Sackett D L, Spitzer W O. Methodologic guidelines for systematic reviews of randomized control trials in health care from the Potsdam consultation on meta-analysis.  J Clin Epidemiol . 1995;  48 167-171
    CrossrefPubMedGoogle Scholar
  • 40 Cauley J A, Seeley D G, Ensrud K, Ettinger B, Black D, Cummings S R. Estrogen replacement therapy and fractures in older women.  Ann Intern Med . 1995;  122 9-16
    PubMedGoogle Scholar
  • 41 Naessen T, Persson I, Adami H-O, Bergstrom R, Bergkvist L. Hormone replacement therapy and the risk of first hip fracture: a prospective, population-based cohort study.  Ann Intern Med . 1990;  113 95-103
    PubMedGoogle Scholar
  • 42 Grady D, Cummings S R. Postmenopausal hormone therapy for prevention of fractures: how good is the evidence?.  JAMA . 2001;  285 2909-2910
    CrossrefPubMedGoogle Scholar
  • 43 Silverman S L, Greenwald M, Klein R A, Drinkwater B L. Effect of bone density information on decisions about hormone replacement therapy: a randomized trial.  Obstet Gynecol . 1997;  89 321-325
    CrossrefPubMedGoogle Scholar
  • 44 Altman D G. A meta-analysis of hormone replacement therapy for fracture prevention.  JAMA . 2001;  286 2096-2097
    CrossrefPubMedGoogle Scholar
  • 45 Torgerson D J, Bell-Syer S EM. Hormone replacement therapy and prevention of nonvertebral fractures: a meta-analysis of randomized trials.  JAMA . 2001;  285 2891-2897
    CrossrefPubMedGoogle Scholar
  • 46 Pickar J H, Archer D F, and the Menopause Study Group. Is bleeding a predictor of endometrial hyperplasia in postmenopausal women receiving hormone replacement therapy?.  Am J Obstet Gynecol . 1997;  177 1178-1183
    PubMedGoogle Scholar
  • 47 Speroff L, Rowan J, Symons J, Genant H, Wilborn W. The comparative effect on bone density, endometrium, and lipids of continuous hormones as replacement therapy (CHART Study): a randomized controlled study.  JAMA . 1996;  276 1397-1403
    CrossrefPubMedGoogle Scholar
  • 48 Ries L AG, Eisner M P, Kosary C L, Hankey B F, Miller B A, Edwards B K. SEER Cancer Statistics Review, 1973-1998. Bethesda, MD: National Cancer Institute 2001
    Google Scholar
  • 49 Collaborative Group on Hormonal Factors in Breast Cancer. Breast cancer and hormone replacement therapy: collaborative reanalysis of data from 51 epidemiological studies of 52,705 women with breast cancer and 108,411 women without breast cancer.  Lancet . 1997;  350 1047-1059
    CrossrefPubMedGoogle Scholar
  • 50 Lelorier J, Gregoire G, Benhaddad A, Lapierre J, Derderian F. Discrepancies between meta-analyses and subsequent large randomized, controlled trials.  N Engl J Med . 1997;  337 536-542
    CrossrefPubMedGoogle Scholar
      >