Brief Report
An analysis of systematic reviews indicated low incorpororation of results from clinical trial quality assessment

https://doi.org/10.1016/j.jclinepi.2004.07.002Get rights and content

Abstract

Objective

We investigated the frequency of quality assessment of randomized controlled trials within systematic reviews and the incorporation of the quality assessment in the analysis.

Study design and setting

We included new systematic reviews of at least five trials of therapeutic or preventive interventions that appeared in issue 2, 2003, of the Cochrane Database of Systematic Reviews. All systematic reviews in the 2002 issues of the Annals of Internal Medicine, BMJ, JAMA, and Lancet were identified in Pubmed. All reviews were assessed under unblinded conditions using preprinted extraction forms.

Results

Trial quality was assessed in all Cochrane reviews and most (74%) of the paper reviews. When we excluded 11 paper reviews that were also published as Cochrane review, the percentage remained similar (67%). Fifty percent of all Cochrane reviews and 61% of all paper reviews incorporated the results of the quality assessment in the analysis.

Conclusion

Half of the reviews did not incorporate the results of the quality assessment in the analysis. Authors, peer-reviewers, and editors should no longer focus exclusively on whether quality assessment has been performed but should also concentrate on incorporation of quality assessments in the analysis of the systematic review.

Introduction

Randomized controlled trials are the gold standard for evaluating the effectiveness of health care interventions. The main reason for conducting a systematic review based on those trials is to minimize bias and the play of chance [1]. Because clinical trials of poor methodologic quality tend to overestimate treatment effects by 30% to 50% [2], [3], quality assessment of trials within systematic reviews is important. We investigated the frequency of quality assessment of randomized controlled trials within systematic reviews and the incorporation of the quality assessment in the analysis.

Section snippets

Methods

We included new systematic reviews of at least five trials of therapeutic or preventive interventions that appeared in issue 2, 2003, of the Cochrane Database of Systematic Reviews (CDSR). Updated systematic reviews in the CDSR were not included. All systematic reviews in the 2002 issues of the general and internal medicine journals Annals of Internal Medicine, BMJ, JAMA, and Lancet were identified in Pubmed with search terms “meta-analysis [pt]” or (“review [pt]” combined with systematic [text

Results

Table 1 shows that trial quality was assessed in all Cochrane reviews and most (74%) of the paper reviews. When we excluded 11 paper reviews that were also published as Cochrane review, the percentage remained similar (67%). In the Cochrane reviews, the results of the quality assessment were always systematically (e.g., in a table) presented and nearly always commented upon in the text (98%), whereas about two thirds of the paper reviews presented the results systematically (71%) or commented

Discussion

In 1999, Moher et al. [4] reported a similar survey of quality assessment in systematic reviews published between 1977 and 1995. They observed that trial quality was assessed in half of the systematic reviews. Hence, compared with the mid 1990s, the situation has improved. The Quality of Reports of Meta-analysis statement [5] and expansion of the Cochrane Collaboration might be responsible for this improvement. The discrepancy between the high frequency of trial quality assessment and reporting

Acknowledgments

We thank Carol Manion, Family Health International, Research Triangle Park, Durham, North Carolina, for help with designing the search strategy to identify eligible publications.

References (10)

  • C.D. Mulrow

    Rationale for systematic reviews

    BMJ

    (1994)
  • K.F. Schulz et al.

    Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials

    JAMA

    (1995)
  • K. Linde et al.

    Impact of study quality on outcome in placebo-controlled trials of homeopathy

    J Clin Epidemiol

    (1999)
  • D. Moher et al.

    Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses

    Health Technol Assess

    (1999)
  • D. Moher et al.

    Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement

    Lancet

    (1999)
There are more references available in the full text version of this article.

Cited by (20)

  • A framework for best evidence approaches can improve the transparency of systematic reviews

    2012, Journal of Clinical Epidemiology
    Citation Excerpt :

    The idea is to ask: Would the same conclusion result if we changed our threshold for including studies? de Craen et al. [9] found that 74% (17 of 23) of Cochrane reviews published in the spring of 2003 contained such sensitivity analyses, as did 40% (11 of 28) of other reviews. Selecting an evidence prioritization strategy involves a number of tradeoffs.

  • Searching for unpublished trials in Cochrane reviews may not be worth the effort

    2009, Journal of Clinical Epidemiology
    Citation Excerpt :

    By means of computer-generated random numbers, we took a random sample of the reviews including unpublished data. Based on other reports, investigating the effect of unpublished trials in meta-analyses [3,8,11,12], we estimated that 20% of all reviews would be a feasible and sufficient sample size to provide relevant results. The reviews in the random sample were scrutinized for number and type of unpublished material.

  • Differences between systematic reviews/meta-analyses of hyaluronic acid/hyaluronan/hylan in osteoarthritis of the knee

    2007, Osteoarthritis and Cartilage
    Citation Excerpt :

    Although trial quality was assessed in all six reviews, not all of them incorporated the results in the subsequent analyses. This trend was noted in an analysis of systematic reviews in Issue 2, 2003 of the Cochrane Database of Systematic Reviews where half of the reviews did not incorporate the results of the quality assessment in the analysis99. There was significant heterogeneity among the trials included in these reviews.

  • Exercise proves effective in a systematic review of work-related complaints of the arm, neck, or shoulder

    2007, Journal of Clinical Epidemiology
    Citation Excerpt :

    Whether “high” quality was based on quality components or a summary score has no influence on the conclusions in this review. Because of the heterogeneity, the more frequently used sensitivity analysis was not possible here [40]. And because of this heterogeneity and the overall low quality, drawing firm conclusions about the efficacy of treatments becomes difficult.

  • Adjustment of meta-analyses on the basis of quality scores should be abandoned

    2006, Journal of Clinical Epidemiology
    Citation Excerpt :

    This has been demonstrated empirically [3], although it is not always the case [4]. In spite of the knowledge that quality of the individual studies is important for the results of a meta-analysis, the proportion of meta-analyses that adjust for the perceived quality of the included studies is low [5]. Many instruments have been devised to help with the assessment of study quality [6].

View all citing articles on Scopus
View full text