Brief ReportAn analysis of systematic reviews indicated low incorpororation of results from clinical trial quality assessment
Introduction
Randomized controlled trials are the gold standard for evaluating the effectiveness of health care interventions. The main reason for conducting a systematic review based on those trials is to minimize bias and the play of chance [1]. Because clinical trials of poor methodologic quality tend to overestimate treatment effects by 30% to 50% [2], [3], quality assessment of trials within systematic reviews is important. We investigated the frequency of quality assessment of randomized controlled trials within systematic reviews and the incorporation of the quality assessment in the analysis.
Section snippets
Methods
We included new systematic reviews of at least five trials of therapeutic or preventive interventions that appeared in issue 2, 2003, of the Cochrane Database of Systematic Reviews (CDSR). Updated systematic reviews in the CDSR were not included. All systematic reviews in the 2002 issues of the general and internal medicine journals Annals of Internal Medicine, BMJ, JAMA, and Lancet were identified in Pubmed with search terms “meta-analysis [pt]” or (“review [pt]” combined with systematic [text
Results
Table 1 shows that trial quality was assessed in all Cochrane reviews and most (74%) of the paper reviews. When we excluded 11 paper reviews that were also published as Cochrane review, the percentage remained similar (67%). In the Cochrane reviews, the results of the quality assessment were always systematically (e.g., in a table) presented and nearly always commented upon in the text (98%), whereas about two thirds of the paper reviews presented the results systematically (71%) or commented
Discussion
In 1999, Moher et al. [4] reported a similar survey of quality assessment in systematic reviews published between 1977 and 1995. They observed that trial quality was assessed in half of the systematic reviews. Hence, compared with the mid 1990s, the situation has improved. The Quality of Reports of Meta-analysis statement [5] and expansion of the Cochrane Collaboration might be responsible for this improvement. The discrepancy between the high frequency of trial quality assessment and reporting
Acknowledgments
We thank Carol Manion, Family Health International, Research Triangle Park, Durham, North Carolina, for help with designing the search strategy to identify eligible publications.
References (10)
Rationale for systematic reviews
BMJ
(1994)- et al.
Empirical evidence of bias: dimensions of methodological quality associated with estimates of treatment effects in controlled trials
JAMA
(1995) - et al.
Impact of study quality on outcome in placebo-controlled trials of homeopathy
J Clin Epidemiol
(1999) - et al.
Assessing the quality of reports of randomised trials: implications for the conduct of meta-analyses
Health Technol Assess
(1999) - et al.
Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement
Lancet
(1999)
Cited by (20)
A framework for best evidence approaches can improve the transparency of systematic reviews
2012, Journal of Clinical EpidemiologyCitation Excerpt :The idea is to ask: Would the same conclusion result if we changed our threshold for including studies? de Craen et al. [9] found that 74% (17 of 23) of Cochrane reviews published in the spring of 2003 contained such sensitivity analyses, as did 40% (11 of 28) of other reviews. Selecting an evidence prioritization strategy involves a number of tradeoffs.
Searching for unpublished trials in Cochrane reviews may not be worth the effort
2009, Journal of Clinical EpidemiologyCitation Excerpt :By means of computer-generated random numbers, we took a random sample of the reviews including unpublished data. Based on other reports, investigating the effect of unpublished trials in meta-analyses [3,8,11,12], we estimated that 20% of all reviews would be a feasible and sufficient sample size to provide relevant results. The reviews in the random sample were scrutinized for number and type of unpublished material.
Differences between systematic reviews/meta-analyses of hyaluronic acid/hyaluronan/hylan in osteoarthritis of the knee
2007, Osteoarthritis and CartilageCitation Excerpt :Although trial quality was assessed in all six reviews, not all of them incorporated the results in the subsequent analyses. This trend was noted in an analysis of systematic reviews in Issue 2, 2003 of the Cochrane Database of Systematic Reviews where half of the reviews did not incorporate the results of the quality assessment in the analysis99. There was significant heterogeneity among the trials included in these reviews.
Exercise proves effective in a systematic review of work-related complaints of the arm, neck, or shoulder
2007, Journal of Clinical EpidemiologyCitation Excerpt :Whether “high” quality was based on quality components or a summary score has no influence on the conclusions in this review. Because of the heterogeneity, the more frequently used sensitivity analysis was not possible here [40]. And because of this heterogeneity and the overall low quality, drawing firm conclusions about the efficacy of treatments becomes difficult.
Adjustment of meta-analyses on the basis of quality scores should be abandoned
2006, Journal of Clinical EpidemiologyCitation Excerpt :This has been demonstrated empirically [3], although it is not always the case [4]. In spite of the knowledge that quality of the individual studies is important for the results of a meta-analysis, the proportion of meta-analyses that adjust for the perceived quality of the included studies is low [5]. Many instruments have been devised to help with the assessment of study quality [6].