Systematic Reviews and Meta Analysis
Meta-analysis of well-designed nonrandomized comparative studies of surgical procedures is as good as randomized controlled trials

https://doi.org/10.1016/j.jclinepi.2009.04.005Get rights and content

Abstract

Objective

To compare the results of meta-analysis of nonrandomized comparative studies (NRCSs) of a surgical procedure with that of randomized controlled trials (RCTs), and to assess the effect of design and conduct issues in NRCSs on measured outcomes.

Study Design and Setting

Two meta-analyses of RCTs and NRCSs (2,512 and 6,438 procedures, respectively) of laparoscopic resection for colorectal cancer were performed according to accepted protocols, and 13 outcomes common between them were compared. Odds ratios (ORs) and 95% confidence intervals (CI) for dichotomous outcomes were assessed for the degree of overlap. Continuous outcomes were compared using cumulative weighted ratios (CWRs) and percentages for which a mean and standard deviation (SD) were calculated. The effects of design and conduct issues in the meta-analysis of NRCSs on measured morbidity rates were assessed using subgroup analysis.

Results

The ORs of the three dichotomous outcomes overlapped widely. For the 10 continuous variables, the mean difference (SD) in the results of the two meta-analyses was only 5.6% (4.9%). Fulfillment of certain quality and conduct issues in the NRCSs determined the statistical homogeneity of the results of meta-analysis and their comparability with the “gold standard.”

Conclusion

Meta-analysis of well-designed NRCSs of surgical procedures is probably as accurate as that of RCTs.

Section snippets

Background

It has been argued that nonrandomized comparative studies (NRCSs) of intervention could either exaggerate or underestimate the measured magnitude of effect size [1]. However, it has also been argued that if they were well-designed cohort studies, they were more likely to be in agreement with randomized controlled trials (RCTs) than other observational studies [1]. Furthermore, and given the paucity of and the difficulties associated with conducting RCTs of surgical procedures, the absolute need

Methods

Two meta-analyses comparing the short-term outcomes after laparoscopic resection (LR) vs. conventional open resection (COR) for colorectal cancer (CRC) in RCTs and NRCSs were conducted according to accepted standards. Details of selection criteria, MeSH terms, databases used in literature search, and period of search have been published elsewhere [2], [3]. The results of these two meta-analyses were compared.

Critical appraisal

Summary of critical appraisal for the RCTs is provided in Table 1 and for the NRCSs in Tables 2. The total quality scores for the RCTs ranged from 4 to 8 out of 11 with a mean (SD) of 6.3 (1.2). For the NRCSs, the quality scores ranged from 10 to 23 out of 24 with a mean (SD) of 16.4 (2.6).

Collated data analysis

For all three dichotomous outcomes, namely mortality, reoperation, and early morbidity rates, the OR in the meta-analysis of the NRCSs overlapped widely with that of the RCTs (Fig. 1).

Morbidity rates

In the meta-analysis of

Discussion and conclusions

Although it has been argued that the results of systematic reviews of observational studies cannot be considered definitive and should be interpreted with caution [10], the results of the current meta-analysis of NRCSs of the short-term outcomes after LR vs. COR for CRC published in the English-language literature by the end of 2003 were remarkably similar to the meta-analysis of RCTs published by the end of 2002.

Reviews of evidence derived from NRCSs of medical interventions have yielded

References (19)

There are more references available in the full text version of this article.

Cited by (0)

View full text