Meta-analyses: a method to maximise the evidence from clinical studies?

Evidence-based medicine (EBM) is becoming the guiding principle for clinical treatment decisions. But evidence remains a loosely defined term. Multiple criteria for evidence criteria have been proposed. Most influential evidence criteria give priority to meta-analyses because they promise an objective procedure to combine the outcomes of all informative, putatively conflicting studies on the same issue in an overall score. However, we claim that meta-analyses are of limited informative value for the following six reasons: (1) meta-analyses are often “overpowered” with clinically irrelevant results that might emerge as highly significant; (2) there is serious concern of publication biases with “negative” studies not being published; (3) meta-analyses consider the variation in the results of the empirical studies included to be random noise, however, the variability of results across studies can be informative; (4) the result of a meta-analysis depends on the strategy used to identify the included empirical studies; (5) the quality of conclusions from meta-analyses depends on the statistical tests used to combine the results of the separate studies; (6) the qualitative conclusions drawn from the meta-analytical combination of individual studies may depend on specific design aspects of the individual studies. Thus, meta-analyses are primarily a method to generate hypotheses through an a posteriori analysis of treatment effects.