Small studies may overestimate the effect sizes in critical care meta-analyses: a meta-epidemiological study

Medline and Embase databases were searched from inception to August 2012. There was no language restriction. The core search terms consisted of critical care, mortality and meta-analysis (detailed search strategy is shown in Additional file 1). Inclusion criteria were as follows: critical care meta-analyses involving randomized controlled trial; the end points should include mortality; at least one component trial had more than 100 subjects per arm on average. Exclusion criteria were systematic reviews without meta-analysis; all component trials were exclusively large (sample sizes ≥100 per arm) or small trials (sample sizes <100 per arm); meta-analyses included duplicated component trials. If there were several meta-analyses addressing the same clinical issue, we included the most updated one. Two reviewers (XX and ZZ) independently assessed the literature and disagreement was settled by a third opinion (HN).

The following data were extracted from eligible meta-analyses: the lead author of the study, year of publication, number of trials, treatment strategy in the experimental arm, proportion of large trials in each meta-analysis, effect size and corresponding 95% confidence interval (CI), heterogeneity as represented by I². For each component trial, we extracted the following data: sequence generating, allocation concealment, blinding, incomplete follow-up data, intention-to-treat analysis, sample size calculation, and year of publication. Sequence generating was considered adequate when the trial reported the method to generate the randomization sequence (for example computer, randomization table). Allocation concealment was considered adequate when the investigator responsible for patient selection was unable to predict allocation of the next patient. The commonly used techniques included the use of central randomization or sequentially numbered, opaque and sealed envelopes. Blinding was considered adequate if the experimental and control interventions were described as indistinguishable by patients or investigators [5].

Small and large trials were distinguished by a cutoff of an average of 100 subjects per arm. For example, if a two-arm trial had 113 patients in one arm and 87 patients in the other, it was considered a large trial. This definition was somewhat arbitrary. However, a sample size of 200 patients allowed an 80% statistical power to detect an absolute difference of 20% for binary outcomes (assuming that the proportion in the control group was 50%) at two-sided α = 0.05. Another reason for this definition was that critical care trials were usually small, and a greater cutoff point would significantly reduce the number of meta-analyses that were eligible for the analysis.

Treatment effects were expressed as odds ratio (OR) for mortality. The number of events and total number of patients in each arm were extracted for each component trial. An OR <1 indicated beneficial effect in the experimental arm. A standard logistic regression model was used to examine whether estimated treatment effects differ according to whether a trial is large or small [6, 7]. Ratio of OR (ROR) was estimated from the regression model. ROR <1 indicates larger effect size in smaller studies and ROR >1 indicates larger effect size in large trials. ROR was calculated separately for each meta-analysis. These RORs were then combined using a meta-analytic approach. Inverse variance weighting and either fixed effect or random effects models were used to pool these RORs. Meta-analyses involving exclusively large or small trials were not included in our analysis and thus did not contribute to the analysis. Heterogeneity between trials was estimated using I². A rough guide to the interpretation of I² can be as follows: 0 to 40% represents unimportant heterogeneity; 30% to 60% represents moderate heterogeneity; 50% to 90% represents substantial heterogeneity and 75% to 100% represents considerable heterogeneity [8]. To account for the difference of estimated effects between large and small trials, the qualities of study reporting including sequence generating, blinding, allocation concealment, incomplete follow-up data, sample size calculation and intention to treat were compared between large and small trials. The proportions of large and small trials were compared based on the year of publication before and after 2002. This was defined arbitrarily. However, we feel that multicenter large trials have increased rapidly in the last 10 years. We analyzed the association between sample size and treatment effects, stratified according to the significance of effect size and heterogeneity within each meta-analysis. All statistical analyses were performed using Stata software version 11.0 (StataCorp LP, College Station, TX, USA). Statistical significance was considered at two-sided P <0.05.

Our initial search identified 371 citations. Of them, 329 were excluded by reviewing the title and abstract because they were duplicate meta-analyses, included non-randomized trials, did not report data on mortality, and other irrelevant articles. Full text of the remaining 42 citations was reviewed, of which 15 citations were excluded. In these excluded 15 citations, eight did not include large trials, study end point was not mortality in four meta-analyses, and three were duplicated meta-analyses. A total of 27 meta-analyses [9‐35] involving randomized controlled trials were finally included in our analysis (Figure 1).

Characteristics of included meta-analyses are shown in Table 1. All meta-analyses were published after the year 2005. These meta-analyses covered all subspecialties of critical care medicine, including mechanical ventilation, sedation, fluid resuscitation, prevention of nosocomial infection, nutrition and critical care nephrology. For the number of component trials in each meta-analysis, we only counted those that reported mortality as an end point. Of note, because one meta-analysis [34] included seven trials conducted by Joachim Boldt that had been retracted [36], Boldt's trials were excluded from analysis. Eight meta-analyses [10, 13, 17, 18, 25, 28, 31, 34] included only one large trial, and the meta-analyses by Zhongheng [35] included mostly large trials (83.3%). Most meta-analyses reported non-significant effect size, and only six meta-analyses [19, 20, 22, 25, 30, 31] reported statistically significant effect sizes (for example mortality). Eight meta-analyses [12, 15, 16, 20, 21, 30, 32, 35] reported high heterogeneity and the remaining 19 meta-analyses showed no significant heterogeneity among component trials. Seventeen meta-analyses [9‐11, 14, 17, 18, 22‐29, 31, 33, 34] reported an I² of 0%, most of which were meta-analyses without significant effect sizes.

RORs were estimated by logistic regression model separately for each meta-analysis (Figure 2). The RORs and relevant 95% CI are shown in the left column of Figure 2. Five meta-analyses [12, 19‐22] showed statistically significant RORs, indicating significantly larger beneficial effects in small studies; nine meta-analyses [10, 11, 13, 15, 16, 26, 27, 31, 33] showed RORs >1, but without statistical significance; the remaining 13 meta-analyses showed RORs <1, again without statistical significance. RORs were combined using a meta-analytic approach with inverse variance weighting. A fixed effect model was used to combine the RORs. Overall, the pooled ROR was 0.60 (95% CI: 0.53 to 0.68); the heterogeneity was moderate with an I² of 50.3% (chi-squared = 52.30; P = 0.002). Meta-regression was performed to test whether the observed RORs were dependent on the quality of meta-analyses. Covariates including concealment (P = 0.88), blinding (P = 0.82), intention to treat (P = 0.72), sequence generating (P = 0.48) and sample size calculation (P = 0.57) cannot explain the heterogeneity between meta-analyses.

To explore possible explanations for the difference of effect sizes between large and small trials, we compared reporting qualities of large and small trials (Table 2). As expected, the large trials showed significantly better reporting quality than small trials. More large trials were well conducted than small trials in terms of sequence generating, allocation concealment, blinding, intention to treat, sample size calculation and incomplete follow-up data. For instance, 82% of large trials explicitly reported allocation concealment, while only 51% of small trials reported this (P <0.001). Intention-to-treat analysis was used in 83% of the large trials, while only 52% of the small trials used this analysis (P <0.001). Sample size calculation was performed a priori in 88% of large trials and only 44% of small trials reported sample size calculation (P <0.001). Some 75% of large trials were published after the year 2002, while 52% of small trials were published after 2002 (P <0.001).

We estimated ROR between large and small trials stratified according to the characteristics of individual meta-analysis (Table 3). Eight meta-analyses [16, 17, 19, 20, 22, 25, 30, 31] reported significant effect sizes, and the pooled ROR was 0.49 (95%: 0.38 to 0.60). The remaining 19 meta-analyses reported insignificant effect sizes, and the pooled ROR was 0.69 (95%: 0.59 to 0.79). Stratified by heterogeneity, eight meta-analyses [12, 15, 16, 20, 21, 30, 32, 35] reported high heterogeneity, their combined ROR was 0.46 (95%: 0.36 to 0.55); the remaining 19 meta-analyses showed a ROR of 0.79 (95% CI: 0.68 to 0.90). The result indicated that small-study effects were more prominent in meta-analyses of high heterogeneity.