Assessment of funnel plot asymmetry and publication bias in reproductive health meta-analyses: an analytic survey

Implementing best practices is a major goal in health services [1]. However, the identification of such practices depends on the evaluation and synthesis of a large amount of scientific information. This may be achieved by carrying out systematic reviews and meta-analyses, which have come to represent important sources of evidence-based knowledge for clinicians, policy makers and researchers [2].

A systematic review is an observational study of the scientific literature based on individual studies. It may contain meta-analyses, which are statistical procedures developed for summarizing effects across individual studies. The ideal meta-analysis should combine data appropriately to produce a more complete and meaningful estimate of the overall effect [3]. Nevertheless, despite efforts to assure high methodological standards, systematic reviews may be affected by publication bias, one of the major drawbacks of such studies and a threat to their validity. Publication bias occurs whenever the results of a set of published studies differ from the results of all the research performed on a specific topic [3]. A publication-biased meta-analysis may present an ineffective or unsafe intervention as being effective or safe, or not recommend an effective or safe intervention because the results of some studies already performed are not included. Furthermore, publication bias may be partially responsible for occasional discrepancies between the conclusions of previous meta-analyses and subsequent large multicenter trials [4].

The assessment of publication bias is a relatively new recommendation but reported in several relevant meta-analyses reporting guidelines [5‐8]. However, it has been observed that only few meta-analyses have actually evaluated publication bias (3.2% – 6.5%) [9]. It is also unclear whether reproductive health meta-analyses are affected by publication bias, since we have been unable to identify any previous reports assessing publication bias and its effects on meta-analyses in reproductive health. The objective of this survey was, therefore, to evaluate the occurrence of publication bias in a collection of high quality systematic reviews on reproductive health.

This is an analytic survey carried out to evaluate the impact of publication bias on the results of meta-analyses of reproductive health interventions. Systematic reviews included in the World Health Organization (WHO) Reproductive Health Library (RHL), issue No 9, were assessed [10]. The RHL is a WHO instrument for documenting and disseminating best practices in the field. It reproduces the most relevant Cochrane systematic reviews related to reproductive health, adding some practical aspects and pertinent comments for developing country settings, as well as implications for research [10].

There are several methods of assessing the occurrence of publication bias. A common approach is based on scatter plots of the treatment effect estimated by individual studies versus a measure of study size or precision (the "funnel plot"). In this graphical representation, larger and more precise studies are plotted at the top, near the combined effect size, while smaller and less precise studies will show a wider distribution below. If there is no publication bias, the studies would be expected to be symmetrically distributed on both sides of the combined effect size line. In case of publication bias, the funnel plot may be asymmetrical, since the absence of studies would distort the distribution on the scatter plot [3].

The "trim and fill" method examines the existence of asymmetry in the funnel plot and is recommended as a tool for the assessment of the robustness of the results of meta-analyses (sensitivity analysis). The method consists of a rank-based data augmentation procedure that statistically estimates the number and location of missing studies. The main application of this method is to adjust for the possible effects of missing studies [11]. If the conclusion of the meta-analysis remains unchanged following adjustment for the publication bias, the results can be considered reasonably robust, excluding publication bias.

In the present study, funnel plot asymmetry was used to assess meta-analyses containing 10 or more trials reporting a binary outcome. In each review, the meta-analysis with the greatest number of trials was selected for evaluation. Therefore those meta-analyses may not necessarily represent the primary outcomes for the review. If two or more meta-analyses had a similar number of trials, the one listed first in the review was selected [12]. To achieve consistency across meta-analyses, endpoints were re-coded if necessary so that an effect size below 1 always indicated a beneficial effect of the intervention.

The following data were extracted from each meta-analysis: study name, subgroups within the study, data on effect size, year of publication of the most recent trial included and assessment of publication bias. After extracting the data and compiling a database, statistical analysis was performed using the Comprehensive Meta-Analysis^® software program (version 2.2.034, USA, 2006). The database was checked twice for the presence of inconsistencies. For each meta-analysis, a funnel plot, the estimated number of missing studies and the adjusted combined effect size were obtained using the "trim and fill method". This procedure was applied to both sides of each funnel plot. The model of effect (fixed-effects or random-effects) used was the same as that applied in the primary meta-analysis. Results of "trim and fill method" were validated by using Stata (Stata Corporation, College Station, TX, USA)

In Cochrane systematic reviews, the Mantel-Haenszel method is usually applied to estimate relative risk by using the Review Manager (RevMan) software program [8]. In order to comply with the requirements of the Comprehensive Meta-Analyses^® software program used in the present study, values obtained for the adjustment for publication bias were converted through the inverse variance method. In case of possible publication bias, the meta-analyses were submitted to a more detailed assessment.

The 9^th issue of the WHO Reproductive Health Library included 105 Cochrane Systematic Reviews. A total of 21 systematic reviews contained meta-analyses with 10 or more trials reporting a binary outcome. In this set of meta-analyses, the number of trials comprising each one ranged from 10 to 83 (median = 13), totaling 379 trials, whose results have been summarized. None of the reviews had reported any evaluation of publication bias or funnel plot asymmetry. During data extraction, the endpoints of one meta-analysis were re-coded to guarantee consistency across meta-analyses [13].

The main characteristics of the selected meta-analyses and the adjustments performed are shown in Table 1. According to the "trim and fill method", some degree of asymmetry in funnel plots was observed in 18 of the 21 meta-analyses evaluated (85.7%). In meta-analyses in which asymmetric funnel plot was found, the estimated number of missing studies ranged from 1 to 18 (median = 3). All summary plots of meta-analyses, together with their respective "trimmed and filled" funnel plots, are shown in Appendix 1 [See Additional file 1].

In 18 of the 21 meta-analyses evaluated, the assessment procedure and the adjustments for publication bias had no effect on the conclusions (Table 1); however, in the remaining three (14.3%, 3:21), the conclusion cannot be considered robust due to a possible publication bias [14‐16]. These results were obtained applying the same model of effect used in the primary meta-analysis (fixed-effects or random-effects), but were confirmed using the alternative method (data not shown).

Figures 1 and 2 show filled funnel plots with summary effect estimates before and after adjustment for the publication bias. Both meta-analyses included a similar number of trials and both presented asymmetric funnel plots of identified studies (open circles). After adjustment for the publication bias, the estimated number of missing studies entered into the funnel plot (filled circles) was moderate, 7 and 5 respectively. The summary of estimates obtained before (open diamond) and after the adjustment (filled diamond) indicates that, if really such a number of missing studies exists, the impact on the conclusion may not be negligible. Figure 3 also presents a filled funnel plot and, although the estimates suggest that only one study may be missing, the practical impact on the conclusion should be considered. Table 2 summarizes the practical impact of the adjustment for publication bias on conclusions in these three meta-analyses.

The main results of this analytic survey suggest that some degree of asymmetry in funnel plots is a common finding in reproductive health meta-analyses. In about 14% of the selected meta-analyses, this type of asymmetry qualitatively impacted conclusions. However, the present study also has some limitations that have to be taken into consideration for the subject to be evaluated within context. If Cochrane meta-analyses differ in their methods from other meta-analyses or if meta-analyses with fewer than ten trials differ from those with more than ten, a selection bias may exist. Despite this possible selection bias, this source of meta-analyses was chosen because of the consistency in the methodology, associated with a widely recognized standard of quality. The number of trials was established as an inclusion criterion since this factor results in the best performance of the "trim and fill" method. Moreover, although other methods are available for the assessment of asymmetry in funnel plots, no consensus has been reached with respect to the superiority of any single method. Therefore, any method used for detecting asymmetry in funnel plots should be considered indirect and exploratory. In this study, we used the "trim and fill" method as an instrument for sensitivity analysis. Our principal concern was not the exact number of missing studies; we were, in fact, interested in how the effect size estimates would be qualitatively changed by the presence of an underlying publication bias.

Asymmetrical in funnel plots are linked to publication bias although there are other sources of asymmetry that have to be considered, including other dissemination biases, differences in the quality of smaller studies, the existence of true heterogeneity, and chance. Asymmetry in funnel plots may be an indicator that a more detailed investigation should be carried out on the presence of heterogeneity, such as sensitivity analysis.

Nevertheless, none of the meta-analyses evaluated in this study reported the use of sensitivity analysis. In the latest version of the software program generally used by Cochrane reviewers (RevMan, version 4.2.8, The Nordic Cochrane Centre, Rigshospitalet 2003), no formal test for the assessment of funnel plots is available. This software program permits the visual subjective interpretation of funnel plots, but such an approach has been shown to include a significant inter-observer variability [17]. These limitations may have restricted the use of this method in this selected sample of meta-analyses.

On the other hand, there is some concern regarding the evolution of effect size over time and the impact of including "old" trials in meta-analyses. In the past, it was possible that the determinants of data suppression and the intensity of publication bias were different when compared to those in current use. We observed that in one-third of meta-analyses, the most recent trials had been published prior to 1997 (8:21) and in more than half (11:21), the most recent trials had been published prior to 2000. In fact, it is unclear whether the date of publication would have any impact on meta-analyses in reproductive health, but caution should be taken when summarizing effects across older trials.

Rather than reviewing the conclusions of meta-analyses, the aim of this study was to provide evidence of publication bias and its consequences in selected reproductive health meta-analyses. Consequently, three examples of possible publication bias were identified. Following adjustment, a reduction in the discrepancy between the conclusions of larger trials and the conclusions of meta-analyses was seen (data not shown). In all three meta-analyses whose conclusions were not considered robust, their suggested post-adjustment conclusion was in agreement with those of the respective larger trials included. In the systematic review assessing interventions for emergency contraception [16], the meta-analysts adopted a different form of sensitivity analysis, by reanalyzing the data, and including only the trials that had adequate allocation concealment. Their findings were similar to the adjustment for funnel plot asymmetry or publication bias.

Asymmetry is a frequent finding in funnel plots of meta-analyses in reproductive health, but according to the present evaluation, less than 15% of meta-analyses report conclusions that would not be considered robust. Publication bias and other sources of asymmetry in funnel plots should be systematically addressed by reproductive health meta-analysts. Next amendments in Cochrane systematic reviews should include this type of evaluation. Further studies regarding the evolution of effect size and publication bias over time in systematic reviews in reproductive health are needed.