Evidence of surgical outcomes fluctuates over time: results from a cumulative meta-analysis of laparoscopic versus open appendectomy for acute appendicitis

We systemically searched the MEDLINE (PubMed), EMBASE, and CINAHL databases for articles in all languages that described RCTs published between 1991, when laparoscopic appendectomy was initiated, and September 2014. In MEDLINE, we utilized the CRD/Cochrane Highly Sensitive Search Strategy [11] with the search terms “appendectomy” and “appendicitis.” We performed the EMBASE search strategy to optimize sensitivity and specificity [12] with the terms “appendectomy” or “appendicitis.” We searched the CINAHL database using a strategy in which terms with the best optimization of sensitivity and specificity [13] were combined with “appendectomy” or “appendicitis.” Reference lists of the review articles and previously published meta-analyses were searched by hand. The search was last done on December 18, 2014.

Our inclusion criteria were as follows: (1) prospective RCTs, (2) studies comparing laparoscopic surgery and open surgery for acute appendicitis, (3) studies with human adult participants, and (4) studies written in any language. We excluded studies with any of the following characteristics: (1) pediatric participants, (2) comparisons of diagnostic efficacy, and (3) assessment of the effectiveness of variations of standard laparoscopic techniques, such as the single trocar technique versus the standard technique.

Outcomes included the incidence of intra-abdominal abscess, the incidence of wound infection, the operative time, and the length of hospital stay. We adopted these four outcome measures because these are most frequently measured in RCTs addressing this topic.

We assessed the risk of bias with respect to adequate sequence generation, allocation concealment, blinding, incomplete outcome data addressed, and selective reporting. Two authors (TU and HT) assessed the studies that met the inclusion criteria (Table 1).

Binary data were extracted for the incidences of intra-abdominal abscess and wound infection, and continuous data were extracted for the operative time and length of hospital stay. Two authors (TU and HT) independently undertook this process, and disagreements were resolved through discussion. Participants who were converted intraoperatively from laparoscopic appendectomy to open appendectomy were included in the laparoscopic appendectomy arm on an intention-to-treat basis.

A meta-analysis was performed using Review Manager (RevMan) software, version 5.3.5, provided by the Cochrane Collaboration, Copenhagen, Denmark. Since a cumulative meta-analysis cannot be performed by RevMan, we used Comprehensive Meta Analysis software, version 3.3.070 (Biostat, Englewood NJ, USA). For the binary variables (i.e., the incidence of intra-abdominal abscess and wound infection), the statistical analyses were performed using the odds ratio (OR) of laparoscopic appendectomy to open appendectomy as the summary statistic. The OR point estimate was considered significant at the p < 0.05 level when the 95 % confidence interval (95 % CI) did not include the value 1. For the continuous variables (i.e., the operative time and the length of hospital stay), the statistical analyses were performed using the mean difference (MD) as the summary statistic, i.e., the time taken for an laparoscopic appendectomy subtracted by the time taken for an open appendectomy. The MD point estimate was considered significant at the p < 0.05 level if the 95 % CI did not include the value 0.

We used both the fixed-effects model and the random-effects model according to the Mantel-Haenszel method [14] for the statistical analysis. The fixed-effects model assumes the homogeneity of the true treatment effect, whereas the random-effects model accepts between-study differences in the treatment effects. The confidence interval thus tends to be wider in the random-effects model when a certain level of treatment effect heterogeneity is observed. We performed both the fixed-effects model and random-effects model, and if their results were similar, the random-effects model was adopted.

We also tested for study homogeneity by calculating I ². This value can be calculated as I ² = 100 % × (Q ‐ df)/Q, where Q is Cochran’s heterogeneity statistic and df is the degree of freedom [15]. An outcome with no events was considered a “zero cell” in the 2 × 2 table. Although correction is needed to pool the ORs of studies that include zero cells, this can influence the results and possibly introduce bias [16]. To conduct a bias-free meta-analysis, we used the Mantel-Haenszel model with a correction factor of 0.5 (0.5 was added to all cells in the 2 × 2 table when there was a zero cell).

This outcome analysis included 51 relevant studies with a total of 6,512 participants (3,273 for laparoscopic appendectomy and 3,239 for open appendectomy) (Fig. 2). The total numbers of events were 61 in the laparoscopic appendectomy group (1.80 %) and 43 in the open appendectomy group (1.30 %). The overall OR was 1.34 (95 % CI 0.92–1.94) in the fixed-effects model and 1.32 (95 % CI 0.84–2.10) in the random-effects model. The overall I ² was 6 %. A visual inspection of the funnel plot for small-study effects did not show asymmetry (Fig. 3). A cumulative meta-analysis demonstrated that the CI narrowed until it identified the first significant difference in favor of open appendectomy in the trial published in 2001 (OR 2.35, 95 % CI 1.30–4.25). However, as more studies were added, the CI shifted to the left in favor of laparoscopic appendectomy. Finally, the CI included the value 1 in 2010, and there was no significant difference (Fig. 4).

Sixty studies and 7,462 participants (3,736 for laparoscopic appendectomy and 3,726 for open appendectomy) were included for this outcome (Fig. 5). The total numbers of events were 123 in the laparoscopic appendectomy group (3.29 %) and 290 in the open appendectomy group (7.78 %). The overall OR was 0.41 (95 % CI 0.33–0.51) in the fixed-effects model and 0.47 (95 % CI 0.38–0.59) in the random-effects model. The overall I ² was 0 %. A visual inspection of the funnel plot showed slight asymmetry in small studies (Fig. 6). A cumulative meta-analysis showed that the significant difference was first observed in the seventh study in 1995, and that this trend did not change substantially with subsequent studies (Fig. 7).

There were 43 studies with a total of 4,202 participants (2,135 for laparoscopic appendectomy and 2,067 for open appendectomy) that compared the operative time between laparoscopic appendectomy and open appendectomy (Fig. 4). The average operative times were 57.3 min for laparoscopic appendectomy and 47.0 min for open appendectomy. The overall MD was 4.4 min longer for laparoscopic appendectomy (95 % CI 3.5–5.3) in the fixed-effects model and 10.1 min (95 % CI 5.9–14.3) in the random-effects model. The I ² value was 95 %. Significant heterogeneity was found, and thus a cumulative meta-analysis was not performed.

Thirty-nine studies with a total of 4,240 participants (2,165 for laparoscopic appendectomy and 2,153 for open appendectomy) were included for this outcome (Fig. 5). The average length of hospital stay was 3.21 days for laparoscopic appendectomy and 4.40 days for open appendectomy. The MD of the length of hospital stay was 1.08 days shorter for laparoscopic appendectomy (95 % CI 1.01–1.75) in the fixed-effects model and 1.12 days (95 % CI 0.77–1.47) for the random-effects model. The I ² value was 95 %. Because significant heterogeneity was found, a cumulative meta-analysis was not performed.

When there is evidence concerning the effectiveness of a medical intervention, one can reasonably conclude that no further research is needed on the topic. However, previous studies have shown that the results of meta-analyses are underused, and many RCTs are conducted even after significant evidence has been demonstrated through a meta-analysis [2, 17]. Some researchers have contended that it is unethical and a waste of resources to randomize participants in unnecessary trials, and they emphasized the importance of avoiding redundant trials. In the meta-analysis from a Cochrane review published in 2010 [10], intra-abdominal abscess was significantly more frequent in laparoscopic appendectomy than open appendectomy (albeit with moderate heterogeneity), but the present study illustrates that a significant result turned insignificant. The findings of our study thus provide an example in which large intervention effects are not always conclusive, and they fluctuate over time. Fluctuation was not found in the wound infection outcome data, and the result was consistently in favor of laparoscopic appendectomy. Penninga et al. have shown strong evidence of favoring laparoscopic appendectomy for wound infection using a trial sequential analysis [18], and our result is consistent with this.

Evidential instability can be explained by the nature of surgical interventions, which are highly complex and difficult to evaluate [6]. Surgical interventions involve many factors, including the surgeons’ skill and judgment, the skills of the treating team, the development of surgical devices, and pre- and post-surgical management. All of these factors change on a daily basis. Second, the effect of the learning curve influences outcomes [19]. For example, surgeons’ performances improve to the point of acquiring expertise as they gain training and experience. The observed shift toward favoring laparoscopic appendectomy, which we observed in later trials, might be explained by the effects of these factors. Because of this phenomenon, surgical trials may differ from pharmaceutical trials, as in the latter, theoretically efficacy does not change over time. To minimize these factors, trial designs that consider the effects of the learning curve or perioperative management should be used [20, 21].

Although we observed a shift favoring laparoscopic appendectomy in later trials in our analysis, an apparent shift in the opposite direction was observed in the early to middle period after good results were obtained for laparoscopic appendectomy in very early trials. Early trials tend to overestimate treatment effects for a variety of reasons, such as the under-reporting of disappointing results or the selection of favorable subgroups [22‐24]. However, as new interventions are disseminated and the study participant inclusion criteria are broadened, positive results become less extreme. Relevant examples can be found elsewhere [25, 26]. We assume that the results favoring open appendectomy in the early-middle period are another example of this phenomenon.

This study has several limitations. First, we observed the change from statistically significant to insignificant findings using the 95 % CI of odds ratios. There are other methods to analyze the results of meta-analyses chronologically, such as the trial sequential analysis (TSA) which takes into account random errors due to repetitive meta-analyses [27, 28]. We performed a TSA for the intra-abdominal abscess outcome, and it did not show significance throughout; i.e., the required information size was not reached and the Z-curve did not cross the trial sequential monitoring boundaries. Therefore, we cannot dismiss the possibility of random errors which brought the statistical significance in the analysis. This strengthens the importance of not relying only on the 95 % CI of the effect size and of performing a TSA to conclude the comparison.

Second, this analysis can be prone to publication bias. It is likely that trials with contradicting results would be published more often than those confirming the existing evidence. Small study effects could also have existed in our analysis. A visual inspection of the funnel plot showed slight asymmetry in small studies (Fig. 6).

Third, we did not conduct an analysis of the operative time or the length of hospital stay due to considerable heterogeneity, and a small study effect could have a substantial impact on the heterogeneity.

Fourth, studies with a high risk of bias could skew the results. We conducted subgroup analyses for trials with low and high risks of bias, and the results showed that the intra-abdominal abscess outcome after laparoscopic appendectomy was considerably more frequent among the studies with a low risk of bias compared to those with a high risk of bias, although a cumulative meta-analysis showed a similar trend toward favoring laparoscopic appendectomy [see the Additional file 1].

Fifth, the rarity of intra-abdominal abscesses may have complicated the analysis. The small number of events can increase the uncertainty. Since there are many trials with zero-events in intra-abdominal abscess, we substituted the correction factor of 0.5 to 0.01 as a sensitivity analysis to test for robustness [29]. The cumulative meta-analysis showed that statistical significance was first observed in the trial published in 2001 and it disappeared as more results accumulated, with the overall OR of 1.24 (95 % CI 0.84–1.81). The results were similar between both correction factors.

Sixth, the number of studies published each year was unbalanced. We included more trials during the time period from 1996 to 2001, which might have biased the results. Nevertheless, considering that the shift from surgeons favoring open appendectomy to those favoring laparoscopic appendectomy occurred after 2002, new findings might have been more evident if more trials had been published after 2001.

Finally, although we report herein an example demonstrating evidential instability in the surgical field, we cannot generalize this observation to all surgical interventions or other fields. Although the numbers of RCTs addressing other surgical topics are generally low [30], more evidence should be accumulated for other topics to understand the stability of evidence by means of not only cumulative meta-analyses, but also TSAs.