A systematic review of individual patient data meta-analyses on surgical interventions

Surgery has advanced spectacularly in the past 50 years, but many advances have not come from carefully planned research using valid study designs [1]. Research on surgical interventions is associated with several methodological and practical challenges of which few, if any, apply only to surgery. Surgical innovation is especially demanding because many of these challenges coincide [2]. Perhaps this situation leads many surgeons to view randomized controlled trials (RCTs), although theoretically advantageous, to be too difficult and impractical to undertake, and even worse, irrelevant to their practice because of concerns about generalizability [2, 3].

The results of RCTs are usually implemented in practice by either treating or testing all patients in case of a ‘positive’ study or treating or testing no-one in case of a ‘negative’ study. Clinicians intuitively know that this approach is oversimplified because in reality some patients benefit more than average whereas others do not benefit. This may explain why around 50% of the RCTs perform subgroup analyses [4, 5]. However, misleading claims about subgroup effects based on a single study are common [6].

Investigating subgroups is a highly relevant, but complex topic because of two interrelated concerns: failure to detect a relevant subgroup effect (false negative), and a misleading claim about a subgroup effect which in reality does not exist (false positive). Both of these problems can lead to suboptimal care for patients. Subgroup effects have been extensively and fiercely debated in the clinical, epidemiological, and statistical literature, especially in the context of single trials or traditional meta-analyses based on published summary results [7‐11].

Individual patient data meta-analyses (IPDMAs) differ from traditional meta-analyses in that an IPD meta-analysis uses the ‘raw data’ of individual patients from included studies instead of the published summary results of studies in a traditional meta-analysis [12]. Compared to subgroup analyses in a single study or in a traditional meta-analysis, an IPDMA offers important potential advantages, such as: (1) increased possibilities to perform more complex statistical analyses that better match the underlying data; (2) more power compared to single studies and traditional meta-analyses; (3) higher validity of subgroup analyses by avoiding ecological bias and by taking the distribution of other patient characteristics into account; (4) improved flexibility and standardization of defining subgroups across studies; and (5) opportunities to examine the consistency of subgroup effects across studies [13‐17].

In this paper we present a systematic overview of all IPDMAs on surgical interventions published. We studied the number and types of subgroup analyses performed, and whether these subgroups analyses influenced decision-making in clinical practice.

Our systematic review of all IPDMAs on surgical interventions published so far provides an overview of the potential advantages of IPDMAs (see Table 2 for examples). In 18 (82%) of the full set of 22 surgical IPDMAs assessed, subgroup analyses were performed to examine whether certain patients benefit more from a specific treatment than others. Eight (67%) of the 12 surgical IPDMAs with significant subgroups reported on what the implications of their findings were for clinical practice. Forty-four percent (8 out of 18) of the significant subgroups were implemented in clinical guidelines.

Although many IPDMAs performed subgroup analyses, the overall treatment was usually the main focus of the paper. Only occasionally subgroup analyses were emphasized. In surgical IPDMAs, similar to IPDMAs in general [42], subgroups were often based on patient and disease characteristics. The median number of subgroups has been reported to range from 2 to 4, the maximum number of subgroups from 15 to 50 [6, 43‐45], which is comparable to our findings. Justification of subgroup analyses, the methods used to perform subgroup analyses, and power calculations for performing subgroup analyses are often not reported in IPDMAs [6, 43‐48]. However, 11 (65%) of the IPDMAs included in our study justified at least one of the subgroups on which they reported, scientific literature being the mode of justification used. This is in line with other studies that found that clinical experience or biochemical justification is rare [44, 46, 49]. Others showed that the proportion of studies that used interaction tests for at least one of their subgroups ranges from 10% to 56% [6, 43‐48], which is slightly lower compared to our findings. So far, few studies mentioned the importance of the power of subgroup analyses [6, 44, 50, 51], and reported that many reports put too much emphasis on subgroup analyses that commonly lacked statistical power. This is in agreement with the results of the present study.

To the best of our knowledge we are the first to study surgical IPDMAs, and illustrate the merits of this method within surgery. However, some potential limitations should also be discussed. First, our literature search for surgical IPDMAs was limited to IPDMAs with IPD obtained from RCTs comparing surgical interventions, excluding IPDMAs regarding drug-eluting medical devices and surgical trials in which a drug was the comparison, and records were limited to the English language. We, however, believe that our review provides a good representation of the method within the surgical field. Second, reporting bias could not be entirely excluded, since reporting of subgroup effects in scientific publications might be influenced by reviewers’ and editors’ opinions. Third, as most studies mentioned multiple subgroups, a clustering effect might occur for reporting on justification and statistical methods. Therefore, the results were reported on study level instead of individual subgroup level. Fourth, in the 12 IPDMAs of cardiovascular interventions reporting subgroup analyses, several studies included a same set of trials and the pattern of exploring heterogeneity among these studies might be similar, that is, there might be a clustering effect. As this might impact the subsequent analyses as well as the conclusion, we performed a sensitivity analysis. The outcomes and conclusions were not substantially influenced by the inclusion of studies using the same set of trials. Fifth, the time from publication to implementation of a result into a guideline or clinical practice can be highly variable, and sometimes takes even more than 10 years [52]. Most surgical IPDMAs included in the present study have only recently been published, and time to possible implementation has been rather short. Therefore, we might have underestimated the implementation of results from IPDMAs into guidelines and/or clinical practice.

One of the challenges in medicine is to rationally implement available therapies in clinical practice, in the appropriate patients at the appropriate time. Findings from IPDMAs might provide insight into opportunities to improve evidence-based treatment decisions for patients. IPDMA is an extremely powerful tool if used correctly and provides the most definitive synthesis of the available evidence, also for potential subgroups. Despite the recommendations available on reporting clinical trials and meta-analyses, such as PRISMA [18], these guidelines have not been specifically developed for IPDMAs. The development of a generally accepted guideline for reporting on IPDMAs including subgroup analyses should therefore be encouraged. This seems the only option to really improve the reporting, analyses, and claims and applicability of subgroup effects in clinical research.