The use of single armed observational data to closing the gap in otherwise disconnected evidence networks: a network meta-analysis in multiple myeloma

A systematic search of the published literature and relevant conference proceedings was conducted to identify eligible studies and is reported following PRISMA guidelines. The review protocol is published in Prospero (www.​crd.​york.​ac.​uk/​PROSPERO/​display_​record.​asp?​ID=​CRD42014013405). In August 2014, the first search was carried out in MEDLINE, EMBASE and the Cochrane Library’s Central Register of Controlled Trials; the search was updated in January 2016 and February 2017 (RCTs only). Papers were first checked by title, and then underwent abstract review. Papers were required to be in English and were included if they presented (a) original studies, (b) clinical effectiveness of any (pharmacological) intervention for the treatment of RRMM, and (c) reported progression-free survival (PFS), overall survival (OS) or time to progression (TTP) as primary or secondary outcome. The analysis presented here focuses on median PFS, study details are therefore restricted to studies reporting this outcome. Phase I dose-escalation studies, studies focusing on patient samples with different or mixed treatment conditions, and studies presenting subgroup analysis of a dataset adopted from a main clinical trial were excluded. For RCTs, conference abstracts and presentations were excluded if a corresponding published paper was available or could be identified snowballing. Conference abstracts for observational studies were excluded as they were limited in intervention and outcome information and lacked evidence of scientific validation. The full electronic search strategy can be accessed in online under Additional file 1. The quality of trials included in the NMA was assessed using the Cochrane risk of bias tool from the Cochrane handbook for RCTs [32] (ÁM, JL) and an adapted Newcastle Ottawa Scale (NOS) for observational studies [33] (JL, NH). Three authors (ÁM, EH, VB) extracted data on population characteristics, intervention description, and outcome measures. Estimates of the relative effectiveness of treatments on the hazard ratio scale, along with a measure of precision (standard error) were extracted, as well as median time to event data for each trial arm.

In total, 2505 papers were identified. After duplicates were removed, 2195 remaining titles and abstracts were screened for relevance. Of those, 1466 papers were excluded leading to 729 studies eligible for full-text reading. In total, 36 RCTs and 114 observational studies fulfilled the inclusion criteria and were used for data extraction. The PRISMA diagram is shown in Fig. 1. Excluding studies which did not report median PFS or TTP, studies investigating different doses or delivery methods of the same intervention, as well as observational studies investigating interventions not part of the RCT network or with incomplete covariate profile for single armed studies, resulted in 25 RCTs and 12 observational studies relevant for the analysis presented here. Reasons for exclusion of the remaining studies are presented in Additional file 4.

Demographic information on the trials included in the analysis is shown in Table 1; the evidence network based on RCT evidence is shown in Fig. 2.

The RCTs were of mixed quality. Many studies were un-blinded, which created a high risk of bias. Additionally, the majority of studies failed to give sufficient information regarding randomisation and allocation concealment to determine the risk of selection bias. In most cases attrition bias was treated appropriately, and only one study presented a high risk, while another presented unclear risk in this regard. All but one of the studies were subject to high risk of bias due to other factors not accounted for in the Cochrane tool, such as sponsor involvement in study design, data collection and analysis and writing, small sample size, and by being a conference abstract rather than a full text peer reviewed paper. The observational studies showed a low risk of bias, with no study scoring below 4 out of a possible 6 stars. Details on the bias assessment are provided as Additional file 5.

Twenty-five RCTs investigating 25 separate treatment regimens were analysed. Of these regimens, 13 treatment combinations are currently licensed in Europe. Since comparisons of these interventions may be of primary interest, we have highlighted these in our results.

The combined RCT evidence forms two separate evidence networks (Fig. 2). Since there was no trial investigating any of the treatment regimens from the larger white network with a treatment investigated in the smaller black network, no comparative estimates between treatments of separate networks can be obtained. The analysis was conducted separately for the white and the black network. Tables 2 and 3 shows the relative HRs and 95% credible intervals for each within network comparison.

The SUCRA score shown as the solid line in Fig. 3 provides an additional summary statistic of each treatment’s overall ranking. Rankograms showing the probability of each intervention to be ranked best, second best etc. are shown in Additional file 6.

Dara+len + dex was estimated to be the best treatment in the white network with respect to PFS, showing a significant improvement (using the 95% credible intervals) compared to all other treatments in the network. This combination was followed by the other triple combinations (carf, ixa and elo in combination with len + dex), among which no significant differences were observed, which did however, show significant improvements compared to other licensed treatments with the exception of bor + PLD. Lowest efficacy was shown by five unlicensed regimens (pom, thal, dex, ob + dex, thal+IFN), which have shown significantly lower efficacy compared to all licensed interventions (with the exception of bor versus pom). Pom + dex and bor appeared the worst ranked licensed treatments, showing no significant difference to non-licensed regimens bor + bev and bor + vor.

Dara+bor + dex was estimated to be the most efficacious treatment regimen in the black network showing a significant improvement over the remaining treatments except for carf+dex and thal+bor + dex. Three of the other licensed treatments follow (carf+dex, thal+bor + dex and bor + dex + pan), as well as elo + bor + dex with similar efficacy to bor + dex + pan. Lowest efficacy was shown by two unlicensed regimens (bor + dex + peri and bor + dex + cyc). No significant difference was found between the licenced combination bor + dex and any of the unlicensed regimens.

The rank analysis showed an increased uncertainty of bor + bev compared to other treatments in the larger white network (see figure (b) of Additional file 6). This increased uncertainty is likely due to its connection through relatively small trials to the remaining regimens. Similar effects were observed in the black network for a number of regimens.

Due to the disconnected overall network, it is not possible to draw any conclusion on between network comparisons based on RCT evidence alone.

After removing trials not reporting the relevant outcome measure or investigating interventions not part of the RCT network, only single armed evidence was left for inclusion. Twelve of these studies provided a full covariate profile and were considered for matching. Table 1 summarises the outcomes and baseline characteristics of these studies. Restricting combinations to matches between trials investigating different treatment regimens, there were a total of 56 possible matches. A distance metric incorporating median age, median number of prior treatment lines, mean baseline stage and proportion of females was calculated for each possible match; results are shown in Table 4.

A distance threshold of 0.1 was applied for the base case analysis, and this resulted in the exploration of 14% (n = 8) of possible matches, which are underlined and marked in bold in Table 4. Table 5 summarises these 8 matched studies included in the analysis.

Five studies had no matched pair below the threshold and were not included in the base case [40‐44]. Of the eight matches explored, one strengthens the within white network evidence, two the within black network evidence and five matches connect both networks allowing for a comparison between all treatment regimens. The evidence network including these 8 matches is shown in Fig. 4.

We first explored the impact of including matches strengthening the within network evidence (match 1 for white and matches 2 and 3 for black network (Table 5)), matches connecting both networks are explored in a second step.

The grey dotted line in Fig. 3 shows the SUCRA score of the first step. The impact of adding a match connecting len + dex and pom + dex in the white network is minimal indicating that the evidence added does not contradict the RCT evidence. Due to network properties, adding matches for the comparative effect of bor + dex and dex + thal only affects the relative effects of these regimens as well as bor + dex + thal. The ranking shows a decrease in SUCRA score for bor + dex + thal as well as dex + thal; however, one should note that the reordering in the ranking only affects interventions between which no significant difference was observed.

The second step analysed the matches connecting both networks. A model incorporating all five connecting matches was fitted as well as five models investigating each match in turn.

The relative HRs and 95% credible intervals of comparisons between treatments licensed in Europe based on the model incorporating all five connections is displayed in Table 6, the SUCRA plot is shown in Fig. 5. All pairwise comparisons including those of unlicensed treatments can be found as Additional file 7.

Dara+len + dex is estimated the most efficacious treatment showing significant improvement compared to all other licensed interventions. Dara+bor + dex, carf+len + dex, ixa + len + dex, elo + len + dex and carf+dex follow, showing no significant differences between each other. With the exception of carf+dex, these treatments show superiority over the remaining strategies, with the exception of thal+bor + dex. Pom + dex, bor and bor + dex show the least efficacy of all licensed treatments and no significant differences among each other. In the ranking of all investigated interventions, eight of the licensed interventions show the highest efficacy and none of the unlicensed strategies show a significant improvement over any of the licensed strategies. However, some of the unlicensed strategies such as elo + bor + dex or bor + bev appear to have similar effects to many licensed treatment pathways.

The SUCRA score of licensed treatments of all scenarios individually are shown as Additional file 8. While the relative ranking of treatments within each network remains unchanged, considerable variation on inter-network comparisons is observed. However, dara+len + dex remains best or second-best treatment strategy in all scenarios (dara+len + dex is exceeded by dara+bor + dex in match 8). Dara+bor + dex also remains in the top 5 treatment strategies in all scenarios. The triple combinations with len + dex are also ranked in the top 5, except for scenario 5, where carf+dex, and triple combinations with bor + dex are ranked higher.

To validate our analysis we compared our outcomes with those of existing inter network comparisons (see outcomes summarised in Table 7). While no gold standard exists, two analyses based on individual patient level data have estimated the relative effect between bor + dex and bor with respect to PFS [29, 39] representing the currently best available evidence. Two recent NMAs made assumptions based on clinical opinion on the same comparison. None of the studies significantly favours either strategy. The two NMAs assume equal efficacy of both interventions, while both individual patient level data studies show a tendency favouring bor + dex. While the point estimate in our study favours bor, there is a large overlap in the confidence intervals. Further, our analysis shows the highest variance in the estimate, which is appropriate given the risk of matching single armed studies, especially based on summary data.

The sensitivity of choice of threshold has been evaluated for this application previously [38]. The analysis investigated the trade-off between strict thresholds, which would reduce the number of matches explored and therefore potentially underestimate the uncertainty, and high thresholds, which may include matches of trials with very different patient populations. A threshold of 0.1 appeared to explore a reasonable level of uncertainty. In addition, we analysed the similarity between arms within RCT trials using the same metric. The analysis was restricted to those studies which report a full covariate profile for each arm. Results indicate that different arms of the same study have an average distance of 0.01 ranging from 0.00 to 0.03. This indicates that a threshold of 0.1 allows for the inclusion of matched pairs which are less similar compared to different arms within an RCT. Only match 5 would be considered if a threshold in line with the distances observed within RCT studies was applied.

Median PFS was used to compare all treatment regimens, reflecting the outcome most widely reported across studies. Analyses of other outcomes, such as overall survival may have produced different results and further research should consider additional outcomes of interest.

Observational studies identified in the initial search varied in methods, from study design through to outcome reporting, and ultimately only 12 studies were considered to supplement RCT evidence. Matching based on study level information is prone to bias, making appropriate capturing of uncertainty highly important. There is no guidance on how similar is “similar enough”. Using a low threshold may result in the underestimation of uncertainty, while a high threshold may result in matching trials too dissimilar to provide useful comparisons. While a threshold of 0.1 appears to provide a reasonable exploration of the associated uncertainty, it is worth noting that differences within RCTs are much lower. Using the same approach, we have calculated the distance between arms within RCTs included in this analysis (where data was available) and the maximum distance observed was 0.03.

We only allowed for matching observational studies with each other to avoid interfering with the RCTs (either by duplicating an arm or inserting an extra arm). Alternative to matching single armed observational evidence with each other, we could have matched observational studies directly to RCTs or connect RCTs with each other [7, 47]. The distance metric indicates similar differences for all approaches (average distance 0.17 (range 0.01–0.48) within RCTs, 0.19 (range 0.02, 0.47) RCT to observational, 0.20 (range 0.03, 0.47) within observational); however, considering a larger space of matches may improve the analysis of variation.