Assessing methods for dealing with treatment switching in randomised controlled trials: a simulation study

Various methods have been used in existing literature in situations where patients depart from their randomised treatment. We refer to them here as simple methods which tend to involve only small adjustments to standard survival techniques. This section will focus on four of these, intention-to-treat, excluding or censoring patients if they switch treatments and modelling treatment as a time-varying covariate.

The two methods described here make adjustments to the hazard ratio in order to take into account patients switching from one treatment arm to the other.

The methods in this section make use of accelerated failure time (AFT) models, an alternative form of survival model to the commonly used proportional hazards model. A proportional hazards model assumes that covariates multiply the hazard by a constant, whereas an AFT model assumes that a covariate multiplies the predicted event time by a constant [19].

These methods have been referred to as randomisation-based efficacy estimators (RBEEs) [6], as they compare groups as randomised and therefore are intended to reduce biases which may be introduced by comparing groups as-treated. The method of Loeys & Goetghebeur [14] described previously is also an RBEE as it preserves the randomisation balance and the significance level from an ITT analysis.

To formally assess the various methods, a simulation study was conducted. Independent datasets were simulated with the true difference between each treatment's effect on survival known and each method applied to the data to see how well they performed in terms of bias, variability and coverage. The simulated data was designed to reflect data which is obtained from real clinical trials based on a review of recent submissions to NICE. This section contains details of the design of the simulation study.

We will first focus on four particular scenarios, 2, 6, 10 and 14. Each of these has 30% of patients with 'good' prognosis, a true treatment difference of β = 0.7 on the hazard ratio scale or e ^ψ = 2.04 on the AFT scale.

The scenarios vary in the difference in survival between 'good' and 'poor' prognosis groups, with 'good' prognosis patient's survival multiplied by 1.2 in scenarios 2 and 6 and by 3 in scenarios 10 and 14. The scenarios also differ in the probabilities of switching in 'good' and 'poor' prognosis groups, with probabilities of 10% and 25% respectively in scenarios 2 and 10 and of 50% and 75% respectively in scenarios 6 and 14. Full results from these scenarios can be found in Tables 3, 4, 5 and 6.

Figure 1 shows mean estimates and mean upper and lower confidence intervals for four simple methods (ITT, PP-EXC, PP-CENS, and TVC) and two adjusted hazard ratio methods (LK, LG). Figure 2 shows mean estimates and mean upper and lower confidence intervals for three simple methods (ITT, PP-EXC, PP-CENS) and for six accelerated failure time model methods (RT-LR, RT-COX, RT-EXP, RT-WB, BW, and WALK).

As expected, the ITT approach underestimated the true treatment effect in each of these four scenarios. This under-estimation was relatively small in the scenarios with a small proportion of switchers (2 and 10), around 0.03 - 0.04 on the hazard ratio scale in both cases. This increased to around 0.11 in scenarios 6 and 14 with a large proportion of control patients switching.

Excluding switchers from the analysis produced relatively small bias in scenarios 2, 6 and 10. However, in scenario 14, where the difference between 'good' and 'poor' prognosis groups and the proportion of switchers were both large, significant bias was seen (0.08 on the hazard ratio scale). The results from this approach are perhaps better than expected with many estimates very close to the true treatment effect, particularly in scenarios where only a small proportion of patients switch treatments. This is possibly explained by the fact that patients who switch treatments have a number of mechanisms acting on them which might cancel each other out. This will be investigated further by comparing biases in scenarios with a smaller and larger true treatment effect in the next section.

Perhaps the most striking results from these scenarios relate to the methods which give particularly large biases, suggesting they are very sensitive to the differences in prognosis between switchers and non-switchers. Of the hazard ratio methods, censoring patients at the time of switching and considering treatment as a time-dependent covariate both produced large biases, particularly when a large proportion of patients switched treatments (Scenarios 6 and 14) with mean hazard ratio estimates of 1.68 and 1.77 for censoring at switch and 2.42 and 2.58 for treatment as a time-varying covariate. These large biases are reflective of what was seen throughout the simulation study for these methods and suggest they may be inappropriate for use due their large sensitivity to even a relatively weak relationship between switching and prognosis.

The parametric method of Walker et al over-estimated the true treatment effect in all four scenarios presented here. This over-estimation was particularly significant in scenarios with a large difference in survival between 'good' and 'poor' prognosis groups (10 and 14), with mean treatment effects of 4.20 and 4.25 over double the true treatment effect of 2.04.

The Law & Kaldor and Loeys & Goetghebeur methods both gave biased estimates in these four scenarios. These biases were particularly large in scenarios with a high proportion of switchers (6 and 14). The Law & Kaldor method seems to underestimate the true treatment effect in all scenarios which is likely to be due to the way in which the method conditions on future events as described by White [13]. Therefore the assumptions made for this method are not met and biases given are likely to be less predictable for a real dataset. The Loeys & Goetghebeur method consistently overestimates the true treatment effect which is perhaps surprising given the method makes the assumption of all-or-nothing compliance, and therefore assumes that a switching patient receives more of the experimental treatment than they actually do. This means that any positive treatment effect seen will actually be due to a smaller amount of treatment than accounted for by the method, so an underestimation of the true treatment effect might be expected. The Robins and Tsiatis method when used with all tests gave very similar mean estimates of e ^ψ , not differing by any more than 0.02, on the hazard ratio scale, in these four scenarios. In all cases the mean estimate of e ^ψ was greater than the true treatment effect of 2.04, suggesting the method is consistently over-adjusting for treatment switching. The mean upper confidence limits given by the Cox model test based method were erratic, suggesting they were being unduly influenced by a few large values. There were also some estimation problems with this method, particularly in scenario 14 with 7.6% of simulations unsuccessful when estimating either e ^ψ or its upper or lower confidence limits.

Very small biases were observed from the Branson and Whitehead method, less than any other AFT method in scenarios 6, 10 and 14. The method also appears to be very robust to more extreme simulated datasets, with 100% successful estimation. Coverage for this method was lower than expected, as low as 82.6% in scenario 14. However, as discussed previously, standard errors calculated from the final regression in the algorithm tend to be too small, giving unduly narrow confidence intervals and therefore lower coverage.

The relationships between point estimates from each method in scenario 14 were further investigated through pairwise scatter plots which can be seen in Figures 3 and 4. Vertical and horizontal reference lines show the true treatment effect of β = 0.7 for adjusted hazard ratio methods in Figure 3 or e ^ψ = 2.04 for AFT methods in Figure 4.

The relationship between ITT and PP estimates appears to be fairly weak, reflecting the unpredictability of estimates due to biases in this particular scenario. The plots also further illustrate dilution of the true treatment effect when analysing patients as-randomised.

Scatter plots for scenarios 2, 6 and 10 showed similar relationships between parameter estimates.

All scenarios focussed on up to this point have had a large true treatment effect (a hazard ratio β = 0.7 or e ^ψ = 2.04). As seen previously, biases seen from excluding all switching patients from the analysis were perhaps not as large as expected. The way in which simulated datasets were generated meant that patients who switch treatments should in general have worse prognosis than those who do not, so excluding these patients from the analysis should make the control group have better survival in general and therefore reduce the observed difference between control and experimental groups. However, these switching patients also go on to receive a beneficial treatment, perhaps meaning their survival is approximately similar to the control patients who do not switch treatments. If this was the case, excluding these patients would have a relatively small effect on the estimate of the true treatment effect.

To investigate the competing factors acting upon patients who switch treatments in these simulations, we consider scenarios 9 and 13, which are identical to scenarios 10 and 14 respectively except with a smaller true treatment effect of β = 0.9 or e ^ψ = 1.23. Scenarios 9 and 10 have probabilities of 10% and 25% of switching treatments in 'good' and 'poor' prognosis groups whereas 13 and 14 have switching probabilities of 50% and 75%. Full details of these scenarios can be found in Table 2. Full results can be found in Tables 5 and 6.

In general, biases observed were greater in scenarios with a larger true treatment effect than a small effect. A notable exception to this can be seen when comparing scenarios 13 and 14 (Table 6). The bias when excluding switchers was greater in scenario 13 with a small treatment effect. This may be because patients in this scenario who switch treatment have worse prognosis but this is "corrected" to a lesser extent by the treatment they switch onto, making the control arm switchers and non-switchers less similar than in scenario 14 with a larger true treatment effect.

The Branson & Whitehead method also seems to have larger bias in scenarios with a smaller treatment effect. However, these biases are still small, with the mean estimate of e ^ψ closer to the true value than when excluding switchers in both scenarios 13 and 14. There also appears to be a greater difference between estimates given by the various Robins & Tsiatis methods when the true treatment effect is smaller as in scenario 13, although estimates are still strongly related.

Most of the methods investigated successfully gave an estimate of the treatment effect in all scenarios. However some of the methods experienced problems in certain situations.

The Walker et al parametric method was particularly unsuccessful in scenarios with a large difference in survival between 'good' and 'poor' prognosis groups and a large true treatment effect, most notably in scenario 12 where the method was successful for only 43.9% of simulated datasets. These problems may have been due to the way the method was implemented in Stata, where attempts to find a maximum likelihood estimate failed to converge. This is further evidence that the method may not be suitable for use, especially given the true treatment effect would not be known in real life.

Some estimation problems were also seen with the Robins & Tsiatis methods when used with a Cox, Weibull or exponential test. Given the similarities between estimates with each test, the logrank test would seem to be the most appropriate choice for this method as it was 100% successful for all scenarios.

As seen previously, the method of Branson & Whitehead performed well, giving particularly small biases in scenarios with a large difference in survival between 'good' and 'poor' prognosis groups and a large proportion of switchers, scenarios which other methods gave very biased estimates for (see Tables 4, 5 and 6).

One of the limitations of this method and its practical use is that estimates are given in the AFT model form which is less commonly seen in medical literature than hazard ratios from a proportional hazards model [34]. However, as seen previously if the shape parameter of the Weibull model γ is known, hazard ratios can be converted to the AFT parameter ψ.

Rearranging (7) gives the following expression for the hazard ratio β in terms of ψ and γ :

By taking the value of γ estimated in the final iteration of the IPE algorithm, a hazard ratio β can be estimated from the method using (10). The standard error of β can be calculated using the Delta method as described by Collett [19]. However, these standard errors are likely to be too small as the standard errors of ψ and γ from which they are calculated are also too small, as described previously. Note that this conversion to a hazard ratio would not be possible for the other AFT methods presented here as they do not directly estimate a shape parameter, γ, from the data.

To investigate this extension to the Branson and Whitehead method further, simulations for the scenarios focused on previously (2, 6, 10 and 14) were repeated, with γ estimated from the last iteration of the Branson & Whitehead method and used to calculate a hazard ratio and its corresponding standard error as described above. This was compared to hazard ratios from both intention-to-treat and per-protocol approaches for the same simulated data. Table 7 shows mean estimates, bias and the mean standard error for each of the four scenarios.

As seen previously, estimates from the ITT approach are biased towards the null in all four scenarios. This bias is particularly large in scenarios 6 and 14 which have a higher proportion of patients switching from the control arm.

There is very little difference between the mean hazard ratios for the PP and Branson & Whitehead methods in scenarios 2 and 6, with the PP approach giving relatively unbiased estimates due to the small difference in survival between 'good' and 'poor' prognosis patients. However, when this difference is increased in scenarios 10 and 14, the bias from the PP method increases, most notably in scenario 14 where the difference between prognosis groups is coupled with a large proportion of patients switching. The Branson & Whitehead method gives estimates close to the true treatment effect for all four scenarios. The method copes particularly well with the large potential biases in scenario 14, giving a mean hazard ratio of 0.73 compared to 0.78 and 0.81 from the PP and ITT approaches respectively.

There is a limit to the number of possible scenarios that can be looked at in any simulation study. Clearly there is a need for further simulation work involving many interesting trial variables whose spread of values, individually and in combination could be explored. Most important is the need to assess the performance of methods seen to be successful here in scenarios which violate their model assumptions. It may have been of interest to consider scenarios with even greater potential for selection bias and see how well each method performed. An even greater difference in survival between 'good' and 'poor' prognosis groups could have been introduced which should ensure that patients who switch and those who do not differ greatly in their underlying survival. In addition, the scenarios considered all reflect broadly patients with advanced disease, i.e. events are relatively common; how well the various methods perform in patient populations with less severe disease and therefore fewer events would also be of interest.

Only two true treatment effects were looked at, hazard ratios of 0.9 and 0.7 to represent relatively small and large treatment effects. More values could be investigated, possibly an even larger true effect such as a hazard ratio of 0.5, or a scenario where the treatment which patients were switching onto actually had a negative effect, so a hazard ratio of greater than 1 (or e ^ψ < 1). The second of these scenarios would involve a patient's observed survival time being shorter than their underlying event time, a situation in which the recensoring used by the Branson & Whitehead method could be adapted [35].

The method of Branson & Whitehead involves fitting parametric models to the data. In this paper a Weibull distribution was used for this, which allowed the conversion to hazard ratios as described previously. Other parametric distributions commonly used with AFT models could also be used to find an estimate of ψ although the same conversion to the hazard ratio scale would not be possible. For example the log-logistic distribution could be used which can deal with non-monotonic hazard functions unlike the Weibull [19]. Branson & Whitehead suggest that a distribution is chosen which best fits the experimental data [23]. A limitation of this work may be that we used a Weibull distribution for our application of the Branson & Whitehead method, and also used a Weibull to generate our simulated data. Potentially, this could make the Branson & Whitehead method appear artificially successful. Further work could be done to investigate how well the method would perform with different choices of distribution, or when applied to data generated from a different distribution.

In simulated datasets patient's switching times were generated from a uniform distribution meaning they were equally likely to switch at any point during their follow-up. This assumption may not be valid in all trial settings and it would be of interest to investigate other switching time distributions, perhaps where the probability of switching is expressed as a function of time since randomisation.

As discussed previously, standard errors given from the last iteration of the IPE algorithm in the Branson & Whitehead method may be too small, with bootstrapping required to give standard errors of the correct size. Given the large number of scenarios considered, and the fact that each of these required 1000 simulations, it was not possible to perform bootstrapping for every one of these. An initial investigation into this was made by repeating simulations for scenario 14 (for which the Branson & Whitehead method previously gave a relatively low coverage of 82.6%) with confidence intervals calculated from 100 bootstrapped samples using the normal approximation method. When using bootstrapping coverage improved to 94.1%.

We have not covered adjusting for baseline covariates, which can be used to control for imbalances between treatment arms (although this is unlikely in large randomised trials) [36]. Differences in baseline covariates may also account for some of the differences in switching pattern between patients, for example patients of a certain age may be more or less likely to switch treatment groups. Adjusting for these baseline covariates could therefore reduce the biases seen when using some of the simple methods. Branson & Whitehead describe how their method is easily extended by simply including variables in the models fitted as part of the IPE algorithm. Investigations could be performed into this and the extent to which adjusting for baseline covariates can reduce the selection bias observed from the simple methods.

All methods presented give one overall treatment effect and are therefore not necessarily suitable in situations where the treatment effect for patients who switch onto a treatment is not the same as for those who were initially allocated to the experimental treatment arm. This may be particularly important in disease areas such as cancer where treatment switching typically occurs upon disease progression. For example, a recent NICE appraisal of treatments for colorectal cancer [37] found treatment to be around half as effective for patients who switched onto the treatment compared to those who received it from the start of the trial. To properly deal with this situation, new methodology may be needed which gives two different estimates of treatment effect dependent on the time from randomisation or stage of disease at commencement of treatment.

Further methods for dealing with treatment switching which have been published in medical literature were not investigated. A large body of work into causal inference to adjust for post-treatment variables (of which treatment switching would be a special case) has been conducted [38], which may merit further investigation. Hernan et al [39] put forward a method in which patients are censored at the point of their treatment switch but then use inverse probability weighting to adjust for the selection bias this may introduce. Shao et al [40] build on the work of Branson and Whitehead by allowing the causal effect of treatment to differ between patients, although concerns have been raised about their method of estimation [35]. Further investigation may be needed to compare these methods with those presented in this paper.

A recent simulation study by Odondi and McNamee [41] also compared methods for adjusting for non-random complicance, including the Loeys & Goetghebeur and Robins & Tsiatis methods considered here. They concluded that all the methods they considered gave small biases, with the Robins & Tsiatis method performing the best in terms of bias and coverage. However their study differs from ours in the way data were simulated and in some of the outcome measures considered.

Another approach to the analysis of a trial of this sort would be to make use of any external information there is about a treatment. Patients in the control group who switch treatments could have their survival adjusted using this prior information to estimate the survival time they may have experienced if they had not switched. Comparisons between treatment groups could then be made as usual. This would, of course, depend on the availability and quality of external information about the treatment and also the way in which switching had been dealt with in the previous studies, if relevant [42].