A retrospective analysis of conditional power assumptions in clinical trials with continuous or binary endpoints

Clinical trials require a minimum number of subjects in order to answer the research question at hand. In addition to likely estimates of the effect size, key components of the sample size calculation are statistical significance/type I error (probability of incorrectly rejecting the null hypothesis) and statistical power (the probability of correctly detecting a significant result of a given magnitude) [1]. Many trials recruit at least the minimum required number of subjects and analyse the results after recruitment is complete.

However, trialists may wish to incorporate an interim analysis for the purpose of making an adaptation to a trial, such as stopping a trial for either efficacy or futility or modifying the sample size in a bid to maintain statistical power. Such decisions may be based on conditional power (CP), the probability of rejecting the null hypothesis at the final analysis, given the data observed at the interim analysis and a future treatment effect for remaining patients [2, 3]. Alternative designs, such as group sequential designs have well established methods including “spending” some type I error at an interim analysis in order to formally perform hypothesis tests on the accrued data and potentially stop for efficacy and/or futility.

Regardless of the assumptions used in the CP equation, it is impacted by the accruing data. An interim analysis very early on in the trial can have huge impacts on trial decisions but may be based on a very different treatment effect size than that which may have been observed at the end of the originally planned trial. Additionally, patients recruited at the start of the trial may be inherently different to those recruited at the end. New emerging medical guidance, opening additional sites in a bid to finish recruitment on time, or even protocol amendments such as a change in eligibility criteria are some of the reasons these patient populations may differ.

There has been some debate in statistical literature over the choice of future treatment effect assumption [4‐6], and each can yield very different results. Therefore, it is critical that these assumptions are understood by trialists wishing to base key decision making on designs using conditional power. This publication presents results from a re-analysis of 14 real-world published trials, with the purpose of investigating the impact of four future treatment effect assumptions in the conditional power equation.

Data was obtained from 14 real-world clinical trials from publicly funded studies in the UK and industry companies participating in the Clinical Study Data Request (CSDR) platform (www.​clinicalstudydat​arequest.​com). Re-analysis of real-world data allows observation of estimate instability during patient accrual, recruitment pace of both sites and patients, and time to availability of primary outcome data.

Trials were chosen to provide a range of sample sizes (from n = 86 to n = 2249) and length of time to follow-up for the primary endpoint (from 7 days to 2 years). Trials were restricted to primary endpoints with either continuous or binary data, as defined in the original analysis by the trial team. For standardised comparisons of trials with binary endpoints, outcome measures were converted to an odds ratio (OR). Additionally, models adjusted for variables used in the final analysis of the original trial data.

Applications for data use was made over a 12-month period, and full details can be found in the online Supplementary material. Initially, 13 publicly funded trials were identified from two National Institute for Health Research (NIHR) journals, where it is strongly encouraged to make trial data available. An additional five were requested internally through the University of Sheffield. An additional 12 industry funded trials meeting these criteria were requested from the CSDR platform. Between initial application and final data sharing agreements in place, three companies left the CSDR platform and a further three datasets were not received by the cut-off date. Data from co-primary or secondary outcomes were used where available to maximise data usage, resulting in a total of 21 distinct outcomes from 14 trials available for this re-analysis. All datasets made available for analysis were from non-adaptive fixed sample size designs.

The stability of the estimate (mean difference for continuous endpoints; log(OR) for binary endpoints) was investigated in order to understand when might be the earliest time that an interim analysis could take place, in terms of patients with data available. The end result of the original trial analysis was taken as the “true” treatment effect. Treatment estimates after every 10–20 patients (depending on the size of the trial) with data available were calculated using the same order of recruitment as in the original trial (‘original sequential order’). This was repeated using the reverse sequential order of patients as a comparison of any potential bias between patients recruited at the very start of the trial, compared to those recruited at the end. Trial participants were then randomly re-ordered 1000 times, and again treatment estimates were calculated. The median treatment estimate of the 1000 random reordering of patients was plotted alongside the original and reverse sequential orders, with 97.25% and 2.5% and 75% and 25% quantiles.

Conditional power is the probability of rejecting the null hypothesis at the final analysis, given the current data, and is calculated using the following formula [7]:where z₁ is the observed test statistic at the interim analysis, \({\hat{\sigma}}_{obs}^2\)is the observed variance of the outcome at the interim analysis which is assumed the same in both groups, \(\overset{\sim }{d}\) is the assumed future treatment difference, z_α is the critical value for the final analysis, Φ() is the standard normal cumulative distribution function, n₁ is the total sample size at the interim analysis, and n denotes the total sample size at the planning stage of the trial.

The most commonly used CP assumptions are (1) current trend (\(\overset{\sim }{d}={\hat{d}}_{obs}\); assuming the data observed so far is likely to continue for the duration of the trial), sometimes referred to as ‘observed conditional power’ [8] and (2) the hypothesised treatment effect (\(\overset{\sim }{d}={d}_{plan}\); assuming the hypothesised treatment effect used in the original sample size calculation), sometimes referred to as ‘assumed conditional power’ [8]. There is criticism in the literature regarding the current trend assumption, due to high variability early in the trial duration and potentially yielding an unstable estimate of conditional power values [6]. An alternative recommendation from the literature [7, 9] is based on optimistic confidence limits of the observed treatment effect, being the single optimistic value of the two confidence limits defined by \(\overset{\sim }{d}={\hat{d}}_{obs}\pm {Z}_{1-\frac{p}{2}}\sqrt{\frac{2{\hat{\sigma}}_{obs}^2}{n_1}}\), where \({Z}_{1-\frac{p}{2}}\)represents the \(\left(1-\frac{p}{2}\right)\) percentage point of a standard normal distribution.

Two further treatment effect assumptions were investigated in the re-analysis based on (1) the 80% optimistic confidence limit (p = 0.2) and (2) the 90% optimistic confidence limit (p = 0.1)

CP was calculated using each of the four chosen future treatment effect assumptions after every patient with completed follow-up data and plotted from patient 20 onwards. A futility boundary of 10% conditional power value is considered for CP assumption comparison in this re-analysis, although other values may be chosen, with boundaries between 10 and 40% being observed in the literature [2, 9]. The smaller value has been chosen for this re-analysis to accommodate sample size re-estimation rules based on CP decisions, with alterations to sample size often occurring below 40% CP [3, 6]. CP is calculated after every patient for illustrative purposes of CP during the trial progression. However, it should be noted that in reality this is calculated just once.

For continuous endpoints, each primary outcome value was multiplied by some constant such that the observed standard deviation in the trial equalled that used in the original sample size calculation. Following this, another constant value was added to each intervention arm patient, such that the mean difference matched that in the original sample size calculation. This procedure was repeated, altering the constant added to the intervention arm in order to reach the desired end treatment effect (as planned, smaller than planned, or zero)

For binary endpoints, the model coefficient, or log(OR), was multiplied by some constant such that the standard error after n recruited patients matched that used in the original sample size calculation. Another constant was added to the transformed coefficients in the intervention group to ensure the log(OR) matched that used in the original sample size calculation. Again, the procedure was repeated, altering the constant added to the intervention arm in order to reach the desired end treatment effect (as planned, smaller than planned, or zero)

Conditional power lines are assessed using the following objective criteria:

Ethics approval from the University of Sheffield for the secondary use of patient data was obtained on the 16 August 2019 (reference number 030485).

After considering conditional power values under these scenarios, the current trend appears to be the most favourable assumption to use when the true treatment effect is very small or no difference at all if wishing to stop for futility due to earlier timings of low observed CP values. However, the current trend assumption does not perform as well in terms of CP when the treatment effect is as planned or smaller than that planned, observing huge fluctuations, and the lowest CP values of the four assumptions across multiple trials (Figs. 3b and 4b). For this reason, trials wishing to stop early for futility may wish to use the current trend assumption and could see this as early as 20–30% trial duration under the current trend assumption (Figs. 3c and 4c) using a 10% futility boundary. However, trialists should note the early fluctuations observed with < 20% trial duration even when ∂ = ∂_plan and may wish to consider a later interim timing, when we have more information.

This effect is switched, however, when using the hypothesised assumption for the future treatment effect; performing well when the treatment effect is as planned (observed high CP values throughout) but slower than the current trend to reach a 10% futility boundary when the true treatment effect is zero. For this reason, designs making decisions based on conditional power that do not wish to stop early for futility may wish to use the hypothesised assumption for the future treatment effect for the conditional power calculations. However, should there be no effect to observe in reality, this design would need a late interim analysis (approx. 80% trial duration/information) in order to stop early for futility.

On the other hand, the optimistic confidence limit assumptions could offer a compromise between the two current accepted assumptions used in practice and could benefit designs making additional decisions at an interim analysis, such as those wishing to modify sample size to maintain power [3].

Given that it is unknown at the planning stage whether the trial will have an effect close to that planned, or no difference at all, the optimistic confidence limit assumptions could again provide an alternative to the current trend and hypothesised assumptions. Whilst it is hoped at the start of a superiority trial that a difference between groups may be observed, it is pertinent to consider the option of seeing no effect and adding a safety net to stop prior to the original planned sample size. A review of 107 publicly funded UK trials publishing results between 2006 and 2016 found that the median standardised target effect size was 0.3 (IQR: 0.2–0.38), whereas the median standardised observed effect size was 0.11 (IQR: 0.05–0.29) [12]. Therefore, an optimistic confidence limit with an interim analysis timing between 60 and 70% could be considered.

At all instances of investigated delta, trials with binary endpoints see highly variable CP values. Due to the conversion of outcome measures to an OR for standardised comparisons, variance may have been affected and contributing to the high level of fluctuations observed in CP values.

One limitation of this paper is that there is no simulation work included. The true value of the end treatment effect has been assumed to be that observed in the original trial data, whereas simulation results would allow the estimation of the true treatment effect. However, a particular strength of this paper is the use of real-world trial data. Time of patient and site recruitment, inherent differences in patient populations between the start and end of a trial, and availability of primary outcome data are all challenging to simulate. Retrospective analysis of trial data allows the observation of conditional power values with realistic inputs of trial progression.

This paper illustrates the importance of thinking about the future treatment effect assumption used in the conditional power calculation prior to start of trial, as assumptions can yield very different results. Additionally, trialists or data monitoring committees may wish to consider more than one conditional power value in which to base a decision. However, designs with formal decisions based on conditional power, such as a sample size re-estimation, may require one pre-specified assumption due to regulatory concerns [3].

When there is a wish to stop early for futility, the current trend assumption with an interim analysis after around 30% of participants have been recruited and follow-up up to the primary outcome, time point might be considered.

Optimistic confidence limits may provide a good compromise between the hypothesised and current trend current assumptions used in the conditional power equation, especially when there is uncertainty over what the trial may demonstrate. However, a later interim analysis timing of around 60–70% data availability should be considered using this assumption where logistically feasible.