Adaptive designs in critical care trials: a simulation study

Randomised controlled trials (RCTs) play a pivotal role in identifying effective and harmful treatments. In critical care trials, mortality is frequently the event of primary interest and, as a result, trials require a very large number of participants to detect relatively small between-arm treatment effects [1, 2]. Recent work by Granholm et. al provided an overview of past critical care RCTs and discussed current major challenges and recent improvements associated with those trials [3]. A traditional parallel group RCT is designed to recruit a fixed number of participants and, apart from interim analyses focused mainly on safety, will continue until the predefined number of participants have been recruited with the sample size unaffected by evidence that accumulates during the trial. While this design minimises the likelihood of stopping early due to an imbalance in outcomes arising from the play of chance, it is inefficient given late-phase critical care RCTs often taking many years to compete recruitment.

Unlike traditional RCTs that follow a design-conduct-analysis sequence with limited interim analysis, adaptive trials perform pre-planned interim reviews of the data during the conduct of the trial and may adapt the trial design based on those accumulated data [4]. An adaptive design is defined as ‘a clinical trial design which allows for prospectively planned modifications based on accumulating data through the trial’ [5]. Many types of modifications are possible and an ongoing trial can be adapted by recalculating the total sample size, adding or dropping treatments or doses, stopping early for evidence of efficacy or lack of efficacy, altering the treatment allocation ratio, and recruiting only subtypes of patients who appear most likely to benefit from the trial intervention [6].

The ability to be highly flexible while preserving validity and integrity is one of the main advantages of adaptive designs [7]. This flexibility can be attractive to stakeholders’ such as funding agencies and trial sponsors [5]. Furthermore, adaptive designs can have greater statistical efficiency and may estimate the treatment effect more precisely than traditional fixed designs [7]. Importantly, adaptive trials can also be stopped earlier based on interim analyses and may thus be more efficient and ethical as further participants are not recruited to a trial that has already generated sufficient evidence to answer the question posed [6].

Adaptive designs are increasingly popular and a recent review showed that their widespread application to critical care trials has the potential to improve the cost–benefit ratio of clinical trials among critically ill patients, with some increased in complexity in statistical preparation [1, 5]. While a number of previous critical care trials including Centocor: HA-1A Efficacy in Septic Shock (CHESS) [8], Adjunctive Corticosteriod Treatment in Critically Ill Patients with Septic Shock (ADRENAL) [9] and Normoglyceamia in Intensive Care Evaluation-Survival Using Glucose Algorithm Regulation (NICE-SUGAR) [10] have used group sequential designs, they included a limited number of interim analyses with conservative efficacy boundaries and no futility monitoring thus limiting the room for ‘true adaptation’. Other adaptive approaches including unblinded sample size re-estimation [11] and a seamless phase II/III design [12] have also been used. More recently, a number of trials including Randomized, Embedded, Multifactorial Adaptive Platform Trial for Community-Acquired Pneumonia (REMAP-CAP), AGILE and Australasian Covid-19 Trial (ASCOT) have used Bayesian adaptive group-sequential designs motivated by the need to identify COVID-19 treatments as rapidly as possible [13‐15].

Despite these recent examples and the increasing use of more advanced adaptive designs, the relative performance of common statistical methods and the effect of the number of interims remains unclear to many triallists. We therefore aimed to examine the impact of various adaptive strategies via a simulation study based on data from two large critical care trials.

Our analysis shows that applying adaptive methods to critical care trials with and without futility monitoring helps reach trial conclusions with a reduced sample size. The re-analysis of the ADRENAL trial shows that the inclusion of futility boundaries would have led to early stopping for futility with four or nine interim looks using either the O’Brien-Fleming or Hwang-Shih-DeCani design after recruiting 2926 (out of 3658) participants. Thus reaching the same conclusion as the original ADRENAL trial findings with a smaller sample size. The NICE-SUGAR trial might have been stopped for futility with four interim analyses using the O’Brien-Fleming design after recruiting 4817 (out of 6022) participants while early stopping for futility might have occurred with nine interim analyses using the same design after 4215 participants have been recruited. While this represents a substantial saving in sample size, it is important to note that these findings are not consistent with the final conclusion of the original NICE-SUGAR analysis which reported significant harm associated with intensive glucose control. With futility monitoring, the NICE-SUGAR trial would have been stopped early for futility on the basis that intensive glucose control would not be shown to be superior to conventional control. While this reduction in sample size looks attractive, it may have had a negative clinical impact as the absence of definitive evidence of harm would have resulted in continued use of intensive glucose control and in a substantial number of preventable deaths assuming that the originally-reported NICE-SUGAR findings represent the true effect. We used binding futility boundaries in our simulation study; however, it is also possible to use non-binding ones which will provide flexibility to continue the trial even if the boundary is crossed despite a slight increase in sample size.

This re-analysis was followed by an extensive simulation study comparing five interim monitoring approaches under null and alternative scenarios. As previously reported, the results showed that as more interim analyses were conducted, the chance of reaching the correct conclusion before the final analysis stage became higher and the sample size was greatly reduced under both trial settings. This was true under both the null (no treatment effect) and alternative (true treatment effect) scenarios.

Under both scenarios, there was a sharp decrease in the expected sample size as the number of interims increased from one to two and a more gradual and steady decrease as more interim analyses were added. The decrease in the expected sample size was predominantly caused by an increased probability of making an early correct decision. While the impact of the number of interim analysis on the expected sample size is well-known, these findings are of particular relevance to late-phase critical care trials given their large sample size and relatively short follow-up times.

Our simulations confirmed that group-sequential designs provide good control of the type I error rate regardless of the number of interim analyses; however, the two Bayesian approaches failed to maintain the type I error rate when increasing the number of interim analyses. Shi and Yin have proposed novel methods to effectively maintain the overall type I error rate for Bayesian sequential designs [32]. Despite using the same sample sizes as the ones used in the original studies, statistical power was generally well maintained at 90% across all approaches. Notably, the HP design always achieves the highest power regardless of the number of interim analyses while maintaining a fixed sample size due to the conservative stopping boundaries. The two Bayesian approaches were also generally well-powered; however, this was accompanied by an inflation the type I error rate.

Under both scenarios, the HP design always resulted in the largest sample size due to a very low chance of making an early decision, whether correct or not. The OBF design was generally more likely to make an early correct decision and hence more likely to reduce the expected sample size compared with the HSD design under the null scenario, however, it became less advantageous under the alternative scenario. The PostP design showed the smallest sample size at most numbers of interims, but compared with the PredP design, it poorly balanced the chance of making an early correct decision and the type I error rate with more than one interim analyses under the null scenario, and it also poorly balanced the chance of making an early correct decision and the type II error rate for the NICE-SUGAR trial under the alternative scenario.

Under the alternative scenario, the estimated treatment effect was found to be increasingly negatively biased from the true effect (i.e. further from an OR of 1 suggesting a stronger treatment effect) as the number of interims increases. This issue with overestimation of treatment effect in adaptive designs is important but less well-studied compared to type I error rate inflation, according to the recent FDA guidance on adaptive designs [33] The bias in estimation of treatment effect can be corrected using various techniques including the median unbiased estimator [34], shrinkage approach [35] and bootstrapping [36]. More recently, Robertson et al. have provided a comprehensive overview of currently available unbiased and bias-reduced estimators of treatment effect for various types of adaptive designs as well as guidance on how to implement and report them in practice [37].

A strength of our study was that it was motivated by two real large-scale high-quality trials representative of late-phase critical care trials. The ADRENAL trial and the NICE-SUGAR trial were used to compare the performance of common group-sequential methods as well as Bayesian approaches used for interim monitoring. Therefore, our findings reflect potential real-life trial application that can also be applied to the design of two-arm trials with other types of outcomes including time-to-event or continuous variables. They are also generalisable to settings outside of critical care.

However, our study has some limitations. First, our simulation study assumed a fixed sample size based on the original ADRENAL and NICE-SUGAR designs. In practice, one would typically increase the target sample size at the design stage to account for the number of interim looks to maintain the desired power. Second, we used a limited number of adaptive methods in our study, focusing on five commonly used frequentist and Bayesian approaches. In addition, approaches such as the HSD design and the Bayesian approaches can be further calibrated by changing parameters or thresholds for stoppings. In particular, the two Bayesian approaches require pre-specification of the thresholds for early stopping. We chose thresholds that have been used in previous studies; however, we found that, as we increased the number of interim analyses the type I and type II error rates were not always preserved.. Third, all interims were assumed to be equally spaced; however, this rarely applies in practice due to non-uniform recruitment rates or logistic considerations. Further work is required to assess whether unequal spacing further decreases the expected sample size. Fourth, our simulations assumed that recruitment was paused at the time of each interim analysis. In practice, most critical care trials would continue recruitment while an interim analysis is performed thus attenuating potential reductions in sample size. Others have discussed this aspect and demonstrated the additional efficiency gains when pausing recruitment at the time of an interim analysis [38, 39]. Wason et al. have also highlighted the logistical complexity of adaptive designs and suggested a careful consideration of efficiency benefits versus potential drawbacks [40]. Fourth, we considered mortality as the sole outcome as is standard for late-phase critical care trials; however, one might also wish to consider the overall cost-effectiveness of adaptive approaches [41]. Flight et al. have recently conducted a simulation study to assess the cost-effectiveness of group-sequential designs at different numbers of interim analyses by constructing an adjusted health economic model [42]. Fifith, important subgroup differences might have been missed due to a loss of statistical power following an early termination of a critical care RCT.

Granholm et. al have recently discussed potential practical drawbacks of critical care trials including oversimplifications of the trials and a lack of harmonisation of trial protocol and methodological drawbacks including overly optimistic effect sizes, limited attention on patient health state outcomes and a lack of flexibility and adaptability. An additional concern, discrepancies between statistical significance and clinical importance caused by dichotomising results using frequentist group-sequential methods has been widely discussed in their research. They also argued that using Bayesian statistical methods result in more nuanced interpretations which avoid dichotomisation, with the use of different priors which enables the formation of context-dependent conclusions as well as different appropriate evidence thresholds depending on the intervention [3].

Ryan et. al recommended a wider application of Bayesian adaptive approaches to phase III critical care trials, based on their case study on the High Frequency OSCillation in Acute Respiratory distress syndrome (OSCAR) trial via simulation [27]. They concluded that Bayesian adaptive designs led to an earlier termination of the trial and a reduced number of patients recruited with over 15% fewer deaths than the original trial design which used two interim analyses with OBF sequential boundaries for early efficacy stopping; the Bayesian adaptive design yielded a similar power and trial conclusions to the original design. Ryan et. al also discussed the impact of the number of interim analyses on the type I error rate inflation using Bayesian adaptive designs without adjustments for multiplicities. They recommended stopping boundary adjustments when using Bayesian adaptive designs that allow for early stopping for efficacy as the number of interims increases in order to control the type I error rate, while no demonstration of the control may be required when using a strict Bayesian approach [43]. Furthermore, Saville et. al demonstrated the advantages of using predictive probabilities compared to group sequential approaches and posterior probability approaches in clinical decision-making process since it appropriately accounts for auxiliary variables and lagged outcomes despite potential computational burdens [26].

Depending on the clinical context and the acceptable risk–benefit balance, it is unlikely that a single approach will consistently outperform all others. In many cases, a group-sequential design with spending functions and an appropriate number of interim analysis, should provide the right balance between correct early stopping and type I or type II error rates. For situations requiring additional flexibility, Bayesian approaches such as PredP and PostP might be more suitable; however, they require more extensive simulations to ensure appropriate calibration of the error rates. Further sensitivity analyses are required for examining the impact of prior distributions including uninformative or informative, positive, neutral or negative, evidence-based or sceptic priors [44].

In conclusion, this study shows that increasing the number of interim analyses with appropriate efficacy and futility boundaries increases the chance of finding the correct answer at an earlier stage thus decreasing the trial sample size and conserving resources. This is highly applicable to late-phase critical care trials which tend to have large sample sizes and outcomes observed over a relatively short follow-up time.

Systematically exploring adaptive methods when designing critical care trials will aid the choice of the most appropriate method. This should be done in consultation with clinicians in order to identify the design with the most desirable properties and while balancing the benefits due to sample size reductions against potential operational and computational burdens.