Highly Efficient Clinical Trials Simulator (HECT): Software application for planning and simulating platform adaptive trials

Overview of inputs and functions

The HECT software application comes with a set of functions and outputs for trial simulation and conventional sample size calculations. The software also provides a brief user manual. Figure 1 shows the opening window of the HECT software. The trial simulator inputs and outputs is found in the first tab to the left, the conventional sample size calculator is available in the middle tab, and the user manual is found in the third tab to the right. The manual outlines each of the input options individually and provides summaries of available outputs. The manual appendix includes a detailed account of the statistical methods used at the back-end of the software. For all functions, the input bar is found to the left in the browser window, and the outputs are obtained and found in the right side of the browser window.

Figure 1. Displays the start up window of the HECT simulator software application

Clinical trial designs implemented

Conventional trial design. For comparative purposes, all simulations will automatically incorporate simulations of a conventional 1:1 randomized clinical trial. The conventional trial will accumulate the maximum allowed number of patients in the simulation, which can be informed by a conventional clinical trial sample size calculation (see Conventional sample size calculation under Statistical methods implementation). Figure 2a display an example of a standard multiple simulations output figure from the software where key properties such as type I error, power, and cost, are compared between a simulated HECT and a conventional trial.

Figure 2.

Displays example trial design performance outputs after 200 simulated trials: (a) comparing the type 1 error, power and overall cost between the highly efficient clinical trial (HECT) and the conventional clinical trial design; and (b) the distribution of sample size at trial termination with the highly efficient design (compared to 6,000 for all conventional designs).

Early stopping of trial for efficacy. The software includes an option to stop the trial early if a strong signal of superiority of one treatment over the other(s) is observed at an interim analysis. The signal is determined by a pre-set threshold for a Bayesian posterior probability of superiority. For example, if the threshold is set to 99%, a single simulated trial will be terminated at the first interim analysis where the probability of any treatment being superior to all other treatment exceeds 99%.

Early stopping or dropping of treatment arm for futility. The HECT software comes with two options for dropping a treatment arm for futility. For two-arm trials, early confirmation of futility may also lead to termination of the trial. First, a treatment arm may be dropped early for futility if the probability that it is superior to all other treatments fall below some pre-set threshold. For example, if the threshold is set to 1%, the treatment arm will be dropped at the first interim analysis where the probability of it being superior to all other treatments falls below 1%. Second, a treatment arm may be dropped with respect to some margin of observed treatment effect, typically an agreed upon minimally clinically important effect. For example, if we consider a relative risk reduction of 20% the minimally important effect for some clinical outcome, we can drop a treatment arm if we are highly certain the treatment effect (compared to control) does not exceed 20%. For example, investigators can insist on being 95% certain the effect is less than minimally important. In this case, the treatment arm will be dropped at the first interim analysis when there is a 95% probability or greater that the relative risk reduction is smaller than 20%.

Platform trial – adding new treatment arm. Similar to dropping a treatment arm, the HECT simulator also allows for the addition of a new treatment arm. This corresponds to using a platform design. The software allows the addition of a new arm to be triggered in two ways. First, if another treatment arm is dropped for futility (see above subsection) a new treatment arm will replace the dropped one. Second, if all arms are dropped except for one, a new arm can be added to compare to the winner of the previous stage. For the latter, this occurs if a trial is stopped for superiority of one treatment, except the platform trial design option allows for a perpetual continuation of the trial into a two-arm comparison.

Statistical methods implemented

Conventional sample size calculation. As a basis for comparison, the HECT simulator includes a sample size calculator for conventional randomized clinical trials (see appendix of User Manual). For multi-arm trials, the sample size is specified to detect the difference between the largest and second largest effect. The sample size calculator also allows for adjustment of multiplicity for multi-arm trials. The sample size calculator is found in the middle tab.

Adaptive design stopping rules. All stopping rules, whether for treatment arms or the whole trial, are based on the calculation of Bayesian posterior probabilities. Currently (Nov 2018), the software facilitates trial simulation for binary and continuous outcomes. As such, the Bayesian models comprise a conjugate Gaussian model for continuous outcomes and a beta-binomial model (i.e. a beta prior and a binomial likelihood) for binary outcomes. In both models a diffuse non-informative prior is specified.

Response adaptive allocation. Response adaptive adaptation is an optional feature under all trial designs implemented in the HECT simulator. With response adaptive allocation the allocation ratio between treatments is adapted based on which treatment appears most likely to be superior. In other words, on average more patients will be allocated to the treatment with the highest Bayesian probability of superiority¹². This method is particularly used when there is a strong incentive to reduce the number of patients exposed to an inferior treatment. While there are many methods of adjusting the allocation ratio, the HECT simulator uses the ratio between square roots of the posterior probabilities of being superior for each treatment¹³. When response adaptive allocation is selected, the allocation rate is set to equal for all arms (e.g., 1:1 for 2-arm trial) up till a pre-specified ‘burn-in’ sample size and is subsequently altered for every accumulated patient.

Simulation analysis functions implemented

The HECT trial simulator allows the user to estimate the trial design properties via simulation of large number of trials as well as simulating individual clinical trials to gain a better understanding of within trial variability.

Trial design properties (multiple trials simulation). To evaluate the overall trial design properties such as type I error, power, expected time to trial termination/average sample size, expected costs, etc. it is necessary to simulate multiple trials and average the performance metrics of these. The trial design properties function does just that. Figure 2 illustrates an example of the graphical outputs provided under the Trial Design Properties function. The user can decide how many trials to simulate or for how long the simulation can run (time maximum defined by number of seconds). The user can inspect the overall (or treatment vs control specific) power and type I error (reported numbers), the distribution of final sample sizes and cost across simulated trials (histogram display), as well as the comparison of power, type I error and expected costs between the HECT trial and a conventional design trial (bar plot display). Lastly, the user can save all or some of the recorded variables for the simulation that has just been run.

Single trial simulation. The single trial simulation function provides several graphical and numerical outputs to inspect within trial trends and variability. Figure 3 illustrates an example of some of graphical outputs provided for single trial simulation. First, the single trial simulation function produces a diagram for the trial flow over time (accumulation of patients) which allows visual representation of when any treatment was dropped or added. Second, a visual representation of where data points fall over time is available (extension of the trial flow diagram). Third, the single trial simulator provides visual inspection of the probabilities of superiority for each treatment over time, specifically at each planned interim look. This option is also available as graphical representation of posterior distributions for each treatment arm at selected interim looks. Finally, the final treatment estimates both for the primary outcome (which adaptations are based on) and optionally some secondary outcome (only monitored) are available to inspect.

Figure 3.

Displays example single trial outputs for: (a) actualized (platform) trial design scheme; (b) the probabilities of superiority for each treatment by interim look; and (c) the probability densities for each treatment by interim look.

User interface

The user interface comprises of three tabs: Trial Simulation, Sample Size Calculation, User Manual. The latter is a brief account of all input options and general outputs. In addition to the user manual, individual explanations are available for several functions when the user hovers the mouse cursor over the function. Both the Trial Simulation and Sample Size Calculation tabs have their input field to the left and the output to the right. At start up, default values are set to avoid empty values. Quality checks are automatically run, and error messages with accompanying instructions will be provided if the user attempts a computation based on input values that are either beyond the allowed range or of the wrong format.

Where radio buttons are used to define the input format (binary or continuous outcomes) for trial design type (compare all arms or compare vs control), the input field will automatically change to match the selected.

The right side panels are for the outputs. For the Sample Size Calculations tab, the ‘Calculate’ button will initiate the conventional sample size calculation and if all inputs are of correct format, the resulting sample size will appear. For the Trial Simulation tab, the right side is split in two panels. The lower panel is for running single trial simulations (one at a time) and inspecting the within trial behaviour. The upper panel is for summarizing the performance properties across multiple trial simulations. In both, the user is required to press the ‘Run’ button for a simulation (single or multiple) to run. A single trial simulation typically only requires a few seconds, whereas multiple trial simulations may require a few minutes. A progress bar will appear in the lower right corner so the user can follow the back-end computations progress. Whether single or multiple simulations are run, the available output graphs will update once the computations are complete.

Underlying data

All data underlying the results are available as part of the article and no additional source data are required.