The ORVAC trial: a phase IV, double-blind, randomised, placebo-controlled clinical trial of a third scheduled dose of Rotarix rotavirus vaccine in Australian Indigenous infants to improve protection against gastroenteritis: a statistical analysis plan

Despite the introduction of rotavirus vaccine into the childhood vaccination schedule in 2006, Northern Territory Indigenous children remain more than 20 times more likely to be hospitalised with rotavirus gastroenteritis than non-Indigenous children in other Australian states and territories [1]. Data from remote communities in the Northern Territory suggests that 77% of children have at least one documented episode of clinic attendance for diarrhoea before their first birthday, with a median of three (IQR 1–5) clinical presentations for diarrhoea per child in the first year of life [2].

The current vaccination schedule with Rotarix is at 2 and then 4 months of age. We hypothesise that the routine addition of a third scheduled dose of Rotarix for NT Indigenous infants, administered between 6 months and less than 12 months old, will improve protection against clinically important rotavirus gastroenteritis.

This statistical analysis plan (SAP) provides a priori specification of the decision-making rules and the statistical methods to be used. It is intended to disseminate practical knowledge on adaptive trials to trialists that are new to these designs. The SAP was prepared after data collection had commenced, but prior to observing any of the data. The coordinating principal investigator (TLS) was responsible for approving and signing off the SAP, and the document has also been reviewed and approved by an independent data monitoring safety board (DMSB). The SAP is consistent with the CONSORT 2010 Statement [3] and further guidelines [4‐6] and supports transparent and reproducible research.

As at August 2020, four interim analyses have been run, but no stopping rules have been triggered.

While fixed design clinical trials are conceptually straightforward, they suffer from rigidity and frequently end with inconclusive results [7, 8]. Contrastingly, adaptive designs (ADs) allow for pre-specified adaptations that modify the design as the trial progresses and as data is accumulated [9].

Typical adaptations include dynamic changes to sample size, dropping treatment arms for futility, response adaptive randomisation, seamless phase II/III trial transitions, study population enrichment and early stopping rules [10] that are triggered under predefined conditions [11]. Consequently, ADs may be completed sooner, cost less to run, reduce the number of patients exposed to inferior treatments and provide more clinically relevant data than fixed designs [12]. However, ADs are not without their own limitations. The processes associated with obtaining funding, planning and designing adaptive trials are complex and can require more effort and time than traditional trials [8]. Furthermore, developing simulations [13] to explore the frequentist operating characteristics (e.g. type I error and power) requires specialist staff and/or consulting services, custom software infrastructure and high-performance computing facilities [7].

Historically, ADs and platform trials (a variant of an adaptive trial) have generally been implemented in phase I and II settings in the USA, with many originating from the University of Texas MD Anderson Cancer Center [11, 14, 15]. However, interest in ADs is growing and they are now being deployed and evaluated in phase III (e.g. [16‐18]) and IV settings [9, 19]. In early 2016, the REMAP-CAP (Randomised, Embedded, Multifactorial Adaptive Platform trial for Community-Acquired Pneumonia) trial, an embedded platform trial run jointly in Europe, Australia and New Zealand, commenced. REMAP-CAP has served as a catalyst, motivating Australian trialists to raise funds, develop capacity and become involved in adaptive trials to answer their research questions. Examples include BEAT CF [20] and GBM Agile [21]. The ORVAC trial is one of several Bayesian ADs that are in development at the Telethon Kids Institute in collaboration with Berry Consultants.

ORVAC is a pragmatic, investigator-led, double-blind, randomised, placebo-controlled Bayesian adaptive clinical trial testing a third scheduled dose of Rotarix rotavirus vaccine (versus usual care) in Australian Indigenous infants to improve protection against clinically important gastroenteritis. It has the following key features:

ORVAC has two population subgroups: (1) children who normally reside in a major city, an inner or outer regional area and (2) children residing in remote or very remote areas according to the Australian Government Department of Health Australian Standard Geographical Classification-Remoteness Area (ASGC-RA) system. While baseline characteristics for immunological status and median time to medical attendance/hospitalisation are not well understood for this population, comparable settings have an RV vaccine efficacy of around 50%. However, in the NT Indigenous population, protection has been noted to wane after the first year of life, estimated to fall to a rate of only 10% [22]. Estimates for the rate of hospitalisations due to acute gastroenteritis are also highly variable with median times to hospitalisations reported to be between 15 and 40 months from birth [1].

The clinical data from this trial will be analysed and reported on an intention-to-treat (ITT) basis with all randomised participants contributing to the analysis of the co-primary endpoints. Specifically, for the ITT analysis:

We will also produce companion analyses on a per-protocol (PP) basis using the subset of participants that completed without protocol violation.

We will prepare an expanded CONSORT diagram suitable for a parallel two-armed adaptive trial. We will record the start and end dates of accrual, the flow of participants through the study including the completeness of follow-up as per the CONSORT statement [23]. We will note participants enrolled and randomised and eligibility for analyses.

Stratified (regional vs remote), random allocation of two treatment arms to contiguous randomisation numbers (1 to 1000) was provided by JM. The allocation was computer-generated using random permuted block sizes between 6 and 20. The allocation ratio within these strata is 1:1, and MAJ maintains the password-protected file that contains the allocation sequences.

The ORVAC trial is double-blind with neither the participants nor the research staff having knowledge to patient treatment status. MAJ is solely responsible for interim analyses and reporting to the Data Safety Monitoring Board (DSMB). JM is responsible for quality control review on the interim analysis reports, with both MAJ and JM being unblinded. JT is responsible for code review, but remains blinded to the randomisation and results. All other staff and investigators are blinded. Interim analyses are discussed in closed session between MAJ and the DSMB with no investigators present. After each interim analysis is completed and approved by the DSMB, a recommendation for continuing the study (with no reference to the results) is reported to TS by the DSMB. All randomisation lists and analyses are kept on a secure server for which only MAJ has access.

The purpose of this study is to determine if Indigenous children who receive an additional dose of Rotarix between the ages of 6 and 12 months will have an increased anti-rotavirus serum IgA seroconversion and an increase in the time to medical attendance due to gastroenteritis in the first 3 years of life, compared to those who receive placebo.

ORVAC is a superiority trial that uses Bayesian methods for inference and decision-making. Unless otherwise noted, all parameter estimates will be reported as means or medians with 95% credible intervals.

The processes around trial monitoring, including data monitoring, are defined in the quality assurance procedures of the ORVAC protocol document.

In brief, data will be sourced from a specifically designed clinical record form (CRF) comprising consent forms, eligibility assessment, visit record, adverse event details and protocol deviations. Data is entered from these sources into a trial database by study personnel. Data queries are raised by the data manager and cleaned by both the data manager and trial statistician. All planned final analyses identified herein are to be performed after the study is completed and the database has been cleaned and locked. However, the interim analyses will be performed on incomplete data in that amendments may occur to the data that was used in an interim at some time after the interim analysis is completed. Our assumption here is that less than 5% of the sample size at any given interim analysis will be subsequently amended.

Within Bayesian analyses, missingness is either ignorable or non-ignorable. In the former case, the parameters relating to the measurement are distinct from those that relate to missingness and the mechanisms are termed either Missing Completely at Random (MCAR) or Missing at Random (MAR). For MCAR, a complete-case analysis will result in reduced efficiency but is unbiased. However, for MAR, a complete-case analysis will be both inefficient and biased. Non-ignorable missingness relates to those data that are Missing Not At Random (MNAR). Under MNAR, an extra model is required to predict the missingness.

One of the advantages of using Bayesian methods is that they offer a natural way to simultaneously impute missing values and fit models on the observed and imputed data [44]. After exploring the amount and patterns of missing data and the association with other variables, we will impute as necessary using a fully Bayesian approach following methods as per [44].

Except for the simplest of cases, the operating characteristics for a Bayesian adaptive trial are analytically intractable. Therefore, these are usually estimated by Monte Carlo methods, which are a general purpose tool for optimisation and integration problems [45]. In the context of exploring the operating characteristics of a given trial, the idea is to formulate a data generating process (DGP) that represents expected and plausibly extreme outcomes for the actual trial. However, as is the case here, it may be necessary to simplify the data generating process and/or analysis approach in order to be able to run the simulations within a workable timeframe. Using the assumed DGP, trial data can be simulated many times and the resulting ‘virtual trials’ are analysed using methods described shortly. Various characteristics of the DGP can then be derived. For example, the expected type I error (false positive rate) is derived from data that are generated under a null effect configuration and estimated from the proportion of times that we falsely detect a difference between the treatment arms in the final analysis.

We have examined the operating characteristics of the ORVAC trial in more than 300 scenarios using Monte Carlo simulation of the parameter space associated with a representative data generating, sampling and modelling process. For the immunological endpoint, seroconversion was modelled via a series of independent Bernoulli trials with control arm probabilities of seroconversion (the baseline seroconversion rate) between 0.1 and 0.7 and a change of probability of seroconversion in the treatment arm between 0 (the null case) and 0.15 above the baseline rate. Additionally, we modelled information delays associated with processing the immunological endpoint of 0.5 and 0.7 months. For the clinical endpoint, we modelled time to event as an exponentially distributed random variable. The control arm median time to event was set at 20–50 months and a change in the median time to event varied between 0 (the null case) and 15 months. Event times were censored at 36 months of age. We modelled accrual using a Poisson process generating approximately 30 and 50 participants per quarter. Age at vaccination was modelled using a uniform distribution with lower and upper bounds of 6 and 12 months respectively. The co-primary endpoints were modelled as independent random variables.

For the analyses, we used conjugate prior models because Markov Chain Monte Carlo estimates of the posterior and posterior predictive distributions were prohibitively costly in terms of available CPU resources. Specifically, we used the beta conjugate prior to the binomial likelihood for the immunological endpoint, and the gamma conjugate prior to the exponential likelihood for the clinical endpoint with both sets of priors configured to be weakly informative. We note that while the simulation methods are representative of the analyses we propose for the trial, they are not identical to them.

For each scenario/subset of the parameter space, we simulated 10,000 trials for the null cases and 1000 trials for the trials where each configuration was for a non-zero difference between the treatment arms. We summarised and reviewed the results with other statisticians and study CIs, explored the trial decision probability thresholds and then re-ran the scenarios. The whole process was repeated until acceptable false positive and other trial characteristics were obtained.

For each scenario, all posterior sampling was based on 2000 draws from the relevant distribution. Posterior predictive assessments were based on a further 1000 posterior predictive draws at each interim for both endpoints.

The simulations were written in R and C++, reviewed by JT and stored under version control. The simulations were coded and run by MAJ on Linux-based multi-core servers.

Table 2, presented earlier, details the thresholds that were identified through simulation to be used for the interim and final analysis.

Table 3 provides examples of the type I error rates obtained over a range of plausible scenarios. In all cases, the type I error/false positive rate is controlled at the α<0.05 level for both co-primary endpoints. Adjustment for multiple comparisons is addressed via the probability thresholds that are selected to be used in the analyses.

Figures 2 and 3 show the power curves for the clinical outcome and expected total sample size respectively. Other aspects of the operating characteristics including probability of expected success, futility and stopping venous sampling and expected sample size of the immunological endpoint over the parameter space are provided as supplemental material Figs. 4, 5, 6, 7, and 8.

The clinical and immunological outcome are analysed independently. For the clinical outcome, power increases monotonically as a function of increasing effect size and lower baseline median time-to-event scenarios. For a baseline median time to event of 35 months, power climbs from 50 to 75% as the difference between the median time to event in the two arms increases from 10 to 15 months. At a baseline time to event of 20 months, the power to detect a 10-month difference is over 70%. Power was relatively insensitive to the accrual rates and information delays that we examined.

For the immunological outcome, power is largely independent of the clinical outcome configuration. For a baseline probability of seroconversion of 0.4, while the power to detect a 0.1 difference in probability of seroconversion is only around 40%, this increases to around 70% to detect a difference of 0.15. However, when the baseline probability of seroconversion is 0.1, the power to detect a 0.1 difference is in excess of 60% and to detect a 0.15 difference is well in excess of 80%.

The probability of expected success increases as a function of the difference between the median time to event in the treatment versus control arm. When the difference between the probability of seroconversion between the treatment and placebo arms is 0.05, the probability of expected success is only around 0.2. However, as the difference in the seroconversion probabilities increases, the probability of expected success becomes increasingly likely.

The probability of stopping for futility is dependent on both the clinical and immunological outcomes. For a difference between the median time to event equal to 5 months, the probability of futility is up to 0.7. However, as the difference between the median time to event and probability increases, the probability of futility drops to less than 0.1.

The expected sample size reduces as a function of clinical endpoint effect size but increases as a function of the immunological endpoint effect size. This is because the probability of ceasing the trial for futility remains high (in excess of 50%) when the immunological endpoint effect size is low. However, as the immunological endpoint effect size increases, so does the probability that the trial will continue to accumulate enough data to detect a clinical treatment effect. Based on the scenarios we simulated, we anticipate a minimum expected sample size of around 300 participants and up to a maximum sample size of 700. We anticipate that the expected number of blood samples will range from 150 to 200.

We used R version 3.5.3, RStudio version 1.1.423 and C++ for the trial simulations and will use these or later versions in the trial analyses.