Bayesian methods: a potential path forward for sepsis trials

A Bayesian analysis can summarize historical evidence on the size of a treatment effect through what is called the ‘prior distribution’. Although many previous studies provide support for the notion that PMX can reduce mortality [20‐23], there are numerous differences between the patient groups in these other studies and the proposed Tigris study, most notably the absence of the EAA biomarker to identify a group most likely to benefit from PMX treatment. The treatment effects from these other studies are not a summary of the evidence for a benefit of PMX in this subgroup, so they cannot be used directly to construct a prior for the treatment effect in Tigris. Furthermore, there are many other differences (e.g., study protocol, patient inclusion criteria, study location, timing of outcomes) that also introduce uncertainty about the applicability of those earlier findings to a new trial. By contrast, when we consider Pocock’s criteria [24] for inclusion of data from historical patients in analysis of a new trial, we find that the treatable cohort [15] from the EUPHRATES study is an ideal source of prior information for the treatment effect in Tigris; standard of care, treatment, patient eligibility, evaluation of outcomes, investigators and ICU locations are largely the same in the two studies. We will demonstrate a range of uses of the EUPHRATES treatable cohort: (a) viewing Tigris as a simple continuation of that cohort, (b) down-weighting the prior evidence it provides or (c) ignoring the results entirely.

The extent to which Tigris can be seen as a continuation of enrollment into the treatable cohort from EUPHRATES determines how we use those historical data to create a prior for Tigris. To illustrate this idea, Fig. 1 shows a range of prior distributions formed from the historical APACHE-adjusted odds ratio (aOR) from the treatable cohort. Figure 1a treats Tigris as a straight continuation, and simply takes the posterior distribution of the aOR from the treatable cohort as the prior for Tigris. The priors in Figs. 1b and 1c acknowledge that the previous results may not be entirely transportable to this new trial and are down-weighted to be equivalent to data with the same observed aOR, but in a sample only 75% or 50% of the actual size; these down-weighted priors express more uncertainty than the prior in 1a about the potential values of the aOR. Figure 1d takes an extreme view—the one taken by a classical analysis that uses no prior- ignoring the results in the previous study entirely and allowing a priori that all values of the OR are equally likely, no matter how biologically implausible. The prior appears almost as a horizontal line. Each figure also shows the prior 95% credible interval (CrI), prior probability that the OR for treatment is less than 1 and, because small differences in tail probabilities appear to understate the different levels of certainty in these priors, the corresponding odds that the OR is less than 1.

There are two broad approaches to specifying how much weight should be placed on the historical evidence. One approach uses clinical judgment to fix the weights as shown in Fig. 1 [25] and investigates the results of analyzing new data for each of a small set of fixed weights (e.g., 75%, 100%). The other approach is statistical and uses the similarity between the new data in the trial and the historical data to infer the weight that should be given to the historical evidence; the more similar the new data and the historical data, the higher the weight, and vice versa. We assessed two statistical approaches. The first uses a normalized power prior [26] and the second uses what is called a commensurate prior [27]. Notably, even when the new data are in perfect agreement with the historical data, each of these statistical approaches still places less than full weight on the historical evidence.

A brief summary of the simulations and statistical analyses of the simulations are provided here; full details can be found in the in the supplement. For each of 25 combinations of control group risk (40% to 60% by 5%) and absolute risk reductions (ARR) (0% to 20% by 5%), 2000 trials of 150 patients (100 PMX and 50 control) were simulated. Each trial was analyzed with 5 different uses of the historical data (100% weighting, 75% weighting, weighting through commensurate and normalized power priors, and 0% weighting) with and without adjustment for baseline APACHE score, for a total of 10 analyses per trial. These Bayesian analyses estimated the odds ratio (OR) for mortality comparing the PMX and control groups and in each simulated trial, checked whether the trial demonstrated benefit for PMX, defined in each trial as a posterior probability greater than 95% that the OR was less than 1. The percentage of the 2000 trials demonstrating benefit was used to estimate the power (or type I error when ARR = 0) of the corresponding analytic method for that control group risk and ARR. As a sensitivity analysis, benefit was defined as a posterior probability greater than 97.5% that the OR was less than 1.

When TIGRIS is complete, the trial report will present a plot of the odds ratio and its 95% credible interval against weights ranging from 0 to 100% to allow the reader to assess the dependence of the results on the amount of historical information that is borrowed. We will also present the posterior probability of benefit as a function of these prior weights and will not dichotomize this probability at a sharp threshold of 95%, for example, as being “significant” or not. However, for the purposes of trial planning and investigating the role of the prior, we use these thresholds, an approach that is in keeping with previous literature [28].

Each plot in Fig. 2 shows power (the probability that we will conclude that PMX is superior to control at the 95% probability threshold) versus the marginal ARR for treatment. Each plot is for a scenario with the baseline risk shown in the row heading analyzed with either an APACHE-adjusted model (left column) or unadjusted model (right column). Each plot shows power curves versus for fixed 75% and 100% prior weighting, for prior weighting based on similarity of new and historical data using the commensurate prior and for a Bayesian analysis including minimal prior information (uninformative prior). The results shown in Fig. 2 help us make some decisions about the choice of a prior and the analysis, no matter what the true treatment effect and prevalence. There are a few clear patterns. First, an analysis that adjusts for the baseline APACHE II score (left column) is always more powerful than the analysis that does not (right column). Secondly, use of a prior putting 75% weight on the results from the treatable cohort in EUPHRATES (gold lines in each panel in Fig. 2) leads to a greater chance of detecting a true benefit for PMX than a prior that bases the weighting on the similarity of new and historical data (red lines) or an analysis that disregards the historical data (black lines). Thirdly, the increase in power with use of historical data comes with this cost: if there is no true benefit of PMX in Tigris (ARR = 0, far left side of each plot), any analysis that combines a positive signal (from the treatable cohort) with what will be on average a null signal (from Tigris), is more likely to produce a more favorable result than an analysis of Tigris alone [29]. The prior that adjusts the weighting based on the similarity of new and historical data is less likely to produce a favorable result in this null scenario than the fixed 75% weight prior. As the normalized power prior and commensurate prior approaches produced results that were practically identical, only results for the commensurate prior are shown. Additional file 1: Figure S1 presents power when benefit is defined as a posterior probability greater than 97.5% that the OR was less than 1.

The analysis using only Tigris data, as expected, has a 5% chance of meeting the 95% probability threshold for benefit (the black lines in Fig. 2 have “power” of 5% when ARR = 0). When the true ARR in Tigris is 0, some Tigris trial results will vary randomly below an ARR of 0 and, when combined with any use of the historical data, may meet the 95% probability threshold for benefit. Our planned analyses therefore have a small chance of determining that PMX is effective when it is not effective in Tigris. Conversely when the ARR in Tigris is in the neighborhood of 15%, there is still a small chance that we will conclude that PMX is ineffective even though it is highly effective. Still, an analysis combining the historical data with data from the new trial will provide a better representation of the true effect than either the historical data or the new trial data taken separately.

Figure 3 provides distributions of observed APACHE-adjusted ORs from the 2000 simulated trials with a baseline risk of 50%, coded according to whether they meet the 95% probability threshold for benefit; a similar plot showing unadjusted ARRs can be found in Additional file 1: Figure S2 and a plot with Bayesian posterior estimated of unadjusted ORs can be found in Additional file 1: Figure S3.

Figures 2 and 3 summarize results over thousands of simulated trials. Table 1 provides a more concrete demonstration of how the different analytic approaches may lead to different conclusions in a single trial. This table shows results of analyses of eight potential trial results, all having an observed control group mortality of 44% (22/50), but with observed mortality in the 100 patients treated with PMX ranging from 24% up to 44%. For observed absolute risk reductions (ARR) from 20% (the first block of rows) and 15% (the second block of rows), Tigris alone would satisfy the criterion of > 95% probability of benefit in both adjusted and unadjusted analyses (cells a and b). The commensurate prior and 75% weighted prior produce still higher probabilities of benefit. In the case of an 11% ARR (33% vs 44%), the adjusted analysis for Tigris alone (cell d in the table), produces only a 92.3% probability of the OR being less than one, a value that rises to 96.1% with the commensurate prior and 98.4% with 75% weight on the prior (cell c). Here, even though the ARR of 11% is almost identical to the 10.5% absolute risk reduction in the treatable cohort, the commensurate prior gives as much credence to the historical data as a 50% weighted prior (result not shown) and returns a probability of benefit of 96.1%, i.e., less than the 97% probability found in the prior. Using the planned 75% weight on the prior, an observed ARR of 7% (37% vs 44%) is approximately the boundary in these eight datasets for reaching the 95% probability threshold for declaring PMX effective. This result gives only a 79.9% probability of benefit in Tigris alone in the adjusted analysis (cell f), but a 95.7% probability when combined with the prior (cell e). In the four potential Tigris trial results that are less favorable (observed ARRs of 5% or less), the posterior probability of benefit in both adjusted and unadjusted analyses fall below the 95% threshold when the prior data are down-weighted to 75%, included through the commensurate prior, or ignored. It may be surprising that a finding of a 1% ARR or even no effect in Tigris (cells g and h) can translate into a posterior probability of benefit of 87%-89% (odds of ~ 7:1 to 8:1) but note that this is lower than the 97% prior probability of benefit (odds 32:1) that we began with; a negative result in Tigris will reduce our belief that PMX is effective. Notably, as an absolute 0% ARR is quite dissimilar to the result in the treatable cohort (10.5% ARR), the commensurate prior gives less credence to the prior data and returns a posterior probability of benefit of 68.3% (cell i).