Heterogeneity of treatment effect by baseline risk of mortality in critically ill patients: re-analysis of three recent sepsis and ARDS randomised controlled trials

We obtained ethics approval for this study (18/LO/1079). VANISH was a 2 × 2 factorial, double-blind, RCT in adult patients with sepsis who required vasopressors, in 18 general adult intensive care units (ICUs) in the United Kingdom (UK) [13]. In the VANISH trial [13], patients were randomly allocated to vasopressin and hydrocortisone (n = 101), vasopressin and placebo (n = 104), norepinephrine and hydrocortisone (n = 101) or norepinephrine and placebo (n = 103). Patients only received the second study drug (hydrocortisone/placebo) if the maximum infusion of the first study drug (vasopressin/norepinephrine) had been reached. Therefore, in the hydrocortisone analysis, only participants who received the study drug were included (hydrocortisone n = 148, placebo n = 148); all remaining analyses are intention-to-treat. The 28-day mortality was 63/204 (30.9%) of patients in the vasopressin group and 56/204 (27.5%) patients in the norepinephrine group (difference = 3.4% [95% CI, − 5.4–12.3%]) [13]. LeoPARDS was a two-arm parallel group, double-blind, placebo-controlled RCT in adult patients with sepsis who required vasopressors, in 34 ICUs in the UK [13]. In LeoPARDS trial [14], patients were randomised to receive either levosimendan (n = 258) or placebo (n = 257) over 24 h in addition to standard care. The 28-day mortality was 89/258 (34.5%) in the levosimendan group and 79/256 (30.9%) in the placebo group (difference = 3.6% [95% CI, − 4.5–11.7%]) [14]. HARP-2 was a two-arm parallel group, double-blind, placebo-controlled RCT in adult patients within 48 h after the onset of ARDS in 40 ICUs in the UK and Ireland [15]. In the HARP-2 trial [15], patients were randomised to receive either once-daily simvastatin or identical placebo tablets enterally for up to 28 days. The 28-day mortality was 57/259 (22.0%) in the simvastatin group and 75/280 (26.8%) in the placebo group (risk ratio = 0.8 [95% CI, 0.6 to 1.1]) [15].]

Four sensitivity analyses for the main baseline risk measure (APACHE II score) were performed. First, we used hospital mortality as the outcome instead of at 28 days, as APACHE II score was originally devised as a prediction tool for hospital mortality. Second, we investigated the potential impact of missing data on the results. In the VANISH trial, there were 47 patients who had at least one element of the acute physiology score missing, and 61 patients in the LeoPARDS trial. In the main analysis, normal scores were assumed for these elements, as for the main trial [13, 14]. In the HARP-2 trial, 66 patients had missing total APACHE II scores and were omitted from the main analysis but displayed in the forest plot. Missingness occurred pre-randomisation and hence is independent of the treatment effect but may affect the precision of the results. In the sensitivity analysis, we assumed patients with missing data were (i) equally likely to be in the high-risk group as those with complete data, (ii) 10% more likely and (iii) 10% less likely. APACHE II category was imputed 20 times under these assumptions, and the difference in RD and ratio of RR computed as for the main analysis, combining results across imputations using Rubin’s rules. A third sensitivity analysis was performed by recalibrating the APACHE II risk prediction using the whole RCT cohort, as internally developed risk models using both treatment arms are preferred to models developed on the control population, as highlighted by Burke et al. [27]. A logistic regression model for 28-day mortality was constructed with the following covariates: APACHE II points from each acute physiology component, age points, chronic health points, post-emergency surgery and diagnostic category weight. The resulting prediction was used as a measure of baseline risk, assessing HTE as in the main analysis. To avoid spurious associations from categorisation of APACHE II score [28], we performed a fourth sensitivity analysis, by treating APACHE II as a continuous variable in a logistic regression model. Relative HTE was quantified by the interaction between APACHE II score and treatment, expressed as a ratio of odds ratios. Additive HTE was illustrated by plotting the estimated absolute difference in mortality between treatment groups across the range of APACHE II.

The 28-day mortality, between the intervention and control arms, in the VANISH, LeoPARDS and HARP-2 trials was not significantly different (Table 1). The illness severity (using the total APACHE II score) was lower in the HARP-2 trial, compared to those in the VANISH and LeoPARDS trials (Fig. 1). The EQuOR highlighted significant heterogeneity of risk of death in all three RCTs for all three risk measures.

The 28-day mortality increased in the vasopressin and in the norepinephrine group, with increasing baseline risk measures (Fig. 1). For the primary analysis with APACHE II score as baseline risk of death measure, there was no evidence of HTE for vasopressin in either absolute terms (risk difference for low APACHE II 0.02 (− 0.09, 0.13) and high APACHE II 0.05 (− 0.08, 0.19); difference in risk difference 0.04 (− 0.14, 0.21)) or relative terms (relative risk for low APACHE II 1.09 (0.64, 1.86) and high APACHE II 1.15 (0.08, 1.64); ratio of relative risk 1.05 (0.55, 2.00)) (Fig. 2). For the secondary risk measures, the estimates of HTE for vasopressin were larger with wider CI for APS-APII (Fig. 3) and smaller in magnitude for R (Fig. 4).

For the primary analysis with APACHE II score as baseline risk of death measure, there was no evidence of HTE for hydrocortisone in either absolute terms (risk difference for low APACHE II 0.02 (− 0.12, 0.17) and high APACHE II 0.06 (− 0.10, 0.21); difference in risk difference 0.03 (− 0.18, 0.25) or relative terms (relative risk for low APACHE II 1.11 (0.62, 1.99) and high APACHE II 1.15 (0.79, 1.67); ratio of relative risk 1.04 (0.52, 2.08)). For the secondary risk measures, the estimates of HTE for hydrocortisone were similar for APS-APII (Fig. 3) and larger in magnitude for R (Fig. 4).

The 28-day mortality increased in the levosimendan and in the placebo group, with increasing baseline risk measures (Fig. 1). For the primary analysis with APACHE II score as baseline risk of death measure, there was no evidence of HTE for levosimendan in either absolute terms (risk difference for low APACHE II 0.05 (− 0.04, 0.15) and high APACHE II 0.04 (− 0.08, 0.16); difference in risk difference − 0.02 (− 0.17 to 0.14)) or relative terms (relative risk for low APACHE II 1.34 (0.78, 2.31) and high APACHE II 1.09 (0.84, 1.41); ratio of relative risk 0.81 (0.44 to 1.48)) (Fig. 2). For the secondary risk measures, the estimates of HTE for levosimendan were larger for APS-APII (Fig. 3) and in the opposite direction for R (Fig. 4).

The 28-day mortality increased in the simvastatin group and in the standard care group, with increasing baseline risk measures (Fig. 1). For the primary analysis with APACHE II score as baseline risk of death measure, we observed HTE for simvastatin in absolute terms (risk difference for low APACHE II − 0.15 (− 0.22, − 0.07) and high APACHE II 0.19 (− 0.01, 0.39); difference in risk difference 0.34 (0.12, 0.55) (p = 0.02)) and in relative terms (relative risk for low APACHE II 0.45 (0.28, 0.72) and high APACHE II 1.61 (0.95, 2.71); ratio of relative risk 3.57 (1.77 to 7.17)). Simvastatin reduced mortality in the low APACHE II group and increased mortality in the high APACHE II group (Fig. 2). As raw data APACHE II score data were not available, we have not reported any secondary risk measures for the HARP-2 trial.

We plotted the proportions of serious adverse events by low and high APACHE II groups in each trial, to explore whether the pattern of adverse event distribution could explain any HTE in mortality. In all three RCTs, both in the intervention and control trial arms, there was no pattern in serious adverse events that could explain HTE in mortality (Additional file 1: Figure S1).

Results from sensitivity analyses were consistent with the main analyses for the VANISH and LeoPARDS trials (Additional file 1: Table S1, Table S2, Figure S2, Figure S3 and Figure S4). HTE effects were attenuated in the sensitivity analyses for the HARP-2 trial under different assumptions for the missing data (e.g. ratio of relative risk was 2.86 (1.47, 5.57) when we assumed that patients with missing APACHE II data were more likely to be high risk; all other results were less attenuated Additional file 1: Table S1). Differences were also smaller when hospital mortality was used as the outcome (difference in risk difference 0.25 (0.03, 0.48); ratio of RR 2.34 (1.31, 4.18), Additional file 1: Figure S1) and when HTE was assessed across the continuous range of APACHE II score (ratio of odds ratio for a 5-point increase in APACHE II 1.33 (0.93, 1.90) Additional file 1: Table S2 and Figure S4).

There are a number of possible reasons why we did not observe HTE consistently in all our analyses. All three trials we assessed have many features of explanatory trials [29], which by their design limit HTE in comparison to pragmatic trials, such as through narrower eligibility criteria, intensity of follow-up and non-mortality primary outcomes. Therefore, demonstrable HTE is less likely in these trials, though its evaluation remains important. Our findings may be true in that HTE may be less marked in sepsis and ARDS as mortality may be driven by non-modifiable risk factors such as older age and presence of comorbid conditions, alongside illness attributable risk, generating many “minimal contributory causes” of mortality [30], when compared to illnesses such as retroviral disease [31]. It could be that the effects of treatments we assessed on mortality are both small and with limited variability across baseline risk of death resulting in minimal HTE. Another explanation for not observing HTE may be that there is no true treatment effect difference between subgroups enriched on prognosis with APACHE II score. Given the sample size in the trials assessed, we may only have power to detect large interaction effects, which requires either a large treatment effect in one or more subgroups, opposing treatment effects between subgroups or differential adverse event risk between subgroups.

A key comparison to consider is the contrasting results with RCT simulations by Iwashyna and colleagues [10]. Their simulations assumed that the trial participants’ odds of 30-day mortality will be influenced by the severity of acute respiratory failure, comorbid conditions, the treatment’s reduction in mortality from the primary illness and the treatment’s fatal adverse effect rates. Furthermore, Iwashyna and colleagues assumed constant relative treatment effects, constant harms and mortality patterns predicted by their simulation model. We used 28-day mortality, for our primary analysis. We considered baseline risk of death as a function of acute illness severity using the total APACHE II score, which is in line with the conceptual arguments put forward by Kent and colleagues [1] that HTE emerges from the risk of outcome (28-day mortality in our study), the risk of treatment-related harm and direct treatment-effect modification. Importantly, the data in our trials do not follow the constant relative treatment effects, constant harms and therefore the mortality patterns described by Iwashyna and colleagues [10].

Recently, Semler and colleagues reported a pragmatic, cluster-randomised, multiple-crossover trial of saline versus balance crystalloids in critically ill patients, with no difference in primary outcome of major adverse kidney events within 30 days (MAKE30), a composite of in-hospital death, new receipt of renal-replacement therapy and persistent renal dysfunction [32], but reported presence of HTE for this outcome with a multivariable model specifically calibrated for this outcome [33]. In contrast, our analytic strategy ascertained whether the observed treatment effect differed by the pre-randomisation baseline risk of death multivariable model (APACHE II score), which helped us to compare multiple treatments in critically ill patients with sepsis or ARDS.

We explored heterogeneity in absolute and in relative treatment effects, in sepsis and ARDS, for four different treatments and using three different measures of baseline risk. The primary baseline risk measure, APACHE II, is an established, validated predictor of mortality in this population. Two variations on this measure were investigated to check the consistency of the results, along with several sensitivity analyses. We used a composite risk score (APACHE II) for its superior performance for baseline risk estimation, as highlighted by Kent and colleagues [1] and a recommendation for future studies on HTE assessment [10]. There were insufficient numbers to examine HTE across smaller groups (e.g. quartiles). None of the RCTs included in this study had 28-day mortality as the primary outcome; it is possible that we were underpowered to detect HTE. The primary outcomes were not suitable for HTE analysis as they were continuous rather than binary and without an appropriate baseline measure, though the existing HTE framework could be adapted for some continuous outcomes, such as change from baseline organ dysfunction.

Aside from the ARDS or sepsis illness characteristics, it is likely that biological mechanisms determining differences in treatment effect will vary with the intervention tested. Therefore, using a generic physiology-based multivariable model such as APACHE II with biomarkers that provide both prognostic and predictive enrichment or intervention-specific predictive enrichment coupled may be a better approach to defining a study population. For example, an ARDS sub-population with greater inflammation and higher mortality was more likely to benefit from simvastatin [21, 34], and aside from severity of septic shock, the treatment effect of vasopressin was associated with biological differences within the trial population [16, 35]. Similarly, biomarkers derived from whole blood transcriptomics could help enrich septic shock patients for corticosteroid therapy [36‐38]. It is plausible that when HTE is assessed for an intervention using data from a single trial, we are unlikely to detect it unless HTE effects are large. This generates an argument to assess HTE using trials of the similar treatment-condition combination or of the same condition and broader group of treatments with similar enough mechanism of treatment effect, or consider intervention-specific multivariable models. As suggested by Iwashyna and colleagues, perhaps HTE assessment should form part of a priori analyses plans in future clinical trials. As HTE is about the variation in effectiveness, standardising the baseline risk measure between RCTs, including HTE assessment as a priori analyses, ensuring that the outcome used in HTE analyses is patient-centered (such as mortality) and incorporating the proposals within the Core Outcome Measures in Effectiveness Trials guidelines will enable pooling of HTE analysis across future trials [39].