This study consisted of a derivation cohort of patients in PARADIGM-HF and a validation cohort in ATMOSPHERE. Patients having an ICD or cardiac resynchronization therapy with a defibrillator (CRT-D) were excluded as these devices selectively reduce the risk of one of the two modes of death of interest. The design and results of both studies are published [15, 16].

Briefly, PARADIGM-HF evaluated the effect of LCZ696 with enalapril in 8399 patients with a left ventricular ejection fraction (LVEF) ≤ 40% (changed to ≤ 35% by amendment) and NYHA class II-IV HF, in addition to recommended treatment including an angiotensin converter enzyme (ACE) inhibitor or angiotensin receptor blocker (ARB) and a beta-blocker (unless contraindicated) and a MRA (if indicated). Patients were required to have a plasma B-type natriuretic peptide (BNP) ≥ 150 pg/ml (or N-terminal pro-BNP [NT-proBNP] ≥ 600 pg/ml), or a BNP ≥ 100 pg/ml (or NT-proBNP ≥ 400 pg/ml) and a HF hospitalization within the past 12 months. The key exclusion criteria included intolerance of ACE inhibitors or ARBs, a history of angioedema, symptomatic hypotension, a systolic blood pressure (SBP) < 100 mmHg at screening (< 95 mmHg at randomization), an estimated glomerular filtration rate (eGFR) < 30 ml/min/1.73m², and a serum potassium level > 5.2 mmol/L at screening (> 5.4 mmol/L at randomization). Patients were accrued from December 8, 2009, through November 23, 2012 from 1043 centers in 47 countries, and the follow-up ended on March 31, 2014. The median follow-up was 27 months.

ATMOSPHERE compared aliskiren monotherapy and aliskiren/enalapril combination therapy with enalapril monotherapy in 7016 patients with NYHA class II-IV HF with a LVEF ≤ 35% and elevated plasma BNP levels (same criteria as in PARADIGM-HF). The main exclusion criteria were very similar to PARADIGM-HF, with more stringent requirements in renal function and serum potassium levels but a lower threshold of SBP. Patients were required to be treated with a beta-blocker (unless contraindicated) and could be treated with a MRA if felt to be indicated by the investigator. Patients were enrolled from March 13, 2009, to December 26, 2013 from 789 centers in 43 countries, and were followed up until July 31, 2015. The median follow-up was 36.6 months.

Both trials used a composite primary outcome of cardiovascular death or HF hospitalization. All patients provided written informed consent.

In each trial, all deaths were adjudicated by the same committee using pre-specified criteria, in a blinded fashion. The same definitions for modes of death were used. SD was defined as death occurring unexpectedly in an otherwise stable patient, further classified as death witnessed or patient last seen alive < 1 h previously, and death in a patient last seen alive ≥ 1 h and < 24 h previously. PFD was defined as death occurring in the context of clinically worsening symptoms/signs of HF without evidence of another cause of death, including death as a complication of the implantation of a ventricular assist device, cardiac transplant or other surgery primarily for refractory HF, and death after referral to hospice specifically for progressive HF.

To identify predictors for each mode of death, a broad spectrum of baseline variables (N = 62) were separately assessed in PARADIGM-HF (Table 1). These variables included demographics, clinical variables, medical history, ECG parameters, and laboratory tests including NT-proBNP. In each trial, patient demographics and medical history were collected at baseline, physical examination, blood pressure, pulse and anthropometrical measurements were also performed, and this information was recorded in the electronic case report form (eCRF) by the investigators. A 12-lead ECG was performed at baseline and interpretation of the tracing was made by a qualified physician and documented on the ECG section of the eCRF. All laboratory tests were performed in a central laboratory, according to the pre-specified laboratory manual with details about specimen collections, shipment of samples and reporting of results, except potassium values and eGFR. These two tests were performed in a local laboratory and eGFR was calculated using the Modification of Diet in Renal Disease (MDRD) equation. A full set of baseline variables was collected in most patients, and patients with missing values were excluded in these analysis (< 2.5%). No difference was observed between the overall randomized patients and the cohort with all baseline variables available.

The baseline characteristics by cohort were compared using Student’s t test or Mann–Whitney U test as appropriate for continuous variables, and Chi-square test for categorical variables. For each mode of death, the event rate was calculated per 100 patient-years, and the cumulative incidences over time were plotted and compared by cohort using the Pepe–Mori test which counted death from other causes as a competing risk.

A univariable Fine and Gray sub-distribution hazards model was first performed to assess the influence of each prediction variable on the cumulative incidence of each mode of death [17]. For each continuous variable, linearity was examined using the restricted cubic spline method. If the response appeared nonlinear, certain cut-off values or transformation were applied according to the spline curves and clinical relevance. For categorical variables, appropriate dummy variables were used. The validity of the proportional sub-distribution hazards assumption was examined using time varying terms. For each variable, the statistical strength for predicting each mode of death was quantified by Χ² values with one degree of freedom.

For each outcome, we used a multivariable Fine–Gray model with backward stepwise selection based on Akaike information criterion (i.e., equivalent to p = 0.157), starting with a full model including all candidate variables. The predictor selection process was repeated in 200 bootstrap samples, each was sampled with replacement from the original PARADIGM-HF dataset with the same sample size as the original. To minimize the chance of inclusion of weak and uninformative predictors which might lead to model overfitting and optimism, we included variables in the final model that were retained in > 50% of all bootstrap datasets and were statistically significant. Since LVEF is an established prognostic factor for pump failure death, we included it in the final model regardless of the abovementioned inclusion criteria. For each mode of death, the final model was refitted into 200 bootstrap samples to get the average predictor coefficients. These averaged coefficients were used to calculate the individual risk score which is the sum of the products of each predictor value and its corresponding coefficient. Predicted cumulative incidences over time by quartile of risk scores were plotted against the observed Aalen-Johansen estimators to assess model performance [18]. Model calibration was examined by comparing observed-predicted pairs of curves in each quartile over time. Model discrimination was examined by visually assessing the spread of each set of curves (the wider the better) and by calculating Harrell’s C and C-index at 1-, 2-, and 3-year adjusting for right censoring [19].

To correct for optimism, internal validation was undertaken by bootstrapping approach. In detail, the C statistic of the derived model was determined in each bootstrap sample from which it was generated, and also in the original dataset, and the difference between these two C statistics was calculated and then averaged over 200 samples to give an estimate of the optimism. The optimism corrected estimate of the C statistic was then calculated as the naïve C statistic minus the estimated optimism. External validation was performed in the ATMOSPHERE cohort by fitting a univariable Fine–Gray regression on risk score which was the sum of average coefficients of predictors for each model from PARADIGM-HF multiplied by its corresponding predictor values in ATMOSPHERE. Model performance in validation was assessed using the same approach mentioned above.

To determine whether the prediction variables had a different effect on each outcome, all predictors from both models were fitted into cause-specific Cox regression models using the Lunn–McNeil method [20].

To validate the SHFM in contemporary cohorts and to compare our models with the SHFM, a SHFM score was calculated for each patient in PARADIGM-HF and ATMOSPHERE and the ability of the SHFM to discriminate between SD and PFD was assessed. We also validated the SPRM in both cohorts using logistic regression analysis and assessed its discrimination using Receiver Operating Characteristic Area Under the Curve (ROC AUC), an equivalent to Harrell’s C.

The derivation cohort included 7156 patients from PARADIGM-HF after excluding 1243 patients with an ICD or CRT-D. As can be seen in Table 1, there was a predominance of males (77%) with a mean age of 63.7 years. The mean LVEF was 29.9%, the vast majority of patients were in NYHA class II–III (mainly in class II) and most had an ischemic etiology (58.7%). There was a high rate of evidence-based treatment with 92.4% having a beta-blocker and 55.5% receiving a MRA.

In PARADIGM-HF, there were 1344 death events including 525 SD and 261 PFD over a median follow-up of 27 months. The annual rate was 3.4 (95% CI 3.1–3.7) per 100 patient-years for SD and 1.7 (95% CI 1.5–1.9) per 100 patient-years for PFD, respectively.

Table 2 shows the 25 most powerful predictors for SD from the univariate analysis, and 10 of them were independent predictors for SD in the multivariable model: male sex, Asian or Black race, NYHA class III/IV vs. I/II, prior CABG or PCI, history of myocardial infarction, cancer history, treatment with LCZ696 compared with enalapril, left ventricular hypertrophy (LVH) on ECG, QRS duration (90–120 ms), and plasma NT-proBNP (log-transformed).

The 25 strongest predictors for PFD from the univariate analysis are displayed in Table 3, 11 of which were included in the multivariable model: SBP (up to 130 mmHg), NYHA class III/IV vs. I/II, LVEF (up to 40%), ischemic etiology, a diagnosis of HF for > 5 years, HF duration > 1 and ≤ 5 years, bundle branch block (BBB) on ECG, serum albumin concentration (30–45 g/L), creatinine (1.0–2.5 mg/dL) and chloride (90–106 mmol/L), and plasma NT-proBNP (log-transformed).

The SD model showed good discrimination, with Harrell’s C of 0.68 (95% CI 0.66–0.71) and C-index of 0.67, 0.68 and 0.67 at 1, 2 and 3 years, respectively. With boot-strapping interval validation, the Harrell’s C corrected for optimism was 0.67. The curves for observed Aalen–Johansen estimators and predicted cumulative incidences were almost identical over time in each quartile of risk score based on the model, indicating good calibration (Fig. 1a). Both sets of quartile risk curves are well separated, confirming the discrimination suggested by the C statistics.

The PFD model also calibrated well: the curves for the predicted cumulative incidence agreed well with the corresponding observed curves for each quartile of risk score (Fig. 1b). Although the PFD model was less able to distinguish between the lowest two risk quartiles, it identified the highest and second-highest quartiles, which had over 10 times and 3 times the risk, respectively, of the lowest quartile at 3 years (Fig. 1b). The excellent discrimination was confirmed by high Harrell’s C values of 0.79 (95% CI 0.76–0.82) and C-index of 0.82, 0.79 and 0.77 over 1, 2 and 3 years, separately. With boot-strapping interval validation, the Harrell’s C corrected for optimism was 0.78.

External validation was performed in 5968 patients in ATMOSPHERE after excluding 1048 patients with an ICD or CRT-D. The baseline characteristics of these patients were similar, for the most part, to PARADIGM-HF. However, some differences were observed. In ATMOSPHERE, there was a higher proportion of Asian patients, but a lower proportion of patients with a history of hypertension, diabetes or renal dysfunction. Fewer patients in ATMOSPHERE had received a MRA. The median plasma NT-proBNP level was lower in ATMOSPHERE. Patient characteristics by cohort are summarized in Table 1.

During a median 37.7 months of follow-up, 1644 death events occurred in ATMOSPHERE including 607 SD and 305 PFD, with the corresponding annual rates very similar to PARADIGM-HF. The cohort-specific cumulative incidences for SD were nearly identical, and this was also the case for PFD (Online Fig. A2).

For the SD model, discrimination was largely stable in the validation cohort with a Harrell’s C of 0.66 (95% CI 0.64–0.69) and C-index of 0.71, 0.68 and 0.67 at 1, 2 and 3 years, respectively. Although the highest quartile under-predicted the cumulative incidence while the second-lowest quartile over-estimated the rate over time, the predicted and observed cumulative incidences were broadly similar in the rest quartiles (Fig. 2a).

For the PFD model, discrimination was slightly decreased but remained robust in the validation cohort with a Harrell’s C of 0.75 (95% CI 0.72–0.78) and C-index of 0.78, 0.76 and 0.73 at 1, 2 and 3 years, respectively. Calibration was reasonable except in the highest risk subgroup, where an underestimation was observed in the early period of follow-up (Fig. 2b).

The probability that out of a (hypothetical) population of 100 patients with the same characteristics as an observed patient, x% will have the event is often loosely described as an “individual risk”. Such a risk for SD and PFD can be calculated by adding up the products of their predictor values and the coefficients from the multivariable models presented in Tables 2 and 3, respectively. For the obtained risk scores, the corresponding cumulative incidences for SD and PFD within 3 years can be estimated using the curves displayed in Fig. 3a, b which shows the distribution of the risk score for each mode of death and its relationship with the corresponding predicted cumulative incidence within 3 years, respectively (Examples are given in Online Supplement).

NYHA class and NT-proBNP were included in both models. More advanced NYHA class and higher NT-proBNP were associated with higher risks of both modes of death, with both associations stronger for PFD than SD, although the differences did not reach significance (Table 4). Ischemic etiology had an opposite association with mode-specific death, i.e. it was associated with a higher risk of SD but a lower risk of PFD.

Male sex, Asian or Black race, a history of MI and LVH on ECG were associated with a higher risk of SD, whereas history of cancer, prior CABG or PCI and treatment with LCZ696 (compared with enalapril) were associated with a lower risk of SD, but none of these variables were predictive for PFD (Table 4). On the contrary, longer HF duration, a higher level of serum creatinine, and lower serum albumin or chloride were associated with a higher risk of PFD but not of SD (p for inequality all < 0.05).

The SHFM is reported to show good discrimination for SD and PFD with 1-year ROC AUC of 0.68 (95% CI 0.65–0.70) and 0.85 (95% CI 0.83–0.87), respectively [12]. However, when validated in our more contemporary cohorts, its discrimination declined for SD, with C statistics of 0.57 (0.53–0.60) at 1 year and 0.58 (0.55–0.60) at 3 years in PARADIGM-HF and 0.62 (0.58–0.66) at 1 year and 0.63 (0.61–0.66) at 3 years in ATMOSPHERE. A marked decrease in discrimination from that reported was also observed for PFD, with 1- and 3-year C statistics of 0.72 (0.67–0.77) and 0.69 (0.66–0.72) in PARADIGM-HF and 0.71 (0.64–0.77) and 0.65 (0.61–0.69) in ATMOSPHERE.

The SPRM showed poor discrimination for SD in PARADIGM-HF (ROC AUC 0.57) and ATMOSPHERE (ROC AUC 0.54). Using a 25% of 1-year mortality rate and a threshold proportion of mortality due to SD of 42% (used by the SPRM authors to identify patients most likely to benefit from an ICD), this model allocated most patients to the upper left quadrant, i.e. those who had a low mortality, primarily due to SD, and an indication for an ICD (Online Fig. A3).