Identifying patients at highest-risk: the best timing to apply a readmission predictive model

Interventions that are aimed at the prevention of hospital readmissions are increasingly guided by computerized risk prediction models, which identify high-risk patients [1]. To date, most readmission prediction models are implemented upon patient discharge [2]. A growing body of evidence, however, indicates that interventions that include an early in-hospital component, such as comprehensive discharge planning [3], are key to reducing readmissions, thus, highlighting the need for early, within hospitalization high-risk prediction.

Early high-risk patient identification is becoming increasingly possible. With the advent of electronic health records (EHRs) [4], detailed data on key risk factors, including clinical and healthcare utilization, are also available from the preadmission period [5]. Previously, we showed that such a pre-admission prediction model (the Preadmission Readmission Detection Model [PREADM]) provides accurate high-risk assessment [6]. Similarly, a multi-condition electronic model, based on data available at admission, showed that meaningful patient-level risk stratification of readmission risk can occur early in the hospital stay without the need to wait for further information at time of discharge [7]. A recent review has demonstrated that preadmission prediction models performed comparably well to the at-discharge models [2].

Whether identification of patients at high-risk for readmission should be performed at the beginning or at the end of the index hospitalization is not only a question of predictive accuracy, it also depends on the types of readmission prevention interventions to which high-risk patients are referred. Thus, in-hospital interventions can benefit from early high-risk identification of targeted patients, and programs targeting the post-hospitalization phase, should rely on risk prediction at the time of discharge. Thus, the trade-off between early versus at-discharge prediction and the optimal timing of high-risk case identification remains to be determined. To address this gap, the aim of this study was to examine a high-risk patient selection with readmission prediction models using data available at two time points: (1) at admission and (2) at the time of hospital discharge.

The final cohort included 35,156 index hospital admissions (24,510 unique inpatients) after we excluded 5096 patient’s admissions who died before discharge or did not have a continuous membership in Clalit. The flow-chart for selection of the study's population appears in Fig. 1. The study population was 47.9% male, 78.9 years of age on average, and predominantly Jewish (88.4%). The mean index hospital admission lasted 5.3 days, and 6933 (19.7%) index admissions resulted in 30-day readmission (Table 1). Patients who were readmitted differed from non-readmitted patients in terms of their demographic, clinical and prior health service use characteristics.

Model derivation was performed on 24,599 admissions and tested on 10,557 admissions. Our final model included 17 variables; the 11 PREADM model variables, and six from the HOSPITAL model. In the validation cohort, the PREADM-H model had fair discrimination, with a C-statistic of 0.68 (95% CI: 0.67–0.70). The PPV of the PREADM-H model in the highest risk categories (top 10 and 20%) was 43.0 and 36.1%, respectively and sensitivity and specificity in top 10% was 21.1 and 92.9% respectively (Table 2).

Figure 2 shows that 78% of those categorized by the PREADM-H at the highest decile of risk (with a PPV of 45%) were classified similarly by the PREADM model. The remaining 22% who were classified by the PREADM-H highest decile, but not by the PREADM, had a PPV of 37%. Conversely, those classified by the PREADM into the highest decile but not by the PREADM-H (n = 218) had a PPV of 31%. A similar picture emerged when examining the differences in populations identified as the 20% highest risk group. In the highest quintile, 82% of those categorized by the PREADM-H (with a PPV of 38%) were also categorized at the highest quintile by the PREADM. The PPV for the remaining 18% who were not identified by the PREADM at the highest quantile was 29%. Thus, applying the PREADM at baseline and PREADM-H at-discharge allowed for accurate detection of an additional 85 subsequently readmitted patients (37% of 229 patients), with a cutoff point for the 10% highest risk group, and 110 patients (31% of 359 patients), using a cutoff point for the 20% highest risk group, who would have otherwise been missed. A detailed account of percent of patients detected at high-risk for 30-day readmission at-admission (PREADM) vs. at-discharge (PREADM-H) appears in Fig. 3.

The characteristics of the two non-overlapping populations appear in Table 3. Patients with a high-risk PREADM score who were not identified as high-risk according to the PREDM-H model had more disability (71.6% vs. 55.5%, p value = 0.001), more chronic renal failure (65.1% vs. 50.7%, p value = 0.003), more hospital admissions in the past year (3.9 vs. 3.4, p value = 0.003) and less days from the previous hospitalization (39.2 vs. 54.0, p value = 0.001). Patients who were identified as non-high-risk by the PREADM model but were identified as high-risk by the PREADM-H model have longer LOS (87.3% vs. 15.6% with a LOS of at least 5 days, p value< 0.001), a larger percent were in an oncology treatment phase (31.4% vs. 12.8%, p value< 0.001), all had an urgent index admission (100% vs. 89.4%, p value< 0.001), almost all had a low hemoglobin level (98.7% vs. 49.1%, p value< 0.001) and about half had low sodium levels (48.9% vs. 4.6%, p value< 0.001).

Our results show that the timing of hospital readmission risk prediction both at admission and discharge should be considered when making the decision regarding which population should and can be identified for inclusion in readmission prevention programs. Use of the PREADM model allowed for early identification of high-risk patients, yet missed a portion (18–22%, depending on whether a 10% or 20% highest risk cut-off was used) whose readmission risk was almost as high. Alternatively, the PREADM-H enabled accounting for risk factors that accrued during the hospital stay, though missed some patients who had an a priori high-risk according to the PREADM and whose actual re-hospitalization rate was much higher than the general population (31% readmission rate). Also, as expected, the clinical characteristics of the population that was identified as high-risk by the PREADM-H model was different than those who were identified by the PREADM model (especially as to the within hospital risk-factors). Malignancy, arrhythmia, and number of primary care and specialist visits in the past year were statistically significantly associated with readmission in the univariate analysis (Table 1), and in PREADM model [6]. Yet, when included in a model with variables from the admission period (PREADM-H) they are no longer statistically significant. This is probably due to the inclusion of the LOS variable, which possibly also indirectly captures the complexity and severity of the patient’s overall condition. This is similar to at least part of the contribution of the above stated variables, which may explain why they were no longer statistically significantly associated with the readmission outcome. Taken together, our findings suggest that readmission risk identification should incorporate a two-time-point approach in which preadmission data are used to identify high-risk patients as early as possible during the index admission (with a PREADM type of algorithm) and an at-discharge “all-hospital” model (such as the PREADM-H model), which is applied to identify those who incur risk during the hospital stay.

A two-time-point risk identification approach is also compatible with evidence that reports on the effectiveness of readmission prevention interventions. Systematic reviews [3, 12, 13] have repeatedly shown that no intervention implemented alone is associated with reduced risk for 30-day readmissions. Rather, interventions including components that are implemented before and after discharge, such as transitional nurse visits and discharge follow-up appointments, achieve the greatest reduction in hospital readmissions [14‐17].

While the PREADM model, already in use in Clalit for early identification of high-risk patients and guiding physicians and nurses in prioritizing patients for inclusion in early readmission preventive programs that are tailored to meet their ongoing care needs (e.g., discharge planning or referral to a transitional nurse care). This study shows that there is value in the PREADM-H model being incorporated into practice to inform interventions implemented at the point of discharge, as well as communicated to the primary or ambulatory care teams to allow for selection of patients for inclusion in post-discharge targeted interventions. Our results provide an example of the potential complementary implementation of the predictive models to maximize their power in identifying various groups of high-risk patients for inclusion in within as well as post-discharge interventions.

This study’s findings add to the recent literature that addresses the need for new modeling approaches that provide innovative, actionable insights for risk stratification to improve the ability to prevent hospital readmissions [18]. An example of such an approach is multi-hypotheses causal analysis, which generates meaningful insights from health care claims data, guiding the design of care and intervention programs by developing more personalized interventions based on readmission risk associated with specific comorbidities [18] or improved understanding of causes of asthma-related readmission [19]. This is similar to our approach which identifies various groups of high-risk patients for inclusion in interventions by checking the patient’s risk at different time points throughout the hospitalization, rather than providing broad-scope interventions for all individuals.

Another consideration in the applicability of risk-prediction models relates to the compatibility between the type of analytical approach used for the development of the model and the purpose of the use of the model. For example, a model developed for risk adjustment purposes to allow for fair comparisons amongst hospitals’ readmission rates should be developed retrospectively, for which timely data availability does not play a factor [20]. Yet, a predictive model that evaluates patients at-risk for deciding on inclusion in preventive interventions requires the inclusion of data available in real-time from an EHR [21].

The timing of readmission risk prediction makes a difference in terms of the population identified at each prediction time point - at admission or discharge. As our model performance data shows, early risk identification (by the PREADM) allowed for accurate detection of patients who were highly likely to be readmitted (actual readmission rate of 31%) but were no longer detected at the highest-risk decile when at-discharge identification was applied (by the PREADM-H). Similarly, application of only early at-admission high-risk detection (PREADM) can miss patients with as high as about 37% likelihood for readmission who are newly identified at-discharge (PREADM-H). Thus, implementing at-admission models allows early identification and intervention but misses a portion of patients who are at high-risk of readmission due to factors accrued during the hospitalization period. Conversely, an at-discharge prediction model not only misses the opportunity for early intervention, but also fails to account for patients at high-risk for readmission who can be identified early in their hospital stay. To account for patient risk as well as the opportunity to intervene in order to address this risk, we recommend a combined and complementary approach of applying readmission risk detection at two-time-points, both at-admission and at-discharge.