Predicting the cumulative risk of death during hospitalization by modeling weekend, weekday and diurnal mortality risks

Prognostic models help clinicians identify those patients most at risk of negative outcomes, and tailor therapy to manage that risk [1, 2]. Such models are typically developed for specific conditions or diseases, combining clinical indicators (such as patient or disease characteristics) and test results that stratify populations by risk [3]. Performance varies and even the best models often have good but not high classification accuracy [2]. Models are most often developed using data from specific periods of time, patient cohorts or geographies, and when they are evaluated against new populations, may not perform as well as in the original population [3].

Variation in predictive accuracy of models in part must be due to variations in disease patterns across populations, but is also likely due to local variations in clinical service and practice. If prognostic models specifically incorporated information about health service delivery the result might be more accurate, generalizable clinical tools. There are for example well known risks associated with hospitalization that can lead to patient harm or death [4]. These risks vary with diagnosis [5, 6], hospital [7, 8] and route of admission [9]. Risks also are known to vary significantly with the time that hospitalization occurs, with weekend admissions carrying greater risk than other days [10‐17].

Modeling the risk of exposure to hospitalization should allow for more informed decisions around the decision to admit or discharge. Indeed, whilst it is standard to develop risk-benefit models for new interventions such as medicines or procedures, we do not routinely apply the same logic to the decision to admit to or discharge from hospital, one of the most universal of all clinical interventions. In the same way that radiation exposure models help determine when a patient has exceeded a safe radiation dose, it should be feasible to develop models that determine when patients have exceeded a safe ‘dose’ of hospitalization.

Some work has explored the risk of harm to a patient following exposure to an average day in hospitalization [10, 18]. Combining models which predict time-varying risks associated with hospitalization with traditional disease-specific prediction rules should theoretically result in much more accurate predictive tools which can be used to update risk estimates as an admission progresses over time. In this paper we report on the development and evaluation of one such family of models, using only standard administrative data.

The demographic distribution of patients in the training (July 2000 – December 2006) and test (2007) data sets is summarized in Table 1. Age and comorbidity patterns were similar in both groups. Mean length of stay dropped from 3.21 days in the seven training years to 2.75 in the final test year, and the death rate (the ratio of deaths during hospitalization and deaths within 7 days of discharge to all admissions) also dropped from 1.8% to 1.3%.

Modeling the time accumulated risk associated with hospital stay has allowed the development of a class of predictive models that, when combined with simple disease data appears to substantially outperform many disease specific models alone. This is despite the disease specific information used in our model being represented by relatively simple linear combination of administrative information including the Charlson index and DRG categories. One would anticipate even better performance if an exposure model was combined with a clinically-based disease prognostic model, relying for example on patient record or disease registry data, as well as better modeling the relationship of variables to mortality (e.g. age and death are better represented as a log linear rather than linear relationship [28]).

A striking feature of our data set is the strong daily rhythm to risk of death, which echoes the well-known weekly variation of risk of death, which peaks with weekend admission (Figure 1). Our data shows the risk of mortality peaks in the early evening and is at its lowest in the morning (Figure 3). Daycare did not appear to increase risk of death but this attenuated at weekends – the weekend effect. Compared to admission on Monday day, the odds ratios for risk of death were worse for evenings and nights, and Sunday daytime had a 1.7 times greater risk than Monday daytime.

It is hard to impute causation to such observational data and we can only speculate why the diurnal rhythm exists. There are two main causal readings to be explored. Firstly, it may be that such risks of death are a feature of the health service, where patients experience increased risk because of reduced availability or quality of clinical services. The push for health services to provide uniform “24/7” care is a response to this concern. The second causal reading is equally plausible, and that is that some patient groups are associated with greater risk of death at different times. This may be due to a selection bias, where sicker patients present to hospital at given times, or in some instances may have a biological cause. Our recent analysis of the weekend effect identified that both causal readings are plausible, and that different patient groups demonstrate either care or illness risk patterns [25].

We show that performance of models using patient subgroups based upon DRGs is also variable. For example, performance was not strong for patients with major arrhythmia or cardiac arrest. As 8,254 admissions with major arrhythmia or cardiac arrest were present in the training data set, with an average LOS of 2.69, it seems unlikely that sample size or LOS were factors influencing performance. In contrast model performance was strong for DRGs associated with leukemia or lymphoma. In our recent analysis of weekend effects, increased weekend deaths for acute cardiovascular events appear to be related to service quality (e.g. lack of availability of imaging or stenting services out of hours). For oncological patients however, the cause seems to be that a sicker cohort of patients is presenting at the weekend [25]. This suggests that the models developed here may be better at modeling disease rather than service specific factors in its exposure risk profile, and warrants further investigation. Where there are time varying risks associated with a health service for a particular patient group, specific modeling of that independent risk may be needed.

Modeling the route of admission (Emergency vs. non-Emergency Department) did not appear to confer any advantage despite it being show by others to be a major predictor of mortality [9]. One possible explanation is that information about route of admission is already captured implicitly in the exposure model. For example, patients admitted out of hours may be most likely to presenting via emergency, and so the out of hours risk is already modeling route of admission as a hidden covariate. Further work is needed to understand the nexus between time-varying risk and route of admission.

Adding new variables to the basic model helped improve model accuracy at the expense of increasing AIC and BIC values (Table 2). However AIC/BIC value increments were modest and the improvement in model performance substantial. Further, interpreting AIC and BIC values is hard with large data sets, and they typically are more valuable when models are developed on small data sets. With large data sets, AIC and BIC may end up preferring over-fitted models [24].

Most current clinical predictive tools provide snapshot predictions of risk, independent of time. Yet clearly risks should vary with time. In this study predictive performance was best in the first week of admission, and performance early in the first week was exceptionally strong. One would expect as admission time lengthens that many additional factors come into play to modify risk, and therefore long run prediction becomes increasingly difficult – a situation well known in forecasting. It is also the case that patient mix changes with time, and those patients with long stays represent a different cohort with different risk profile, and may need to be modeled separately. Sample sizes available for such model development also diminish as length of stay increases, making it harder to develop generalizable models.

Models such as those presented here could be used to forecast risks such as death or specific clinical events, and to update those forecasts as additional information accumulates. Using the full model developed here, it is clear that the risk profile for patients varies with both time and day of admission (Figure 4). Given that predictive accuracy changes with LOS, it would also be appropriate to provide clinicians with estimates of the accuracy of each prediction and emphasize for example that confidence is stronger in the short rather than long run. The next stage in developing the models presented here would be to evaluate their capacity to forecast future risk of death for patients given their current exposure to hospitalization.

One challenge for creators of clinical prediction models is that they often fail to generalize to settings beyond those from which they are created, because they model a point in time, local disease patterns or service provision patterns [1]. The approach developed here is likely to be highly generalizable as it relies on no location specific information, except for the weightings associated with different model variables. These weightings could be calibrated using the best available data for a given location. Further, the evaluation here was on patients from a large geography encompassing urban, rural and remote settings, and hospitals ranged from public to private, and small to large teaching and specialist referral centers. Addition of location specific variables and disease specific elements are likely to further enhance the general model reported here.

Risks of death associated with hospitalization vary not just with the day of the week but also time of the day, and can be used to make predictions about the cumulative risk of death associated with an individual’s hospitalization. Enhancing disease specific prognostic models with estimates of the cumulative risk of exposure to hospitalization appears to greatly enhance predictive performance. Risk exposure models should find utility both in enhancing standard prognostic models as well as estimating the risk of continuation of hospitalization.