A review of statistical estimators for risk-adjusted length of stay: analysis of the Australian and new Zealand intensive care adult patient data-base, 2008–2009

Length of stay during an intensive care unit (ICU) or hospital admission is a function of diverse patient and organisational input variables [1]. It is widely used as an indicator of performance [2] and is a determinant of costs, although resource allocation is also known to affect length of stay [3]. Not surprisingly, ICU length of stay has been the subject of frequent analysis [4‐9], with the majority of studies presenting cross-sectional analyses over a relatively short periods of months [10] to 1–2 years [9].

ICU patient length of stay (and costs) demonstrate skewed distribution and various statistical modelling strategies have been employed in analysis of such data [11‐14]; albeit linear regression (ordinary least squares regression, OLS) of the logged dependent variable has demonstrated a remarkable persistence [15]. Individual patient data, as accessed from ICU data-bases, have an intrinsic hierarchical structure (patients within ICUs) and due analytic consideration of this structure is also appropriate [16]. Using such data from the Australian and New Zealand Intensive Care (ANZICS) adult patient database (APD) [17], calendar years 2008–2009, the purpose of this paper was to: (i) compare the predictive performance and model specification of conventional estimators for skewed data (ICU length-of-stay); OLS (with both raw and log-scaled dependent variable) and generalised linear models (GLMs [14]), with more innovative approaches: multilevel or hierarchical linear mixed models (LMM) incorporating random effects [11, 16]; extended generalised linear models (EEE) with flexible link and variance functions [18]; estimators utilising skew-normal and skew-t multivariate distributions [19]; and finite mixture (FMM) models which consider the dependent variable as a mixture of distributions [20‐22]; and (ii) determine the effect of (ICU) mortality outcome upon length of stay, allowing for “endogenous variable bias” [23] by means of a treatment-effects model [24, 25] where specific allowance is made for the correlation of independent predictors and error terms. Under such conditions, model coefficients are biased [26].

The ANZICS adult patient database (APD) is a bi-national (Australia and New Zealand) voluntary data collection of individual ICU admissions which commenced in 1990. The data set requirements are specified in a data dictionary [27]. The current data-base does not incorporate coronary care units. Data is collected at the individual ICU and uploaded to the central repository (ANZICS APD) for processing and quality assurance; which consists of a cycle of error and exception checks, site feed-back, resubmission and incorporation into a final reporting data-set.

The ANZICS APD was interrogated to define an appropriate patient set, over the time period 2008–2009. Physiological variables collected were the worst in the first 24 hours after ICU admission. All first ICU admissions to a particular hospital for the period 2008–2009 were selected. Exclusions: patients with an ICU length of stay ≤ 4 hours; and patients aged < 16 years of age. Specific attention was directed to the fidelity of severity of illness records; in particular the scoring of the Glasgow Coma Score (GCS). Records were used only when all three components of the GCS were provided. Records where all physiological variables were missing were excluded and for the remaining records, missing variables were replaced to the normal range and weighted accordingly [28]. ICU length of stay, initially recorded in hours was transformed to days; patients with an ICU length of stay > 60 days were not considered, no formal trimming methods were employed [29].

The database for the period 2008–2009 contained records of 114798 patients with exclusion (4.20%) of patients having: ICU length of stay < 4 hours, incomplete GCS or APACHE III records. The final data set had missing ICU length of stay in 0.03% and 141 patients (0.001%) with ICU length of stay > 60 days (mean 81.98(24.37), range 60.05-199.67) not considered in analysis. Patients (n = 111663) admitted to 131 ICUs in the same number of hospitals, were of mean(SD) age 60.6(18.8) years, APACHE III score 52.5(28.9); 43.0% were female, 40.7% were mechanically ventilated (on the day of admission) and ICU mortality was 7.8%. ICU length of stay was 3.4(5.1) (median 1.8, inter-quartile range 2.8 (0.93-3.7)) days. Summary statistics of length of stay, APACHE III score, age and gender by ventilation status and patient surgical type are shown in Table 1. Overall ICU length of stay and the log transform are displayed in histogram form with normal curve overlay in Figure 1 (left and right panels respectively); raw scale length of stay demonstrated marked kurtosis and right skew (29.4 and 4.4 respectively) whilst the log transform demonstrated a more symmetrical distribution (kurtosis 3.1 and skewness 0.5). Comparison of the empirical distribution of ICU length of stay and its’ log and square root transformation showed poor approximation to standard theoretical distributions (normal, lognormal and gamma; Figure 2).

Comparative performance of the estimators for ICU length of stay is seen in Tables 2 and 3; the regression models had 52 fixed parameters, including the constant term. Over the range of measures assessing predictive performance and model specification (including goodness-of-fit), the LMM demonstrated a consistently better performance compared with the other 11 estimators; albeit all estimators revealed some problematic aspects of residual analyses. BIC for all the log-scale estimators (LMM, treatment effects- and log-skew-regressions) were considerably less than for estimation on the raw scale (Table 2). The final model variable set displayed a condition number of 19.4 and a mean VIF of 3.76; the best R² and concordance correlation coefficient was 0.22 and 0.334 respectively.

The outcome discrimination (Died-in-ICU) of the probit sub-equation of the treatment effects regression was excellent for both determination and validation sets (ROC curve area 0.92). The null hypothesis, that the correlation (ρ=0.065) between the error terms of the separate estimations is zero, was rejected at P = 0.0001. The Poisson GLM exhibited overdispersion (P value for “z” = 0.0001), the negative binomial GLM demonstrating an expected decrease in BIC; other performance measures were comparable between these two models. However, over-fitting, as assessed by the Copas test, was demonstrated in all other models except for the negative binomial GLM (Table 3). The “GLM family test” of the variance function of GLMs (Vary|x = α.[E(y|x)]γ [60]) rejected tests of γ for 0, 1, 2 and 3 (P ≤ 0.0001). The modified Park test generated estimates for γ of: 0.348(0.333, 0.364) for log-scale OLS; 1.659(1.609, 1.708) for GLM Poisson-log; 2.106(2.005, 2.207) for GLM gamma-log; and 2.352(2.088, 2.616) for GLM inverse Gaussian-log. The EEE link parameter (λ) estimate was 0.152(0.019, 0.285) and the variance function (θ₂) was 2.115(2.013, 2.218). The log-skew-normal and log-skew-t regressions reported α values of 1.073(0.933, 1.213) and 1.490(1.323, 1.659) respectively, suggesting positive-skew; the degrees-of-freedom for the log-skew-t regression was 10.53(7.97, 13.91) implying heavier-than-normal tails for the conditional distribution of (log) ICU length-of-stay. The 2 component FMM gamma model generated predicted mean ICU days of 0.56(0.32), range 0.06, 3.64 (component 1) and 3.69(2.45), range 0.49, 19.1 (component 2) as illustrated in Figure 3. The FMM negative binomial models demonstrated non-convergence.

Figures 4, 5, 6 show (i) upper panels: violin-plot distributional form of predictions and (ii) lower panels: Hosmer-Lemeshow-test parameter estimates, for determination and validation sets for each of the estimators, grouped by increasing scalar values of the Hosmer-Lemeshow parameter estimates. The graphical displays show a progressive lack-of-fit (Hosmer-Lemeshow test estimates) across Figures 4, 5, 6 in concert with corresponding over- and under-prediction as revealed by the violin plots. The LMM, log-scale OLS and treatment effects regression were the best calibrated; the EEE showed good calibration across the linear predictor deciles except for the upper deciles where substantial discordance was evident.

The β regression estimates for the binary variable “Died-in-ICU” are seen in Table 4, final column, and range from a low of 29% (treatment effects regression) to 295% (raw-scale OLS).The average marginal effects are also tabulated, and were generally decreased in magnitude compared with the β estimates. Figure 7 illustrates the contrasts of predictive margins for “Died-in-ICU” across increments of APACHE III score for raw-scale OLS and log-link Gaussian family GLM (see also Table 4). Similar decreases (not shown) of the contrasts of predictive margins across the APACHE III score range were evident for the (log) linear predictor for estimators using log-ICU length of stay.

Methodological recommendations for non-normal data analysis have usually been subsumed under the rubric of cost analysis [12, 15, 69]. The distribution of positive health expenditures (and length of stay) exhibit skew, kurtosis and heteroscedasticity, with a non-constant and increasing variance, albeit particular cost problems, such as the accommodation of a probability mass at “zero”, may not have immediate relevance to length of stay analysis [51, 69]. This review has suggested that a LMM had better overall performance compared with 11 other estimators. However, a number of issues pertain to comparator studies of possible estimators; in particular data structure and the full assessment, or otherwise, of model specification.

The hierarchical nature of some clinical and/or administrative data sets and the requirement for appropriate analysis has been commented upon above. As opposed to the volume-outcome literature [70], mixed models would appear to have been have been applied in a relatively small number of studies for length of stay analysis [16, 71‐75]. The use of administrative data [5, 16] may also be associated with integer based calendar–day recording which lacks the accuracy of fractional day based “exact” times [10, 71].

Because of the non-normality of length of stay, formal trimming [76, 77] or truncation [71, 77, 78] of the data, or deleting [5, 73, 79] “outliers” has been undertaken prior to possible data transformation. For instance, studies implementing the APACHE III [78] or IV [71] algorithms for predicting ICU length of stay have truncated the latter at 30 days, at a 1% data level; the same fraction as seen with data deletions [73]. The current study used a deletion fraction of 0.001%. The motivation for such strategies are various [80], but the theoretical basis of these data revisions has been questioned [29] and bias in the estimation of the mean has also been suggested [29, 69]. More importantly, large values of length of stay are “true” values [69] and data reduction cannot necessarily be defended as “…represent[ing] unusual cases…”, or because of “…analytical problems…” [73]. That trimming will cause decreases in residual variance in LMM was pointed out by Lee et al. [72] and was easily demonstrated in the current data set; where the residual variance was 0.646 for the full data, 0.600 at the 99^th percentile trim (< 26 days) and 0.508 for the 95^th percentile trim (< 12 days) of ICU length of stay. Model assessment must take note of data revisions.

Similarly, the repeated assertions [71, 72, 76, 77, 80, 81] that the objective of such data manipulations are model (in particular, OLS) requirements for normality of the dependent variable are problematic. As observed by Buntin and Zaslavsky “Plotting of the data with various transformations is thus a useful preliminary step, but ultimately the distribution of the residuals from the transformed model is critical because the model assumptions concern the distribution of the residuals, not of the data” [59]. That the residuals “…usually have a similar distribution to the original data…” [12] does not absolve the analyst from model based residual interrogation; the simple reporting of overall measures such as R²[77] or agreement indices of observed versus predicted [71] are not satisfactory proxies. Moreover, inference on β-coefficients is biased under heteroscedasticity.

Effect of ICU death on length of stay.

The β coefficient for “Died-in-ICU”, as a main effect, ranged from +29.7% (treatment effects regression, log scale) to +300% (OLS, raw scale), consistent with the β estimates for the factor “Death” of +0.536 (age < 65 years) and +0.439 (age ≥ 65 years) reported by Angus et al. [5], using a GLM (geometric family, identity link). However, the results of the marginal analysis (Table 4) implied over-prediction of the β-coefficient of “Died in ICU” across the estimators, albeit the average marginal effect was considered, as opposed to the marginal effect at the mean or at a particular value [65, 66]. Furthermore, there was a progressive decrease in the contrast, died in ICU versus alive in ICU, of predicted mean ICU days with an increase in the APACHE III score (Figure 7); this effect being consistent with findings from our previous study [82] and those of Rapaport et al. [8] and Woods et al. [9]. Of more interest was the demonstration, using the treatment effects regression model, that the covariate “Died-in-ICU” (which also has the status of an “outcome”) was associated with a significant correlation (ρ) between the error terms of the separate estimations (probit and linear regression) and was appropriately considered as endogenous [26]. That ρ was positive at 0.065 also indicated that OLS overestimated the “treatment” effect (Died-in-ICU); that is, a face-value interpretation of the β mortality coefficient was problematic. Endogeneity may also be suspected in mortality models where length of stay [7] or mortality probability [83] are entered as predictive covariates. Such regression of a variable upon its components has been termed a “dubious practice” [15]. This being said, the extension of 2-stage methods to non-linear models may not be without its own issues [15, 84, 85].

Estimators of length of stay.

The statistical basis of the various estimators used in the current paper has been canvassed in “Statistical methods”, above. These estimators, assessing the mean function u x, have seen extensive application and assessment in the cost data literature, but not necessarily for length of stay. The recent introduction of log-skew-normal and log-skew-t estimators has not afforded opportunity for extensive assessment. Within the ICU literature, the predominant estimators have been ordinary least squares regression, with [8] and without [7, 71, 78] log transformation; generalized linear model (geometric family with identity link) [5] and random intercept linear model [83]. Lee and co-workers have explored alternative approaches suited to the analysis of hospital length of stay [74, 86].

It is perhaps not surprising that in a large hierarchically structured data set that the LMM appeared to dominate. The suggestion of lack of discriminatory power across alternative estimators in cost analysis, albeit with small sample size [15], also appears to be vindicated in the current analysis. However, the further suggestion that large sample size together with “…simple methods…” [69] and an identity link [12, 15] may be analytically sufficient seems not to be the case with our data, although the LMM and log-OLS models were relatively easy to fit and interpret. The estimated link function (λ=0.152(0.019, 0.285) supports neither an identity (λ=1) nor log link (λ=0), rather a fractional- but not a square-root (λ=0.5) function. Although the Park test and the EEE estimation of the variance function approximated 2, the GLM variants demonstrated poor goodness-of-fit (Figure 5) with the exception of the EEE (not with-standing the upper most decile of the linear predictor, Figure 6) where the link and variance functions were estimated, not fixed. The narrow confidence limits in the Copas test (Table 3) for the EEE estimator presumably reflects this re-estimation of link and variance functions in the cross-validation process. The residual behaviour of (i) the log scale estimators (Table 3) was generally satisfactory and empirical estimation suggested a quadratic mean-variance relationship which was consistent with the log-OLS estimator and (ii) the GLM models, was variable and with heavy tails. The use of gamma distribution in the presence of heavy tails has recently been cautioned [69]. Estimators on the raw scale (OLS and FMM) demonstrated consistent but sub-optimal residual behaviour; and despite modest BIC values, the log-skew estimators had an overall poor residual performance.

In the current study, the range of R² was modest (0.18-0.22; Table 2); consistent with that previously reported, 0.13 [78] to 0.21 [71, 83], and the observations of Diehr et al. that R² for cost utilisation data are usually ≤ 0.20 [12]. Increases in R² have been reported, not surprisingly, across groupings; for instance, ICU units (R² = 0.78; [7]). In the current study, R² across ICU units (n = 118) was 0.92 (LMM, determination set).

Model choice under the conditions of the current data set would appear to favour the LMM and log-OLS. The treatment effects model afforded extra explanation with respect to the covariate “Died-in-ICU” and had a good fit to the data. Mechanistic approaches to estimation are represented by the FMM which captures the bi-modal nature of the data (Figures 1 and 3), although the fit was not optimal; and the negative binomial GLM which represented the length of stay as “waiting times”. Neither the log-skew-normal nor log-skew-t estimator would appear to recommend themselves as the preferred analytic approach for the current data set.