Statistical process control of mortality series in the Australian and New Zealand Intensive Care Society (ANZICS) adult patient database: implications of the data generating process

Statistical process control (SPC), deriving from Shewart’s work in 1920-30 and in the 1950’s with Deming’s refinements [1], has been more recently applied in health care and public health surveillance [2], generating considerable interest in the general [3‐5] and specialist medical literature [6‐10]; and has been subject to a detailed exposition from a “quality-in-medicine” perspective [11]. Important statistical principles underlying SPC or control-chart methodology are those of the monitored process being “in control” and subject to the independence of observations [12]. The presence and impact (possible increase in frequency of false alarms) of process autocorrelation in industrial/engineering series have long been documented [13‐16]. Somewhat surprisingly, little formal attention has been directed to this problem in the bio-medical literature [17, 18], one review suggesting that there was “…limited advice on how to manage [autocorrelation]…” [5].

We have previously drawn attention to the data-generating mechanisms of overall monthly mortality series, at the aggregate level, from a bi-national intensive-care (ICU) database, where persistent autocorrelation (to lag₂₄) was evident in a seasonal ARIMA (auto-regressive integrated moving average) model of the mortality series [19]. We now extend this study to further characterise the data generating process of mortality series at the individual ICU level and the impact of autocorrelation upon (i) mortality monitoring using EWMA (exponentially weighted moving average) control charts and (ii) time-series modelling of the data process using residual control charts.

As previously described [19, 20], the ANZICS (Australian and New Zealand Intensive Care Society) adult patient database [21] was utilised to define an appropriate patient set, 1995-(end)2009. Physiological variables collected in accordance with the requirements of the APACHE III algorithm [22, 23] were the worst in the first 24 hours after ICU (intensive care unit) admission, and all first ICU admissions to a particular hospital for the period 1995-2009 were selected. Records were used only when all three components of the Glasgow Coma Score (GCS) were provided; records for which all physiologic variables were missing were excluded, and for the remaining records, missing variables were replaced with the normal range and weighted accordingly. The mortality endpoint was at hospital discharge. Exclusions: unknown hospital outcome; patients with an ICU length of stay ≤ 4 hours, and patients aged < 16 years of age. Access to the data was granted by the ANZICS Database Management Committee in accordance with standing protocols; local hospital (The Queen Elizabeth Hospital) Ethics of Research Committee approval was waived.

The overall data set, 1995-2009, consisted of 491324 records from 137 ICU sites; mean (hospital) mortality was 14.07%. The random coefficient logistic regression model (Hosmer-Lemehsow statistic 62.97, ROC area under the curve 0.89) generated an overall predicted mortality probability of 0.1407 (SD 0.0202, range 0.00004-0.993). Over the 137 sites mean raw and expected (RA-) mortalities ranged from 0.012(0.113) and 0.013(0.045) to 0.296(0.457) and 0.278(0.247) respectively.

Of the raw mortality series from the 137 ICUs, 71 had continuous monthly data (excluding missing values or zero monthly mortality) for assessment up to or beyond lag₄₀. For 25 of these series (35%), there was a significant Q test (null hypothesis being that the series is white noise) and autocorrelation through to lag₄₀. Thirty six had continuous monthly data (excluding missing values) for ≥ 72 months; all series demonstrated marked seasonality and 30 demonstrated an obvious trend decline in mortality. Of the expected mortality series, 72 had appropriately assessable data to lag₄₀ and in 46 (64%) there was a significant Q test and autocorrelation through to lag₄₀. Similarly, in the same 36 series with continuous (raw) monthly data for ≥ 72 months, all expected mortality series demonstrated marked seasonality and 30 demonstrated an obvious trend decline in mortality.

Data from site “4” over 1995-2009 was used to generate an exemplar mortality time series. The mean raw mortality was 0.139(0.047) with skewness 0.216 and kurtosis 2.53; and the expected mortality was 0.138(0.028) with skewness 0.361 and kurtosis 3.47. The Shapiro-Wilk normality test was not rejected for either series (P =0.23 for both series). Kernel density estimates of raw and expected mortality are seen in Figure 1 (upper panel), with obvious difference in the degree of kurtosis between the two series. Time series plots, 1995-2009, for raw and expected (RA-)mortality are seen in the lower panel; a gradual time-decline in mortality for both series is evident. Additive seasonal decomposition of both series is seen in Figure 2, revealing marked seasonality and a trend decline in mortality. Autocorrelation plots are seen in Figure 3, showing correlation (positive and negative) decreasing variably with increase in lag in both series.

Figure 4 displays a plot of raw mortality series with control limits as 3SE of expected mortality (upper panel) and a scenario based mortality increment of 5% (5% false positive rate and desired ARL= 6 months) with control limits as 3SE of expected mortality. Frequent signalling is seen in both panel-plots. Figure 5 shows a plot of the raw mortality series with (fixed) EWMA 3 SE control limits derived from a projected 5% (upper panel) and 10% (lower panel) increment in expected mortality, assuming: an in-control ARL of 370, mean (expected) mortality 0.1381(0.0276) and target mean (expected) mortality of 0.145 (5% increment) and 0.152 (10% increment), for an EWMA λ = 0.02 and 0.05, respectively (calculations preformed in Stagraphics®). For both 5% and 10% projected increments of expected mortality, the raw mortality series signalled frequently, mainly in the early periods. Figure 6 shows the same scenarios with a time-varying variance EWMA control chart; again, there was frequent signalling of the raw mortality series.

The autocorrelation evident in the raw and expected mortality series suggested a formal time series approach to SPC:

Both the GARCH and ARMA models were considered parsimonious and the de-trended signals for each model were within 3SE limits of respective model predictions (Figure 7). The residuals from both the formal GARCH and ARMA models (mean: 0(0.0423) and 0(0.0257) respectively) satisfied multiple criteria of Gaussian white-noise and were within residual-EWMA control limits (default values of “optimal exponential coefficient” in Stata™ statistical software; 3SE control limits; λ = 0.0001 for both series; Figure 8). To address any potential inertial problems consequent upon the small λ, control limits were also established for projected 1 (λ=0.16), 2 (λ=0.42) and 3 (λ=0.71) SD increments of the mean of the GARCH residuals; the latter were within these control limits (Figure 9).

The current analysis of monthly mortality series confirms the existence of autocorrelation and seasonality in both the raw and expected series at the individual ICU level, avoiding any potential confounding at the aggregate level due to Simpson’s paradox. We thus concur with the findings of Alwan [13, 50] and Bisgaard and Kulahci [51], who documented the pervasiveness of autocorrelation in a variety of series, industrial and non-industrial. We also established that out-of-control signalling of the raw mortality series with respect to both 3 standard error risk-adjusted and RA-EWMA control limits was not evident with analysis of the residuals from the GARCH time series model. Thus the identification of (G)ARCH processes is an important issue for SPC [35].

As our focus was directed to an understanding of the underlying data-generating process [45] and the performance of the RA-EWMA control limits under conditions of autocorrelation, we deemed it appropriate to also subject the expected series, from which the control limits for the raw mortality series were established, to formal time-series estimation. Not surprisingly, as the underlying mortality estimates from a random coefficient model are obligatorily “smoothed” (see also Figure 1), no ARCH effects, representing “volatility”, were demonstrated and a relatively simple seasonal autoregressive model was established (Table 1). As the EWMA is based upon an ARIMA(0,1,1), that is an integrated moving average process [45, 52], it has been applied to autocorrelated data [15], although the majority of studies have used relatively simple non-seasonal autoregressive models (AR(1) or AR(2)) with fixed λ (usually 0.2, which is the default for the SPC model of Statgraphics software). Residual-EWMA charts, in the context of time series modelling, would appear to be more robust than EWMA applied to the original (autocorrelated) data [53, 54]. Reynolds and Lu have recommended that under autocorrelation “…traditional control chart methodology should not be applied without modification…” [55] and Human et al. have recently sounded a cautionary note about the robustness of the conventional EWMA [56].

For the classical SPC model, a process is in control if the mean and standard deviation estimate remain within prescribed control limits [57], usually three-sigma; that is, for a normally distributed series, 99.7% of observations should lie within the limits [58] and there a probability of 0.0027 that any point exceeds the control limit [32, 59]. However this definition does not necessarily entail the formal time-series notion of stationarity (strict or weak), where the requirement for stationarity is that the first two moments (mean and variance [45, 50]) and the autocorrelation function are time-invariant, albeit a stationary processes may be auto-correlated [60]. In the industrial/engineering sphere, practitioner response to process autocorrelation [61] was to embrace a time-series paradigm and apply SPC methods to the residuals of a formal time-series model [45, 62], albeit there were different tactical approaches [15, 63]; or to develop modified control limit schemes [64‐66]. It is instructive to note that the non-model based EWMAST chart (EWMA chart for stationary processes [66]), recommended by Winkel and Zhang [11], pre-supposes a stationary (not “in-control”) process. In a systematic review of the application of statistical process control in healthcare, Thor et al. [5] adduced only one literature reference [67] and a calendar year 2003 monograph which discussed autocorrelation in medical series. As argued by Alwan and Roberts [45], systematic non-random patterns in series make separation of the classic common and special causes difficult, as departures from control, nominally traceable to special causes, are confounded by autocorrelation and, in the current series, seasonality. Two further concerns were raised by the authors; first, the undue emphasis placed upon normality and the (erroneous) assumption that “approximate normality” implies a state of statistical control; and second, in the presence of a well-fitting time series model with residuals consistent with white-noise (“randomness”), it is “…futile to search for departures from statistical control and their corresponding special causes…”. The latter caution resonates with the current finding of frequent signalling of the raw mortality series compared with in-control residuals from an apposite time series model; with respect to the error process, such signalling represents false positivity [13].

Cook and co-workers, “…explicitly compare[d] EWMA(observed) and EWMA(predicted) …[with] thresholds around the EMWA(predicted)…”, employing the EWMA (λ = 0.005-0.020) to “…effectively attenuate noise in the data and smooth an erratic but unbiased risk model” [25], although no criteria of “erratic” were provided. Smoothed control limits for the expected series were also utilised in a review paper by Cook et al. ([68]) and Pilcher et al. ([69], λ = 0.005), albeit the data structure differed; sequential plotting of each patient admission versus monthly mortality rates in the current paper. Our focus and methodology were different, in that we were concerned to both understand and formally model the “noise in the data”. This being said, in the current series, the smoothed EWMA (λ = 0.51) raw series (Figure 4) was demonstrated to signal using 3 standard error expected mortality control limits.

The sophistication of time-series modelling in standard statistical software packages makes the formal analyses of the current study feasible; in particular, automated routines for application of time series models [70]. However, for the application of appropriate SPC to mortality series from multiple ICUs in a data-base, there are unresolved statistical issues [71, 72]. From the perspectives of this study, a multivariate approach may be established using more conventional estimators (multivariate GARCH [73] and vector autoregression models [74]) or by newly described hierarchical/functional time series [75, 76].

The underlying data generating process of monthly mortality series at the ICU level displayed autocorrelation and seasonality, with volatility evident in the raw mortality series. Failure to accommodate these characteristics by SPC measures resulted in false-positive signalling. A time series approach to SPC, using residual control charts, would appear to resolve such issues.