Background
Short-term forecasts were widely used during the COVID-19 pandemic to support public health policy through planning and resource allocation. During 2020 and early 2021, it was an ongoing concern that demand for hospital care would exceed capacity.
Defining hospital catchment areas serves a number of purposes, primarily for healthcare administration: first, to estimate the size of each catchment population—the number of people using, or who may use, the health services—and the catchment population’s demographics; this facilitates planning, such as allocating budgets and determining staffing and other resource needs; second, to identify regions with under- or overprovision of hospital services; finally, to calculate admissions rates per capita or estimate vaccination coverage, using the catchment population size as the population denominator.
This last application is also highly relevant for modelling of hospital admissions, including short-term forecasting. In England, local-level COVID-19 cases and hospital admissions are reported on different scales: cases are reported by the local authority (a small subnational administrative unit), and admissions by UK National Health Service (NHS) Hospital Trusts (comprising a small number of hospitals providing care to a small geographical region or for a specialised function). Previous evaluations of local-level COVID-19 hospital admissions forecasts have shown that using cases as a leading predictor can improve forecast accuracy [
1]; however, to use “local” cases as a predictor of hospital admissions requires the forecaster to define “local”, that is, to define each hospital’s catchment area.
Hospital catchment areas in England are not well-defined, since prospective patients have a right to choose where to seek health care [
2]. According to the Office for Health Improvement and Disparities, NHS Trusts do not have geographically defined boundaries for service access, nor a complete and up-to-date list of registered patients [
3]. Any proposed catchment area definition should aim to capture a significant proportion of the hospital’s activity, reflect physical barriers to access (travel distance or time), differ by speciality, and increase with hospital size [
4].
Hospital catchment areas are estimated using geographically linked admissions data, or simple heuristics. If data on historical admissions is available, then this data could be used with a model to define the catchment areas. Two simple models are frequently cited: first-past-the-post, where geographical areas are assigned to the single hospital to which the majority of patients sought care, and proportional flow, where populations are split between hospitals according to the proportion of admissions to each hospital (Jones et al. 2011). The UK Office for Health Improvement and Disparities uses the proportional flow model [
3]. Both the first-past-the-post and proportional flow models assume no future change in hospital admissions patterns. Gravity models [
5,
6], and other more complex regression models [
7], are also used but require yet more data on population demand, hospital capacity, and travel distances or times between patients’ homes and hospitals. Using more complex models may also reduce understanding, interpretability, and uptake for stakeholders unfamiliar with modelling methodology.
Geographically linked hospital admissions data may not always be readily available [
8]. In these scenarios, heuristics can be used instead. The simplest assigns geographical areas to their nearest hospital using an adapted Voronoi decomposition (also known as a Thiessen decomposition) which conforms to administrative boundaries; this produces non-overlapping catchment areas where prospective patients are assumed not to have any choice in service provider. However, numerous studies in a wide range of settings have shown that patients often choose to travel further to seek alternative facilities [
9‐
11] or to seek highly specialised care. Alternatively, catchment areas can be defined as any geographical region within a fixed distance [
12] or, where appropriate and available, travel time [
8,
13]. Other algorithmic approaches include K-means clustering [
4] and label propagation [
14].
Estimated catchment areas and populations have been used to predict changes in emergency admissions in response to changes to hospital capacity in London [
5] and to estimate the cumulative rate of confirmed malaria cases for six healthcare facilities in Uganda [
12], amongst others [
13]. Only a few studies considered the sensitivity of their results to the catchment area definition used [
12], despite acknowledging that these areas were not well-defined. To our knowledge, no forecasting studies have assessed the sensitivity of quantitative forecasting performance to the choice of hospital catchment area definitions.
In this study, we evaluated the impact of using different hospital catchment area definitions on the accuracy of local-level hospital admissions forecasts, using COVID-19 during September 2020–April 2021 as a case study. We used a scaled convolution of local cases (as defined by each of the catchment area definitions) and a delay distribution as the forecasting model. We showed previously this model makes accurate forecasts of local-level COVID-19 hospital admissions, under the condition that we can make consistently good case forecasts [
1]. We measured and compared the performance of the resulting forecasts and summarised average performance by forecast horizon, forecast date, and location. We considered both a retrospective scenario, where future cases are assumed to be known in order to provide a best case for the forecast performance, and a real-time scenario, where future cases are not known and a case forecast is used instead. We also explored whether there was any change in our main conclusions when using catchment areas defined at different subnational spatial scales.
Methods
Data
Trust-level COVID-19 hospital admissions
A confirmed COVID-19 hospital patient is any patient admitted who has recently (in the last 14 days) tested positive for COVID-19 following a polymerase chain reaction (PCR) test, including both new admissions with a known test result and inpatient tests. Data on daily COVID-19 hospital activity, including COVID-19 hospital admissions and COVID-19 and non-COVID-19 bed occupancy, are published weekly by NHS England and were accessed via the
covid19.nhs.data R package [
15]. Data are aggregated and reported by NHS Trust, organisational units of NHS England comprising a small number of hospitals (typically between one and three) and providing care to a small geographical region or for a specialised function. There were 129 NHS Trusts in England until 24 January 2021; then 128 until 14 March 2021; then 127 until the end of April 2021.
Local authority-level COVID-19 cases
A confirmed COVID-19 case in England is defined as an individual with at least one confirmed positive test from a PCR, rapid lateral flow tests or loop-mediated isothermal amplification (LAMP) test. Positive rapid lateral flow test results can be confirmed with PCR tests taken within 72 h; if the PCR test results are negative, these are not reported as cases. Aggregated data are published daily on the UK Government dashboard [
16] and reported totals include both pillar 1 (tests in healthcare settings and for health and care workers) and pillar 2 (community) tests. We use data published by specimen date, the date on which the test is taken. Data are reported by lower- or upper-tier local authorities (LTLA and UTLA, respectively). In summary, local authorities are small geographic regions, where each UTLA comprises one or more LTLA; as of April 2021, there were 174 UTLAs and 331 LTLAs in England [
17,
18]. Local authorities also function as a level of local government: during the COVID-19 pandemic, some restrictions were implemented at the local authority level, although all decisions were made at the national level. Further details are given in Additional file
1: Sect. 3.1.
Hospital catchment area definitions
Since COVID-19 cases and hospital admissions are reported on different scales (local authority and NHS Trust, respectively), we need to define each Trust’s catchment area to estimate COVID-19 cases “local” to each Trust.
We considered six definitions of hospital catchment areas for Acute NHS Trusts in England. The catchment area definitions are based on heuristics about where people may seek hospital care, or derived from data on historical hospital admissions. We defined a Trust’s catchment area with respect to lower- or upper-tier local authority boundaries, and for each local authority-Trust pair assigned a weight, p, that represented the expected proportion of people/admissions from that local authority that would seek hospital care at the given Trust. Catchment area definitions are defined to ensure that each local authority is matched to at least one Trust, and vice versa; consequently, the catchment area boundaries are overlapping, since each local authority can be mapped to more than one Trust.
The six catchment area definitions are as follows:
-
Marginal distribution (baseline): Patients from all local authorities go to Trusts according to the marginal distribution of hospital admissions June 2020–May 2021. For example, if Trust X admitted 10% of all COVID-19 patients in this time period, then the marginal distribution assigns a weight of 10% between all local authorities and Trust X.
-
Nearest hospital: Patients from each local authority go to any Trust within the local authority with equal probability, or to their nearest Trust with probability 1 if there are no Trusts in the local authority. Distance between local authorities and Trusts is measured as the Euclidean distance between the population-weighted centre of the local authority (based on shapefiles as of December 2021 [
17,
18]) and the main site of the Trust [
18,
19].
-
Nearby hospitals: Patients from each local authority go to any Trust within a 40-km radius from the population-weighted centre of the local authority. Patients go to any Trust within the 40-km radius with equal probability. The value of 40 km was chosen to be larger than the maximum minimum local authority-Trust distance for any local authority; a lower threshold would result in a catchment area definition for which some local authorities were not mapped to any Trusts.
-
Emergency hospital admissions, 2019: Defined by the distribution of all emergency hospital admissions from UTLA to Trust in England during 2019 (January through December) [
3].
-
Elective hospital admissions, 2019: Defined by the distribution of all elective (non-emergency and pre-planned) hospital admissions from UTLA to Trust in England during 2019 (January through December) [
3].
-
COVID-19 hospital admissions, June 2020 through May 2021: Defined by the distribution of COVID-19 hospital admissions from UTLA to Trust in England during June 2020–May 2021. Confirmed COVID-19 cases were linked to hospital admissions by case/patient ID [
15]. For each case, we keep at most one admission, namely the first admission only from all admissions where (i) the admission date was earlier than the test specimen date plus 28 days and (ii) the discharge date was after the test specimen date.
Comparison of catchment area definitions
We compared the six hospital catchment area definitions to understand where and how the definitions differed from one another: any difference in forecast performance under the different catchment area definitions will be contingent on differences in the definitions themselves and on differences between trends in COVID-19 cases between local authorities within each catchment area. We compared the catchment area definitions using a number of summary statistics, as well as quantifying how much the two catchment area definitions overlap. We also quantified the similarity of the COVID-19 case time series between local authorities within Trusts’ catchment areas.
Descriptive statistics
We summarised and compared the following Trust-level properties of the hospital catchment area definitions. First, the distribution of catchment area weights. Second, the number of local authorities under each catchment area definition for which the assigned weight is more than x%, that is, for which at least x% of the local authority population are assigned to that Trust; we consider x = 0%, 1%, and 10%. These thresholds were chosen to capture either all local authorities (0%) or local authorities with a non-negligible or significant weight (1% and 10%, respectively). Third, the median distance from each Trust to any local authority for which the assigned weight is more than x%; again, we consider x = 0%, 1%, and 10%. Finally, we calculate the proportion of a catchment area’s total weight that is from either the nearest or nearby (< 40 km), local authority or authorities.
Quantifying the similarity of catchment area definitions
To summarise and quantify the similarity between pairs of catchment area definitions for a given Trust, we defined an overlap-similarity metric. The
asymmetric overlap-similarity between catchment area definitions
\(X\) and
\(Y\) relative to definition
\(X\) is defined as the proportion of the definition
\(X\) that is contained in the overlap with
\(Y\). The
overlap-similarity between
\(X\) and
\(Y\) is then simply the minimum of the asymmetric overlap-similarity relative to each of
\(X\) and
\(Y\). See Additional file
1: Sect. 3.2.2, for details of these definitions.
By construction, the overlap-similarity is equal to zero if, and only if, the two catchment area definitions do not include any of the same local authorities. The asymmetric overlap-similarity between \(X\) and \(Y\), relative to \(X\), is equal to 1 if, and only if, the weights assigned by \(X\) are less than or equal to the weights assigned by \(Y\) for all local authorities. The overlap-similarity metric is equal to 1 if, and only if, the two definitions assign exactly equal weights to all local authorities. A higher overlap-similarity score indicates that \(X\) and \(Y\) are more similar, that is, the weights assigned to each local authority are similar for definitions \(X\) and \(Y\).
Quantifying the similarity of COVID-19 case time series
We expect that any difference between forecasts made using different catchment area definitions will require some heterogeneity in the trends of reported COVID-19 cases in the local authorities in the catchment area—if nationwide trends are aligned, the definition of catchment areas should not matter. To this end, we used the Spearman correlation coefficient to quantify the similarity between COVID-19 case time series between local authorities. First, for each forecast date and pair of local authorities, we calculated the Spearman correlation between the time series of COVID-19 cases in each local authority from 21 days before until 14 days after the forecast date. Then for each forecast date, we calculated two values: first for each local authority the median correlation coefficient with all other local authorities, and second for each Trust the median correlation coefficient between all local authorities for which any catchment area definition assigns a weight of more than 10%. The threshold value of 10% is arbitrarily chosen, but such that only the main local authorities in each catchment area were included.
Admissions forecasting model
We made forecasts of hospital admissions using a convolution of estimated Trust-level COVID-19 cases with the delay from report to admission [
1,
20], henceforth referred to as the case-convolution model, for brevity. Full details of this case-convolution model are in Additional file
1: Sect. 1.2. We showed previously that this model makes accurate forecasts of local-level COVID-19 hospital admissions: it clearly outperformed simple trend-led time series models, especially at longer forecast horizons, albeit under the condition that we were able to make good case forecasts [
1].
To forecast admissions, the case-convolution model takes COVID-19 cases after the forecast date as input. Here, we considered two scenarios: retrospective and real-time. In the retrospective scenario, we use future observed cases with no uncertainty, and so this represents a best-case scenario in which any difference in forecast performance is a direct result of using the different catchment area definitions. Under this scenario, we evaluated forecast performance for both lower- and upper-tier local authority definitions of catchment areas.
In the real-time scenario, we used forecasts of cases that were made via estimates and forecasts of the real-time reproduction number, Rt, accounting for uncertainty in the delay distributions and day-of-the-week effect, produced and published daily [
20,
21]. A summary of this is given in Additional file
1: Sect. 1.1 [
1,
20‐
23]. Under this scenario, we evaluate real-time forecast performance, as well as assess how forecast performance changes when uncertainty in future cases is included. As case forecasts were only made at the UTLA level, we evaluated forecast performance in the real-time scenario for the upper-tier local authority definition of catchment areas only.
Forecast evaluation
We assessed different aspects of forecast performance using standard metrics, described below. We summarised forecast performance against these metrics by forecast horizon, by forecast date (the date on which each forecast is made, 15 total), and by location (acute NHS Trust, 138 total).
Calibration
Calibration measures models’ ability to correctly quantify predictive uncertainty. We assessed calibration by calculating the empirical coverage: for a given forecast horizon and prediction interval width, the empirical coverage of a model is the proportion of forecast targets (here, across all dates and locations) for which the prediction interval contains the true value. We calculated the empirical coverage for the 50% and 90% prediction intervals. A perfectly calibrated model has empirical coverage equal to the width of the nominal prediction interval.
Probabilistic forecast error
We measured probabilistic forecast accuracy with the weighted interval score (WIS), a proper scoring rule, a rule for which a forecaster is incentivised to give their honest forecast to obtain the best score [
24]. The WIS comprises a weighted sum of interval scores for quantile forecasts of increasing widths, measuring both the sharpness (the width of the forecast interval) and accuracy (whether the interval contains the true value) of each quantile forecast. By combining these interval scores, the probabilistic forecast accuracy of the full forecast distribution is summarised in a single value. The relative weighted interval score (rWIS) is an adjusted WIS value that allows us to compare the performance of different definitions over forecast dates or between Trusts, both of which vary in the number of admissions.
Discussion
This study systematically evaluated the impact of the choice of hospital catchment area definitions on the accuracy of local-level forecasts of COVID-19 hospital admissions during September 2020–April 2021. While previous studies have set out to define hospital catchment areas or catchment populations [
4‐
8,
14], or to quantitatively evaluate local-level hospital admissions forecasts [
1], none have considered the sensitivity of admissions forecasts to the choice of catchment area definitions.
At longer forecast horizons, the small improvement in forecast accuracy acquired from using any sensible catchment area definition is overshadowed by the decline in performance when uncertainty and/or inaccuracy of future cases is included. As noted previously [
1], the benefit of including leading predictors, such as cases, is limited by the quality of forecasts of these predictors.
Our study focussed on only a single forecasting model, namely a scaled convolution of local cases and a delay distribution. We showed previously this model makes accurate forecasts of local-level COVID-19 hospital admissions, under the condition that we can make consistently good case forecasts [
1]. As our aim here was to evaluate the relative performance of the catchment area definitions, this condition is not a limitation. However, we note that other forecasting models, including ensemble forecasts, may produce better forecasts in some scenarios, especially if these forecasts are made in real time.
Overall, we found that the choice of hospital catchment area definition affected the accuracy of hospital admission forecasts. The marginal distribution definition consistently resulted in the least accurate forecasts: it was clearly important to apply some definition of catchment areas. Moreover, forecasts made using a context-specific catchment area definition based on COVID-19 admissions data were the most accurate (or one of the most accurate) amongst all non-trivial catchment area definitions: the COVID-19 admissions data definition outperformed the other definitions when evaluated by forecast horizon and was highly ranked more often when evaluated by forecast date or location. In addition, the relative performance of this definition was less variable across forecast dates and locations. The nearby heuristic also performed well, but was markedly less consistent than the COVID-19 definition when forecasts were evaluated by location. In total, these results demonstrate clear value in understanding the factors that drive local admissions patterns and, subsequently, constructing catchment area definitions that are relevant to any given pathogen or health emergency. We note that the COVID-19 definition was based on admissions during an acute phase of the outbreak (June 2020 through May 2021): the benefits of using a context-specific catchment area definition could change outside of this period, or if we consider a scenario in which context-specific catchment areas are learned in real time.
The difference in performance between the COVID-19 admissions definition and simpler heuristics was small; however, we posit that there are settings for which the choice of catchment area definition may be more important. Here, we showed that COVID-19 cases in local authorities within a catchment area for NHS Trusts in England were often strongly correlated with each other. Where the trend in cases is broadly similar within a catchment area, the exact catchment area weights will have little-to-no impact on forecasting performance. On the other hand, we would expect to see more of an effect where local case trends are more heterogeneous. We saw some evidence of this during December 2020, where local case trends were more heterogeneous and the marginal distribution baseline performed especially poorly.
This opens a number of options for future work. First, we can test our assertion that the choice of catchment area definition is more important in heterogeneous settings. We propose the best approach would be to develop a small-scale simulation study, where the correlation between cases within a catchment area is systematically varied. This synthetic study would be the best-case scenario for seeing a difference in forecast performance by catchment area definition. We should also determine for which diseases, if any, it is feasible that this heterogeneity could arise. At the same time, we can replicate the study framework for COVID-19 in other countries, or for other diseases entirely. Determining when and where context-specific definitions are less important is valuable in real-time outbreak response settings: instead of dedicating significant time to constructing disease- and country-specific catchment area definitions, it may be sufficient to use simple heuristics to determine hospital catchment areas. When data and/or people-hours are limited, valuable resources can then be directed at more urgent analyses. Additionally, we could evaluate the marginal improvements of context-specific definitions when they are learned in real time, rather than already known retrospectively.
We also note a number of other limitations to this study. First, we assumed that catchment areas were fixed over time. Although preliminary analyses of COVID-19 admissions in England did not show any substantial change in catchment areas over time, this assumption will not always hold and a time-varying definition may be more appropriate in other contexts. For example, if hospitals regularly reach 100% capacity (from lack of availability of beds, equipment, or staff), then patients may be transferred to other facilities, or if new hospitals are opened or old hospitals closed. Second, we only defined catchment areas up to lower-tier local authorities. In some scenarios, this could mask fine-scale variation in reported cases; however, as we saw no improvement when using lower- vs. upper-tier local authority definitions here, we think it is unlikely yet smaller subnational regions would make a difference here. Additionally, using a definition based on smaller subnational regions would require substantially higher computational resources to make the case forecasts, which was already limited by this requirement to upper-tier local authorities only.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.