External validation of the detection of indicators and vulnerabilities for emergency room trips (DIVERT) scale: a retrospective cohort study

We conducted a population-based retrospective cohort study of home care clients in the provinces of Ontario and Alberta, and in the regions surrounding Winnipeg, Manitoba and Whitehorse, Yukon.

We linked multiple de-identified administrative health databases to construct our cohort. Home care clinical assessment data were extracted from the Home Care Reporting System. This national database contains demographic, clinical, functional and service utilization information on publicly funded home care clients in Canada. ED utilization data were extracted from the National Ambulatory Care Reporting System, which houses comprehensive population-level data on hospital and community-based ambulatory care visits in Canada. The databases used in this study are routinely checked for validity and have been used extensively in health services research [18‐22]. The Hamilton Integrated Research Ethics Board granted ethics approval for this study. No additional administrative permission or licensure was required for this project.

Home care clients in Canada are periodically assessed using the Resident Assessment Instrument for Home Care (RAH-HC). We created a retrospective cohort of all RAI-HC assessments completed between April 1, 2011 and September 30, 2014. Data were accessed on clients in the provinces of Ontario and Alberta, as well as in the Winnipeg Regional Health Authority in Manitoba and the Whitehorse census subdivision of the Yukon Territory. The cohorts in Manitoba and Yukon were restricted to areas surrounding the cities of Winnipeg and Whitehouse due to limitations in the coverage of the National Ambulatory Care Reporting System (NACRS). The RAI-HC assessments in the cohort were linked to ED records in the NACRS dataset to identify all ED visits within 6 months of the assessment date.

The DIVERT scale is a prognostic model developed using a classification and regression tree and recursive partitioning methods. Assessment items of the RAI-HC were considered as candidate predictors for model derivation. The RAI-HC is a comprehensive clinical assessment of over 250 items that have demonstrated validity and reliability in documenting the domains of function, health, social support and health service use [22, 23]. The RAI-HC is currently used during standardized home care assessments in most Canadian provinces and territories, half of the United States, and in many countries around the world including Estonia, Finland, Hong Kong, Iceland, Ireland, Italy, Japan, the Netherlands, New Zealand, Singapore, Spain and Switzerland. At this time, the DIVERT scale is a standard measure within the RAI-HC assessment. Figure 1 displays the DIVERT scale and the predictors used in discriminating patient risk for ED visitation.

The primary outcome of this study was an ED visit within 6 months after a RAI-HC assessment date. Secondary outcomes include: (a) two or more ED visits within 6 months of a RAI-HC assessment and (b) any ED visits that resulted in hospital admission. Data were censored at the date of death. All outcomes were selected to parallel the figures in the original derivation study [14]. A six-month follow-up period was chosen to reflect the contemporary home care assessment intervals.

The DIVERT scale was derived and validated using home care assessment data linked to ED records from Ontario and Winnipeg between 2006 and 2010 [14]. Our examination of data from Ontario, Alberta, Winnipeg, and Whitehorse between 2011 and 2014 enables us to test the scale’s external validity within the two original regions, during a different time. We measured the discriminative ability of the DIVERT scale using the area under the receiver operating curve (AUC), or c-statistic. The AUC is the area under the curve created by plotting sensitivity against 1-specificity at various thresholds and is a common measure of the discriminative ability. Within each region, we calculated the AUC of the DIVERT scale for each outcome. All analysis was performed using SAS/STAT 13.1.

Within the four regions of our study, we identified 1,001,133 RAI-HC assessed cases occurring between April 1, 2011 and September 30, 2014. The vast majority of cases were in Ontario (88%), followed by Alberta (8%), Winnipeg (4%) and Whitehorse (< 1%). A descriptive profile of the home care clients within each region can be found in Table 1. Overall, the home care clients in our sample were predominantly female, with a mean age of approximately 79.

Home care clients in Ontario had the highest level of physical and cognitive impairment, while clients in Whitehorse had the lowest. Home care clients in Ontario were also more likely to have a live-in caregiver and were more likely to have informal caregivers who express distress. However, clients in Whitehorse were the most likely to rate their health as poor and had the highest proportion of clients scoring three or higher on the Depression Rating Scale.

The proportion of outcomes and the discriminative ability of the DIVERT scale across all regions are displayed in Table 2. For the primary outcome of ED visitation, the AUCs of the DIVERT Scale within Ontario, Alberta, and Winnipeg are very similar to one another (0.617–0.624) and the original validation work (0.62). The AUC for the Whitehorse cohort was slightly higher for this outcome (0.647).

For the secondary outcome of two or more ED visits in the 6 months following a RAI-HC assessment, the AUCs within all four regions were similar to one another (0.628–0.634) and the original validation work (0.63). For the outcome examining hospital admission, the AUCs within Ontario, Alberta and Winnipeg were again comparable to one another (0.617–0.624), while the AUC for Whitehorse residents was slightly higher (0.664). Supplemental measures of predictive accuracy (e.g., sensitivity, specificity, etc.) were similar to the findings reported in the original validation paper.

Our study provides external validation of the DIVERT scale and demonstrates that the scales discriminative ability is generalizable to external populations in new geographic regions (Winnipeg, Manitoba and Whitehorse, Yukon Territories). Furthermore, we demonstrated that the performance of the DIVERT scale is robust to temporal changes. The DIVERT scale was able to adequately predict ED visitation, multiple ED visits and hospitalization in a sample of Canadian home care clients.

To our knowledge, the original derivation and validation study for the DIVERT scale was the first to predict ED use among older home care clients using population-level data. The DIVERT scale performed similarly to other predictive models examining ED and hospital use among older adults [24‐27]. Only one other study to date has attempted to predict ED use among home care clients. Jones et al. used a series of machine learning algorithms to predict ED use for an injurious fall within 6 months of a RAI-HC assessment [27]. A key difference between the DIVERT scale and prior work is that the DIVERT scale aims to inform program planning and preventative interventions prior to ED visitation by stratifying risk among subgroups of home care clients.

Health systems (both public and private) are often limited by budgetary constraints, resulting in some clients not receiving adequate care; this is especially true of home care services, where many Canadian clients receive only partial care needs [28]. Limited home care services in Canada underscore the utility of systematically identifying clients who are at a greater risk of hospital use, to target enhanced risk assessment or preventative efforts.

The DIVERT scale provides real-time risk estimation and information that can be used to supplement decision making surrounding resource allocation and preventative interventions. For example, patients who score high-risk on the DIVERT scale (e.g., ≥ five or greater) with unstable cardiorespiratory symptoms are provided with a multifaceted and tailored service plan to meet their distinct needs [17, 29]. The DIVERT score can be further utilized among health care providers to ensure continuity of care and open communication regarding the risk of adverse events and health service use upon transitioning back to home care. As of recent, patients enrolled in the DIVERT trial are scheduled to receive nurse-led interventions focused on chronic disease management and patient education to facilitate patient-centred care.

Beyond its use for case finding, the DIVERT scale can be used to stratify or adjust organizational, regional and national level ED utilization metrics in home care settings. Our study demonstrated the external validity of the DIVERT scale and the utility that this prognostic tool is likely to have for home care clients across Canada. The RAI-HC is currently implemented as standard practice in many nations around the world, suggesting that knowledge translation efforts and clinical integration of the DIVERT scale are feasible in countries outside of Canada. Heterogeneity in the use of the DIVERT scale is likely to differ across health care systems, given the influence of funding models and health care standards.

The use of the DIVERT scale is limited to a predominantly frail population of community-dwelling older adults who receive home care services. The current study was limited to the person-level variables available in the RAI-HC assessment and could not capture all relevant determinants, particularly primary care utilization. Next, the predictive accuracy of the DIVERT scale was limited by the features of a complex adaptive system in home care delivery. Prior work has reported the difficulties in decision-making and predicting health outcomes in a complex adaptive system as they are more likely to feature non-linear interactions and a co-evolution of decisions and events [30‐32]. Regardless of data complexity, the DIVERT scale achieved an AUC greater than 0.6 in all subgroup analyses in the original and external validation studies. This level of performance is similar to prior prognostic models aiming to predict ED and hospital use [24‐26, 33]. However, we caution against precise comparisons of performance given that they cannot be compared within the study sample or against the same outcome measures. Further work is needed to understand what types and intensity of interventions are feasible and effective in the community. Future research should aim to replicate these findings in countries outside of Canada.