Performance of quantitative measures of multimorbidity: a population-based retrospective analysis

This was a retrospective cohort study based on anonymized records of the entire population of Catalonia, a North-East region in Spain with approximately 7.5 million people. The regional Health Department of Catalonia provides universal healthcare to the Catalan population through a network of 64 general hospitals, 27 psychiatry hospitals, 375 primary care centres, 91 skilled nursing facilities for intermediate care, and 130 ambulatory mental health facilities. Data were retrieved from the CHSS, which stores clinical and resource utilization information from various registries, including hospitalization, primary care visits, emergency department visits, skilled nursing facilities, palliative care, and mental health services, information on pharmacy dispensation, out-patient visits to specialists, home hospitalization, medical transportation (urgent and non-urgent), ambulatory rehabilitation, respiratory therapies, and dialysis. The source registries were originally developed for healthcare planning and invoice control to the Catalan Health Service (i.e., the public healthcare insurance in Catalonia). The registries have an automated data validation system aimed to identify inconsistencies between variables (e.g., age-diagnostic correspondence, temporal sequences, among others) and undergo external audits periodically to ensure provider payment accuracy [26]. It is estimated that approximately 20-to-25% of the Catalan population has simultaneous public and private coverage and may, therefore, use resources not reported to the CHSS. However, owing to the pharmaceutical co-payment system, nearly all patients with chronic diseases eventually visit the primary care resources (which records all chronic diagnoses) to get drug prescriptions and benefit from the pharmaceutical co-payment. All data used for the analysis were recorded in the source registries during 2017 and were retrieved in July 2019. The study protocol was approved by the Independent Ethics Committee of the IDIAP Jordi Gol (Spain) (ref. 21/042-P).

The performance of the multimorbidity tools was assessed on 10 subpopulations: all adults (i.e., aged > 17 years), and 9 subpopulations of special interest because of the expected high health risk and/or frequency of resource utilization like people aged > 64 years, people aged > 64 years and institutionalized in a nursing home for long-term care, and people with highly prevalent chronic diseases, including the following diagnose codes of the international classification of diseases (9th and 10th versions, clinical modification; ICD-9-CM and ICD-10-CM; all converted to ICD-9-CM): ischemic heart disease, cirrhosis, dementia, diabetes mellitus, heart failure, chronic kidney disease, and chronic obstructive pulmonary disease. Children and people with chronic mental health diseases were not considered as subpopulations for analysis because current measures of multimorbidity are not suited to capture the complexity of these scenarios.

Our analysis aimed to compare tools that allow summarizing the comorbidity burden using a single numeric index. In addition to the GMA tool, we selected indices frequently reported in the literature that cover different approaches regarding two features: (1) the exhaustivity in the list of diagnoses considered and (2) the weight of each diagnosis according to severity (Fig. 1). The analysis included the following five tools: the Charlson index score [11], the count of chronic diseases according to three different proposals (i.e., the QOF [8], HCUP [13], and Karolinska institute [12]), and the multimorbidity index score of the GMA tool. Briefly, the Charlson index was designed as a tool for predicting life expectancy from a list of 17 comorbidities weighted according to their 1-year risk of death. The QOF was intended as a tool for allocating healthcare resources and incentivizing care of patients with chronic diseases, and its predictive capacity for health risk indicators such as mortality has been explored [27]. The QOF tool defines multimorbidity based on the presence of more than one diagnostic from a list of 17 important chronic conditions. The HCUP measures multimorbidity by counting the number of chronic diseases among all conditions codified in the chronic condition indicator (CCI) and grouped with the clinical classification software (CCS) [28]. The HCUP defines a chronic condition based on two criteria: (a) the given disease place limitations on self-care, independent living, and social interactions, and (b) result in the need for ongoing intervention with medical products, services, and special equipment [29]. The Karolinska proposal is a clinically-driven measure of multimorbidity based on the count of chronic diseases from a list of 918 ICD-10 codes (one- to -four-digit level), which are grouped into 60 chronic disease categories to retrieve a summary measure of multimorbidity [12]. In the Karolinska proposal, chronic diseases are selected based on the following criteria, applicable to older populations: prolonged duration and either (a) left residual disability or worsening quality of life or (b) required a long period of care, treatment, or rehabilitation. The GMA tool considers all chronic diagnoses (identified using the CCI of the HCUP) present at a given time and acute diagnoses reported during the study period. The GMA index score is computed by adding the weights of each diagnosis group (defined using the CCS of the HCUP system). For instance, unlike measures based solely on diagnosis counts, the GMA multimorbidity index algorithm gives a different complexity score to patients with hypothyroidism and eczema than those with asthma and diabetes, although accounting for two chronic conditions in both cases. Supplementary file 1 provides further details on the GMA algorithm.

We investigated the contribution of each multimorbidity measure to explaining eight outcomes associated with chronic patients: all-cause death, hospitalization, non-scheduled hospitalization, number of primary care visits (including general practitioner, nurse, and social worker, either at the primary care facility, home or via teleconsultation), visits to the emergency room (ER), medication use, admission to a skilled nursing facility for intermediate care, and high expenditure [30]. All outcomes were assessed in a 1-year time frame from January 1 to December 31, 2017.

The characteristics of the study population were described as absolute and relative frequencies and rates across all investigated outcomes. To identify individuals with high utilization rate of healthcare resources, we transformed continuous outcomes into to binary based the 95th percentile of the given variable among the overall population as cut-off. The resulting outcome definitions were as follows: admission to a skilled nursing facility for intermediate care (i.e., one or more admissions), admission to hospital (i.e., one or more admissions) and the emergency room (i.e., three or more admissions), visits to primary care services (i.e., more than 21 visits), medication use (i.e., dispensing of more than 13 drugs belonging to a different 5-digit group of the anatomic-therapeutic classification), expenditure (i.e., healthcare cost above € 4315.1). The cut-off value set for the overall study population was used for the sub-analyses on other sub-populations. Supplementary Figs. S2-S5 (Supplementary File 1) show the distribution of each study population for the continuous outcomes transformed based on the 95th percentile. To assess the performance multimorbidity measures, we built six logistic regression models for each of the investigated outcomes adjusted by age, gender, and socioeconomic status: a baseline model (i.e., age, gender, and socioeconomic status as independent variables, and all first-order interactions between them), and five models that added each multimorbidity measure to the baseline model. Multimorbidity measures were introduced into the models as continuous variables, using the indices, which were computed as described by the authors. The socioeconomic status was stratified into four categories of pharmaceutical co-payment: very low (recipient of social rescue aids), low (annual income < € 18,000), moderate (annual income € 18,000 to € 100,000), and high (annual income > € 100,000).

The performance of each model was assessed using four different statistics. For the primary analysis, we chose the area under the curve of the receiving operating characteristics (AUCROC) curve, which assesses the discrimination capacity of the model as the threshold varies and ranges from 0.5 (low discrimination capacity) to 1 (high discrimination capacity). Additionally, we conducted secondary analyses using the Akaike information criterion (AIC), pseudo-R squared (pR²), and the area under the precision-recall (AUC-PR). The AIC estimates the in-sample prediction error by taking into account the trade-off between the goodness of fit (overfitting) and the model simplicity (underfitting); the range of values that may take AIC depend on the study sample, with lower and higher values indicating better and poorer performance, respectively [31]. The pR² assesses the goodness-of-fit and the variability explained and ranges from 0 (poor fitness of the model) to 100 (very good fitness of the model). The AUC-PR curve shows the trade-off between precision (i.e., low false-positive rate) and recall (i.e., low false-negative rate) and returns a value between 0 and 1, less biased than the ROC curve towards overestimating in outcomes with low frequency [32]. All analyses were performed using the R statistical package (version 3.6.2) [33].

The analysis included 6,224,316 adult individuals (i.e., the entire adult population of Catalonia by the end of 2017) and the following subpopulations: older than 64 years (n = 1,472,623), older than 64 years institutionalized in a nursing home for long-term care (n = 67,456), ischemic heart disease (n = 244,311), cirrhosis (n = 45,126), dementia (n = 100,786), diabetes mellitus (n = 588,521), heart failure (n = 210,697), chronic kidney disease (n = 284,873), and chronic obstructive pulmonary disease (n = 357,989). Table 1 summarizes the main sociodemographic characteristics and rate of each study outcome for the adult population. The occurrence of most outcomes showed an increasing trend with higher age and lower socioeconomic status. Mortality was similar in the two genders; however, women tended to show higher rates of scheduled and non-scheduled hospitalization, primary care visits, ER admissions, medication use, and admissions to skilled nursing facilities. Men more frequently had expenditure below the threshold of € 4315.1. Tables S1-S9 (Supplementary file 1) summarize the characteristics of individuals included in the subpopulations.

The baseline model based solely on age, gender, and socioeconomic status showed the most deficient performance in explaining the investigated outcomes in all subpopulations according to AUC-ROC estimate (Fig. 2). The poorest performance of the baseline model was consistent across all other statistics (i.e., AIC, pR², and AUC-PR) (Table S10). Likewise, the addition of any multimorbidity measure to the baseline model improved the performance in explaining all investigated outcomes according to all statistics. In all models, admissions to the ER showed the lowest performance values. The GMA index (added to the baseline model) showed the highest performance in predicting all investigated outcomes. This trend was confirmed in the analyses using other statistics (Table S10). Of the other multimorbidity measures, the Charlson index score showed better performance in explaining mortality than the rest of the explored outcomes. The addition of the Karolinska and HCUP proposals for measuring multimorbidity showed better performance in explaining high use of medicines (i.e., more than 13) and primary care visits (i.e., more than 21).

Regarding the various subpopulations investigated, all models showed a trend to perform better on the general adult population and worse on people > 65 years institutionalized in nursing homes for long-term care. The GMA index showed lower variability across the investigated populations than the rest of the multimorbidity measures. This trend was confirmed when assessing performance with other statistics (Table S11-S19).