The ability of comorbidity indices to predict mortality in an orthopedic setting: a systematic review

This systematic review included studies that compared the performance of different comorbidity indices in predicting mortality in an orthopedic setting.

Studies were included if they:

Studies were excluded if they only included comorbidity indices that were validated only for specific diseases (e.g., the Cardiac-specific Comorbidity Index [11]). We did not have exclusion criteria in form of a minimum age limit nor for whether the study participants was in- or outpatients.

Primary outcome measures were defined as in-hospital, 30-day, 90-day, and 1-year mortality. Secondary outcome measures were not prespecified.

The language of included studies was restricted to articles written in English. No publication year, publication status, or study design limitations were imposed.

We searched the following databases: Ovid Embase Classic + Embase (1947–present), Ovid MEDLINE (R) all (1946–present), and the Cochrane Library. The last search was carried out on 2 May 2019. The reference list of included studies was screened by PHG and MJ for additional eligible studies. Grey literature was also searched through OpenGrey [12], the International Clinical Trial Registry Platform [13], and Orthopaedic Proceedings [14].

The search strategy was developed by the author team with assistance from librarians at the University of Southern Denmark and was primarily developed for Embase; thereafter, it was adapted for the other databases. The searches were constructed around two primary focal points: a comorbidity index and orthopedics. No language or publication year restrictions were used in the search. The following search terms were used to search Embase:

comorbidity ind*; comorbidity scor*; comorbidity scale*; exp Charlson Comorbidity Index; Charlson* adj2 (ind* or scale* or score*); Elixhauser*; Chronic Disease Score*; Rxrisk; Rx-risk; Rxrisk-v; Rx-risk-v; Medication-Based Disease Burden Index; exp orthopedics; orthop?edic*; exp orthopedic surgery; exp arthroplasty; arthroplast*; hemi-arthroplast*; hemiarthroplast*; exp osteoarthritis; osteoarthritis; exp arthroscopy; arthroscop*; exp fracture; fracture*; exp traumatology; traumatolog*; exp hip surgery; exp knee surgery; exp shoulder surgery; hip* or knee* or shoulder* adj3 (surg*).

A full description of the search strategies for Embase, MEDLINE, and the Cochrane Library can be found in the Additional file 2.

Eligibility assessment of the studies was performed independently in a blinded standardized manner by PHG and MJ using Covidence [15]. The two authors assessed the 5338 articles by screening title and subsequent reading the abstract of the studies and classified the studies as “eligible,” “not eligible,” or “perhaps eligible.” Following the independent assessment, the two authors then compared result. Disagreements between PHG and MJ were resolved by consensus and by consulting with the other authors.

The data were extracted using a prespecified data collection form in Microsoft Excel.

Authors of three studies [16‐18] were contacted for further information on the 95% confidence intervals (CIs). Bulöw et al. [16] and Kurichi et al. [18] responded and provided additional numerical data.

The following information was extracted from each included study: (1) characteristics of participants (orthopedic diagnosis or surgical procedure); (2) type of comorbidity index; (3) relevant outcome measures (mortality); (4) database/register used; (5) data period; (6) country; (7) number of patients; (8) study design; and (9) relevant summary effect estimate.

The risk of bias for each individual study was assessed by PHG and MJ using Risk Of Bias In Non-Randomized Studies of Interventions (ROBINS-I) [19].

No summary effect measure was prespecified in the protocol. All 16 included studies reported the area under the curve (AUC) of receiver operating characteristic curve. Therefore, AUC was chosen as the primary effect estimate in this review; the AUC measures the predictive performance of a comorbidity index. The AUC value ranges from 0.5 to 1.0, where 0.5 indicates no predictive ability and 1.0 indicates perfect predictive ability. In general, an inexact guide can be used: 0.50: no discriminating ability, 0.60–0.69: poor accuracy, 0.70–0.79: fair accuracy, 0.80–0.89: good accuracy, and 0.90–1.00: excellent accuracy [20]. Confidence level of 95% is used when reporting AUC values.

We originally intended to do a meta-analysis on the studies that compared CCI with ECI. However, due to the variation in the different versions and models of the comorbidity indices, differences in reported outcomes, and validity issues concerning the use of meta-analysis of predictive models [21] we omitted the meta-analysis and instead did a more narrative report of the studies.

The searches of the MEDLINE, Embase, and Cochrane Library databases provided a total of 6333 citations (Fig. 1). After adjusting for duplicates using Endnote and Covidence, 5338 citations remained. Of these, 5300 studies were discarded after reviewing the titles and abstracts (Fig. 1).

The full text of the remaining 38 citations was assessed. Sixteen studies met the inclusion criteria and were included in the systematic review (Table 1). No further studies were identified by screening the references of the included articles or searching the grey literature.

All of the studies were cohort studies (Table 1). All but one of the included studies [22] used various types of registers to gather the administrative data, with the National Hospital Discharge Survey being the most commonly used database. The data periods varied from 1990 to 2014. Nine of the studies were carried out in the USA. The number of patients included in each study ranged from 506 to 33,194,141.

Eight of the included studies compared only the ECI with the CCI [16, 18, 23‐29]. In addition to these two indices, six other indices were also assessed: a Multipurpose Australian Comorbidity Scoring System, the Cumulative Illness Rating Scale, the Preoperative Score to Predict Postoperative Mortality, the Iezzoni Index, and the Clinical Classifications Software and Modified Frailty Index.

Regarding the outcome measures, in-hospital mortality was the most commonly reported outcome, followed by 1-year and 30-day mortality.

Most studies were evaluated as having a moderate risk of bias. Two studies were evaluated to have serious risk of bias due to either confounding or bias in reported results (Table 2).

The 16 included studies varied markedly regarding the type and version of comorbidity indices assessed, and the follow-up time, which ranged from in-hospital to 14 years (Table 1). In eight studies (Bülow et al. 2019 [16], Bülow et al. 2017 [23], Menendez et al. 2014 [26], Menendez et al. 2015 [28], Menendez et al. 2015 [29], Ondeck et al. 2008 [31], Radley et al. 2008 [32], and Toson et al. 2016 [33), the comorbidity indices was compared to a base model, comprising of demographic variables (e.g., age and sex) and in two of those, the base model outperformed both CCI and ECI.

In nine studies, the comorbidity indices were combined with demographic variables (e.g., age and sex). In all nine studies, this combination increased the predictive power.

This systematic review summarizes the evidence on how different comorbidity indices compare to one another in predicting mortality in an orthopedic setting. Most studies compare the ECI and CCI and found no clear difference in the predictive ability of these two indices. Six studies compared other comorbidity indices to the CCI and ECI and four of those performed equally good or better. For all comorbidity indices, adjusting for demographic factors together with comorbidities generally provided better predictive ability.

We found no clear difference in the predictive ability of the ECI and CCI. Both have good to excellent predictive ability for in-hospital mortality, but only poor to fair performance for all other time points. As the ECI and CCI comprise of several of the same comorbidity conditions this result might not be surprising. However, these findings are inconsistent with other studies on this topic. Yurkovich et al. [8] conducted a systematic review on comorbidity indices using administrative data and concluded that the ECI consistently outperformed the CCI in predicting both short- and long-term mortality. Sharabiani et al. [9] conducted a systematic review on comorbidity indices and found that comorbidity adjustment methods showed better prediction for long-term (> 30 days) than short-term (< 30 days) mortality. However, neither of these reviews specified a particular patient setting. Therefore, one hypothesis to explain this discrepancy of results can be the differences in the patient settings. This corresponds well with the statement by Yurkovich et al. [8] that the predictive ability of the comorbidity indices primarily depends on the outcome and study population. Comorbidity indices might not perform as well in an orthopedic setting compare to other settings. Therefore, any difference between comorbidity indices might not be evident. Another difference to consider is that the current review included fewer studies than the two previously mentioned reviews.

The Multipurpose Australian Comorbidity Scoring System [33], the study by Menendez et al. [29], and the study Inacio et al. [17] were all characterized by including a large number of comorbidities and all three performed well. The systematic review by Sharabiani et al. [9] found the same association. However, the evidence is not sufficiently extensive at this point to make a clear recommendation to use these newly developed comorbidity indices. As this study shows that the predictive ability of comorbidity indices are very similar, the choice of which comorbidity index to use in adjusting of comorbidity must to a large extend depend on other variables. The most correct method of choosing which comorbidity index to use is to do a validation of the indices for the study population of interest. However, this is a time- and labor-consuming work. Other factors might also play a role, such as comparability to previous studies, which use a specific index, and which comorbidity variables that are available in the relevant register.

Since comorbidity is associated with mortality, one would presumably expect that adjustment for comorbidity would increase the predictive ability considerably. This was not always evident in this review. One challenge with using administrative data to gather information on comorbidities is the potential underreporting in administrative databases and coding practices. Several studies have shown that administrative data can be an unreliable source [34, 35]. This might explain the varying and sometimes poor predictive ability of the comorbidity indices. The choice of which administrative database to use is of importance, and research on this matter is limited.

There are several limitations of this work. This review was limited by heterogeneity in the included studies. For instance, the CCI was assessed as one index all together when in fact it includes all of its subtypes, with various amounts of comorbidities (ranging from 12 to 19) and different weighting algorithms. Initially, we wanted to do an exploratory subgroup analysis for the subtypes of CCI and ECI, but we were not able to define all of them due to insufficient descriptions in each study.

Additionally, although the patient population was more specific than in previous systematic reviews on this topic [8, 9], it could still be classified as broad. We can assume, for instance, that patients undergoing arthroplasty surgery for osteoarthritis often differ significantly from patients with hip fracture. Therefore, it is debatable how comparable these patient groups are.

Further, we were not able to define in-hospital mortality. Therefore, this time period could vary markedly between studies and might cause some misinterpretation of the results.

Finally, at the moment, the overall evidence is not sufficiently extensive to determine the appropriate index for every mortality time point.