Comparison of risk adjustment methods in patients with liver disease using electronic medical record data

The data used in our study was derived from the EMR of Beijing YouAn hospital, one of the leading teaching hospitals specialized in liver diseases in China and treating over 300,000 patients from all over China each year. In 2008, the EMR system was officially implemented in YouAn hospital and inpatient documentation completely switched from paper charts to EMR. For each patient, the EMR contains a front summary page, as well as sections with detailed information on admission, discharge, surgery/procedure, death, laboratory test results, radiology test results, pathology report, physician’s notes, hospitalization billing records, and electronic prescription. Among these sections, laboratory test results, electronic prescriptions, and billing records are completely structured without any free text. The front page, admission and discharge records, and radiology test results, however, are only semi-structured and contain both structured drop-down lists and free-text fields. The hospital assigned a unique identification number to each patient; all sections of EMRs are linked using the identification number.

The study population included patients with cirrhosis hospitalized at Beijing YouAn hospital between January 1st, 2010 and September 30th, 2014, who were at least 18 years old and consented to use their EMRs for research (nearly all patients provided consent), and excluded patients with missing in-hospital mortality status. We excluded 145 (1.3%) patients due to missing information on in-hospital mortality (the missing was likely caused by physicians’ unintentional incomplete documentation) and 180 (1.6%) patients who underwent liver transplantation, given this group of patients were much more complicated in contrast to other patients. In total, 11,122 adult cirrhosis patients were analyzed. This study was approved by the YouAn Hospital Research Board of Ethics and the Health Research Ethics Board at University of Calgary (Ethic committee’s reference number: REB14-0815).

The outcome measure was in-hospital mortality that was recorded in the EMR. Liver disease variables were defined using our previously developed and validated EMR case definitions [30]. The validation study showed that most of the case definitions had high validity (positive predictive value over 80%). Using the validated case definitions, we defined the following variables: cirrhosis, PLC, hepatitis, hepatic encephalopathy (HE) and ascites, as well as the Charlson and Elixhauser comorbidities at the time of admission. In addition, the laboratory test results required to construct the CTP and MELD scores were directly extracted from the EMR system. These laboratory test results included the serum level of albumin, total bilirubin, creatinine, sodium (Na), and the international normalized ratio of prothrombin time (PT-INR). For inpatient episode with multiple laboratory tests, results from the tests conducted at or immediately after admission were used. Only 232 patients had missing values of one of above laboratory tests. We assumed these missing values fell in the normal range at admission. Chart review on 30 charts randomly selected out of these 232 patients supported this assumption.

Using EMR data in the latest admission, we defined in-hospital mortality and the laboratory test results. To define chronic diseases (e.g., comorbidities) we included the information in the multiple admissions within 1 year prior to the latest admission date.

Commonly used variants of CCI [10, 12] were tested: the all individual comorbidities of CCI (referred to as CCI), the number of Charlson comorbidities categorized (0, 1, 2, ≥3 comorbidities) (referred to as CCI categorized), the score of CCI (referred to as CCI score), which is the summation of the weighted score of each comorbidity, and the categorized CCI score (0, 1–2, 3–4, ≥5 points) (referred to as CCI score categorized), (See detailed description of the tested models in Table 1). For ECI, models using the individual Elixhauser comorbidities (referred to as ECI), and the number of Elixhauser comorbidities categorized (0, 1, 2, ≥3 comorbidities) (referred to as ECI categorized) [11] were tested (Table 1). Both CCI and ECI contain variables related to liver diseases. We excluded “mild/moderate to severe liver disease” in CCI and the “liver disease” in ECI. PLC was excluded from the variables of “any malignancy”, “metastatic solid tumor”, and “solid tumor without metastases”.

For MELD, three common variants were tested (Table 1), including MELD score (referred to as MELD score) [6], MELD sodium score (referred to as MELDNa score) [31], and five-variable MELD score (referred to as 5vMELD score) [32]. MELD score = 3.78 × ln[serum total bilirubin (mg/dL)] + 11.2 × ln[INR] + 9.57 × ln[serum creatinine (mg/dL)] + 6.43) [6]. To avoid scores below 0 in the logarithm, value less than one is rounded to 1 (e.g., for total bilirubin with 0.75, a value of 1.0 is assigned). MELDNa = MELD score + 1.59 [135 - Na], where Na is bounded between 120 and 135 mmol/L (Na lower than the low limit is assigned with a value of 120 mmol/L, and Na higher than 135 mmol/L is assigned a value of 135 mmol/L) [31]. 5vMELD score = MELDNa + (5.275 × [4-albumin]) – (0.136 × MELDNa × [4 - serum albumin]), where albumin is bounded between 1 and 4 g/dL [32]. Two variants of CTP were tested: CTP classification (referred to as CTP) and CTP score (referred to as CTP score) (Table 1). The CTP score is defined by summing the assigned score for each of the five variables including HE (absence = 1, slight-medium = 2, and refractory = 3), ascites (none = 1, mild = 2, and moderate to severe = 3), total bilirubin (<34 μmol/L = 1, 34–50 μmol/L = 2, and >50 μmol/L = 3), PT-INR (<1.7 = 1, 1.7–2.3 = 2, and > 2.3 = 3), and albumin (>3.5 = 1, 2.8–3.5 = 2, and <2.8 = 3) [5]. Calculating CTP score requires the refined severity of HE and ascites; however, 13.5% patients had unknown severity of HE, and the patients with unknown severity of ascites accounted for 52.6%. We excluded these patients from the CTP score model because we were not able to calculate CTP score for these patients. To include the patients with unknown severity of HE or ascites, we also categorized HE and ascites into binary variables (presence or absence) in the CTP classification model. In addition, we tested risk adjustment models using combination of CTP, MELDNa scorer and ECI (Table 1). For these models incorporated both comorbidity index and liver disease severity score, we tested the interactions between different risk adjustment instruments.

Descriptive analysis was conducted and logistic regression models (as described above) were used to predict in-hospital mortality. The baseline model consisted of age, sex, and admission status (urgently or not). Concordance-statistic (c-statistic) was used to assess the performance of the risk adjustment models [33, 34]. C-statistic of 0.5 means that the ability of discrimination of the model is zero; the discrimination power is regarded as “unacceptable” when c-statistic range from 0.50 to 0.69; or “acceptable” when c-statistic range from 0.70 to 0.79; or “good to excellent” when c-statistic is 0.80 or greater. The 10-fold cross validation [35] was used to calculate the corrected c-statistics to adjust for the number of independent variables in the model considering that c-statistic increases with the number of independent variables. We also conducted bootstrapping (1000 samples) and calculated 95% confidence interval for c-statistics (95% CI) for internal validation of the c-statistic of each model [36].

Probability of death for each patient was calculated by the logistic regression models; patients were ranked and allocated to different risk groups based on the predicted probability of death. The agreement of observed and expected number of death was assessed. Graphs were plotted to show the expected and observed mortality rates across the various risk groups.

In addition, similar analyses were conducted using the subsample of patients with viral hepatitis, alcoholic hepatitis, PLC, decompensated cirrhosis, and no-procedure subgroups (without undergoing hepatectomy, liver transplantations, transcatheter arterial chemoembolization, and endoscopic treatment). All analyses were performed in SAS version 9.4 (Cary, NC).

At time of admission, 25.0% (2764) of the cirrhotic patients were diagnosed with hyponatremia (Na < 135 mmol/L), 19.7% (2190) with high creatinine level (>88.4 umol/L), 9.5% (1051) with abnormal PT-INR (>1.7), 40.98% (4558) with high total bilirubin level (>34.2 umol/L), and 51.7% (5752) with hypoproteinemia (albumin < 2.8 g/dL). At time of admission, 18.6% (2069) of the cirrhotic patients had HE, and 58.3% (6478) had ascites. The most common five comorbidities were diabetes uncomplicated (35.6%), hypertension (complicated and uncomplicated) (28.5%), alcohol abuse (25.0%), fluid and electrolyte disorder (15.4%) and peptic ulcer disease (11.1%).

In general, in-hospital mortality was higher among male patients, older patients, urgently admitted patients, patients with abnormal clinical variables, patients with a certain comorbidity (except for acquired immune deficiency syndrome and peripheral vascular disease), or patients with higher MELD, MELDNa or 5vMELD score than their counterparts (see Table 3). As number of Charlson or Elixhauser comorbitites increased, so did in-hospital mortality. A similar pattern was found with the number of abnormal CTP variables.

The c-statistics and its 95% confidence intervals (CI) of the risk adjustment models predicting in-hospital mortality for overall cirrhotic patients were presented in Table 4, while those for the subgroups of cirrhotic patients (viral hepatitis, alcoholic hepatitis, PLC, decompensated cirrhosis, and non-procedure) were presented in Table 5.

Results on model performance within patient’s subgroups (those with viral hepatitis, alcoholic hepatitis, PLC, decompensated cirrhosis, and no-procedure subgroups) remained the same: c-statistics increased in a consistent order from the CCI, ECI to MELDNa score (or CTP). Compared with models employing only single risk adjustment model, c-statistic of models that combined both liver disease severity and comorbidity indexes was shown to be better. Model combining CTP and ECI improved the c-statistic compared with the CTP model (c-statistics: 0.887 versus 0.851, p < 0.0001). Similarly, model that combined ECI and MELDNa score outperformed model that includd MELDNa score only (c-statistics: 0.882 versus 0.849, p < 0.0001).

Figure 1 presents the observed and expected mortality across model-defined risk groups for the six models (CCI, ECI, MELDNa score, CTP, ECI + MELDNa score, and ECI + CTP) in the overall sample. The “spread-out” of the expected mortality generated from combined models (i.e., CI + MELDNa score and ECI + CTP) was much wider than the models with only comorbidities, MELD or CTP.

We assessed the performance of risk adjustment models in predicting in-hospital mortality for patients with cirrhosis, using a single model or a combination of two models among ECI and CTP (or MELDNa score). All models significantly outperformed the baseline model with age, sex, and admission status. These results provided support of the use of these models as risk adjustment instruments for liver disease. While all models were shown to have reasonable predictive power, liver disease severity indexes (CTP and MELDNa score) were shown to be better than the comorbidity indexes (CCI and ECI). Moreover, comparing with individual comorbidity or liver disease severity index, combined models (e.g., CTP + ECI or MELDNa + ECI) demonstrated higher performance in predicting in-hospital mortality.

Between the two comorbidity indexes tested, ECI was found to be more predictive than CCI among all cirrhotic patients as well as for all the subgroups. This result was consistent with findings in the existing risk adjustment literatures for liver disease that used administrative data [9, 16, 18, 37]. The better performance of ECI could be explained by that ECI identified substantially more conditions than CCI, which contributed to a higher c-statistic [16, 37]. In our study, we used a category of number of comorbidities presence as one independent variable for ECI and CCI. This method showed similar c-statistics for ECI and CCI (0.794 versus 0.786).

Among the liver disease severity indexes tested, the discrimination ability of CTP was consistently shown to be higher than MELD and 5vMELD scores, and close to MELDNa score among all of the subgroups. This proved the appropriateness of ongoing use of CTP in practice to predict in-hospital mortality in cirrhotic patients. However, refined degree of HE and ascites may not be available in many datasets, making it impossible to use CTP as a risk adjustment instrument. The construction of MELDNa score only requires routine laboratory test results, which makes MELDNa score more reproducible, reliable and easier to apply [20‐22]. More importantly, our results showed that the performance of MELDNa score were very close to or better than that of CTP. This indicates that using MELDNa score instead of CTP might simplify the analysis without compromising the predictive accuracy.

MELDNa and 5vMELD scores had similar performance in predicting in-hospital mortality. 5vMELD score was generated through adding serum albumin level to MELDNa score. The additional variable in 5vMELD did not significantly improve its predictability of in-hospital mortality. The possible reason is that albumin level measured during hospitalization did not reflect the patient’s severity of disease because albumin was commonly administrated in inpatients with cirrhosis.

Overall, the liver disease severity indexes (MELD score and CTP) outperformed the comorbidity indexes (CCI and ECI) on prediction of in-hospital mortality. The possible reason is that the most recent laboratory test results within one hospitalization episode could reflect the severity of liver disease at the occurrence of hospitalization outcome (mortality). We conducted sensitivity analysis to address this explanation. We calculated MELDNa score at near discharge time and fitted model to predict in-hospital mortality. The c-statistic for MELDNa model increased significantly from 0.849 (95% CI: 0.838–0.861) for near admission time to 0.912 (95% CI: 0.903–0.921) for near discharge time. This supported our hypothesis that the performance of risk adjustment instruments improves when they are constructed based on information collected close to the outcome event.

The model incorporated MELDNa score and Elixhauser comorbidities obtained significantly higher predictive ability compared to the MELDNa score model. This indicated that to increase predictive probability of mortality during hospitalization, physicians should not only consider the MELDNa score but also presence of comorbidities. Further research is required to develop summary score and cut-off value to predict individual patient’s outcome.

Our study has several limitations. First, we used data derived from one hospital EMRs, and the generalizablility of the results may be a concern. However, generally the c-statistics of the compared risk adjustment methods were consistent with results from other studies in existing literatures [9‐12, 15, 20‐23]. Second, we only analyzed inpatient EMR data and were unable to assess patients’ outcome after discharge. Third, the odds ratios of certain predictors were not reliable due to low prevalence. The possible reason for the low prevalence of these diseases is that the data is from a hospital specialized in liver disease. However, the purpose of this study is to compare the performance of the common risk adjustment instruments. Lastly, the missing values on certain variables were common. In our EMRs, presence of ascites was well recorded but degree of ascites was often missing (more than 50%). Exclusion of these patients from the CTP score model could under-estimate the c-statistic. Other study also reported that severity of HE and ascites was commonly missing [22].