Background
Approximately 10% of ulcerative colitis (UC) patients require a colectomy within 10 years of diagnosis [
1]. Colectomy for UC is a technically demanding operation associated with morbidity and mortality [
2,
3]. Patients undergoing elective procedures have lower risk of postoperative mortality, ranging from 0.0% to 1.0% [
4‐
7]. In contrast, mortality in those requiring emergent colectomy was as high as 6.9% [
8‐
10]. Other factors that have been shown to influence postoperative outcomes include older age and comorbidities [
11].
Previous studies reporting postoperative outcomes in UC patients have used medical chart review to obtain clinical information. However, these studies are rarely population-based allowing for referral bias and have small sample sizes. Consequently, investigators have relied on administrative databases to study population-based estimates of postoperative outcomes [
11]. Administrative databases are time and cost efficient resources but interpretation of results derived from administrative data is dependent on the validity of administrative coding for predictors and outcomes. Therefore, validation of these databases is a priority in health services research [
12].
Although numerous studies have used administrative data to study UC outcomes [
11,
13,
14], few have validated the accuracy of administrative data in identifying UC patients who underwent a colectomy. Furthermore, the accuracy of administrative data in characterizing risk factors such as comorbidities is inconsistent. Under-reporting of comorbidities is high[
15] and differentiating postoperative complications from pre-admission comorbidities can be challenging. Consequently, inherent misclassification may be present when administrative databases are used to identify preoperative risk factors of postoperative complications [
16].
Thus, we compared estimates of the preoperative risk factors (age, emergent colectomy, and comorbidities) associated with postoperative complications in UC patients undergoing colectomy, derived from two data sources: chart review and administrative data. Subsequently, we evaluated the accuracy of administrative databases in defining: 1) the UC study population; 2) preoperative risk factors; and 3) postoperative complications.
Methods
Study Population
The Data Integration, Measurement and Reporting Hospital Discharge Abstract Database (DAD) captures all hospitalizations in the Calgary Health Zone of Alberta Health Services, Canada. The Calgary Health Zone is a population-based health authority under a public, single payer system, with an estimated population of 1.3 million in 2009[
17]. The DAD database used the
International Classification of Disease, Ninth Revision, Clinical Modification (ICD-9-CM) up to March 31, 2001; ICD-10-CA and the
Canadian Classification of Health Intervention (CCI) coding have been used since April 1, 2002.
The DAD was searched to identify adult patients (≥18 years) admitted to hospital between January 1, 1996 and December 31, 2007 with a diagnosis of UC (ICD-9 556.X, ICD-10 K51.X). We then identified UC patients who had a code for colectomy (ICD-9-CM 45.7, 45.8 or CCI: 1.NM.87, 1.NM.89, 1.NM.91, 1.NQ.89, 1.NQ. 90). Recognizing that the administrative database may have missed some UC patients who underwent a colectomy, we identified a cohort of patients admitted for an UC flare without a colectomy. All patients with UC at the primary diagnosis coding field and a random subset of patients with UC coded in the second or third diagnostic position were identified. All medical charts of patients identified by the administrative database were reviewed using a standardized, a priori defined electronic data extraction form. Data was extracted by five trained research assistants who were blinded to the original administrative coding. Fifty patient charts were used as a reference standard; all reviewers extracted data on these fifty charts and their abstraction was verified by a gold-standard reference data abstracter (SD) to minimize inter-observer variability.
Outcomes
The primary outcome was occurrence of in-hospital postoperative complications, defined as unexpected medical events occurring between the start of the operation and discharge from hospital. For the chart review, postoperative complications were stratified by severity using the Clavien Classification of Surgical Complications [
18] system (See Additional file
1). Each patient was assigned a postoperative status (≥ 1 versus 0 complications) and severity by Clavien class (I-V). For patients experiencing more than one postoperative complication, the most severe complication class was assigned. Complications were also stratified by category: gastrointestinal, cardiovascular, infectious, etc. See Additional file
1: Table S2 for the specific complications comprising each category. In the administrative database, we identified postoperative complications based on pre-defined ICD-9 and CCI codes that have been commonly used to identify postoperative complications [
11]. Complication codes used in the analysis can be found in Additional file
1: Table S3.
Variables
Variables extracted from both chart review and administrative data included age at colectomy; emergent versus elective operation; reason for colectomy (UC refractory to medical management, dysplasia or cancer, and acute complication of UC); and pre-admission comorbidities defined by the Charlson-Deyo[
19] and Elixhauser [
20] indices. Comorbidities were stratified by activity status to identify medical conditions that were managed during the admission. Colectomies were documented as elective if the decision to operate was made prior to hospital admission; in contrast, the decision for emergent colectomy occurred during the admission (e.g. in response to acute life-threatening complications of UC flare or medically refractory disease). In the administrative database, elective colectomies were defined as those coded with an ‘elective’ status, while emergent colectomies were defined using a composite of either ‘emergent’ or ‘urgent’ codes. Age, comorbidity, and admission type were
a priori defined as preoperative risk factors and subsequently validated because previous studies have shown that they were associated with postoperative complications in UC patients [
11,
21].
Data analysis
The administrative coding was validated against the chart review for the study population, admission type, comorbidity status, and postoperative complications. In our primary cohort, we validated the accuracy of administrative data in identifying UC patients undergoing colectomy. Secondarily, a cohort of UC patients admitted for flare without operation was reviewed to detect colectomy patients not captured in our primary cohort. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) with 95% CIs were calculated for UC diagnostic codes alone and the combination of UC + colectomy codes. For patients presenting with UC flare, the analysis was also stratified by UC diagnostic position. In secondary analysis, we validated the accuracy of specific procedural colectomy codes. We also validated the accuracy of administrative data defining emergent or urgent versus elective colectomy. In a sensitivity analysis, we excluded patients admitted urgently, and compared only emergent versus elective patients.
The validity of administrative data in capturing comorbid conditions in colectomy patients was also assessed. We searched the administrative database to identify patients coded with Charlson-Deyo comorbidities and cross-matched these with the true comorbidities identified by chart review. Sensitivity, specificity, PPV, and NPV with 95% CIs were calculated for administrative data predicting whether patients had 0 versus ≥ 1 pre-hospital comorbidities and for each of the 17 specific comorbidities. Subgroup analysis of only comorbidities active on admission was performed. Analyses were repeated for Elixhauser comorbidities (Additional file
1: Table S4-S6.
We assessed the validity of administrative data in capturing in-hospital postoperative complications. Patients coded with complications in the administrative database were cross-matched with those identified by chart review. Sensitivity and specificity with 95% CIs was then calculated under 3 scenarios: 1) none versus any complications; 2) none or Clavien class I versus class II-V complications; and 3) none or class I-II versus class III-V complications. Sensitivity and specificity in capturing specific categories of complications was also determined.
Multivariate logistic regression analysis was performed to examine the association between preoperative risk factors and postoperative complication. In primary analysis, postoperative complication status was defined as none versus any complication. Age (defined as 18–34, 35–64, and ≥ 65 years), comorbidity (0 versus any Charlson comorbidity, and secondarily Elixhauser), and admission type (emergency versus elective) were a priori included into the logistic regression model. Odds ratios with 95% confidence intervals (CI) were calculated for each preoperative risk factor. Two logistic regression models were developed for comparison: 1) data derived from chart review and 2) data derived from the administrative database.
Statistical analyses were performed using SAS statistical software (version 9.2, SAS Institute Inc., Cary, NC). Ethics approval for the study protocol was granted by the Conjoint Health Research Ethics Board at the University of Calgary, study #21833.
Discussion
We conducted this study to evaluate whether outcomes derived from administrative databases accurately represent outcomes obtained from retrospectively reviewing medical charts. Both the administrative database and the chart review identified age, preoperative comorbidities, and emergent surgery as risk factors for postoperative complications following colectomy for UC. However, administrative data overestimated the magnitude of the risk for comorbidities and emergent operations as compared to chart review. Differences in risk estimates were in part explained by misclassification errors associated with the administrative database defining the study population, preoperative risk factors (i.e. comorbidity and emergent colectomy) and postoperative outcome (i.e. complications). The administrative database was more accurate at identifying comorbidities active at admission and the most severe postoperative complications; this selective coding likely biased the risk estimates away from the null hypothesis.
Both clinical [
22,
23] and administrative database studies [
11,
24] have identified advancing age, comorbidities, and emergency operations as risk factors for postoperative complications following colectomy and other abdominal surgeries [
25‐
28], though few have evaluated the difference between the two methods[
29]. In our analysis both the administrative and chart review data predicted an approximately two-fold increase in complications in those aged ≥ 65 compared to age 18–34 years. Close agreement of OR between administrative and chart data was expected because age is objective and reported with near perfect accuracy in both data sets.
The magnitude of effect for emergent operations was greater with the administrative data as compared to chart review. The administrative database was less specific for identifying emergent colectomy, with a high prevalence of false positives. Sensitivity analysis excluding patients with ‘urgent’ codes demonstrated improved specificity suggesting that the code ‘urgent’ is more aligned with an elective, rather than emergent admission. For example, patients electively admitted for an operation occurring within 24 hours of admission were at times coded as ‘urgent’.
Adaptations of the Charlson and Elixhauser comorbidity indices [
20,
30] have been validated for risk adjustment of postoperative morbidity and mortality [
31‐
33]. In our analysis, Charlson comorbidities were significantly associated with worse postoperative outcomes; though, the magnitude of effect was greater in the administrative database than chart data. Preferential recording of comorbidities actively managed in-hospital may explain this difference. The sensitivity for most comorbid illnesses was low, but increased when the analysis was restricted to active comorbidities. Our findings were similar to other validation studies that have found poor sensitivity of comorbidity coding[
15] and underreporting of chronic comorbidities not requiring treatment [
34]. Despite the low sensitivity of administrative data, other studies have found that prediction of in-hospital mortality was identical to indices derived from chart review [
31,
35]. Additionally, among patients with multiple postoperative complications physicians may record more comorbidities in the discharge summary to explain the poor outcomes, while this may not be detailed in patients with an uncomplicated postoperative recovery.
The administrative database was 86% accurate in identifying patients with UC undergoing colectomy. Additionally, a small subset (n = 15) of UC patients who underwent colectomy were not recorded in the administrative database. Thirumurthi
et al. found the sensitivity of the diagnostic code 556 × for hospitalization of UC was 84% [
36]. Diagnostic coding for UC may be less accurate than for other conditions; for instance, validations of administrative data in patients presenting with heart failure, acute COPD exacerbations, acute coronary syndromes, and subarachnoid haemorrhage have consistently demonstrated PPV of diagnostic codes exceeding 95% [
37‐
40]. In UC, the lower PPV may reflect uncertainties in diagnosis, especially from Crohn’s disease and other causes of colitis. A previous study also reported higher PPV (96.1%) for colectomy codes[
41] compared to our findings, although that validation was performed in a cohort of general surgery patients, with a smaller sample size (n = 56), and included procedural codes for rectal resections (484, 485, 486). In our study, follow-up procedures such as second stage ileopouch anal anastomosis were commonly misclassified as colectomies.
Administrative data did not reliably identify UC patients admitted with a flare without colectomy when the first three diagnostic positions were searched. Although nearly 80% of admissions with UC coded in the primary diagnostic position represented an acute flare of disease, UC recorded in the second or third diagnostic positions represented an acute flare in fewer than 10% of cases. This misclassification error is evident in the literature, as one study demonstrated strengthening of risk estimates when a sensitivity analysis was conducted to exclude Crohn’s disease patients admitted to hospital with a secondary diagnosis of Crohn’s disease [
42]. Consequently, prior studies using administrative databases have likely overestimated the true hospitalization rate of UC patients admitted for an acute flare of disease when non-primary diagnostic positions were searched.
The validity of postoperative complications in UC has not been reported. In our study, administrative data was 68% sensitive in identifying patients experiencing at least one complication after colectomy. Previous studies have also shown underreporting of complications in administrative data [
43‐
47]. Misclassification of postoperative complications contributed to the discrepancy observed between administrative and chart review data. The accuracy of administrative data in coding postoperative complications was correlated to complication severity: sensitivity increased when less severe complications were excluded from the analysis while the specificity decreased. Administrative data poorly identified minor complications (i.e. Clavien I), but captured the more severe and clinically significant postoperative complications. These findings were similar to our comorbidity validation, supporting the notion that administrative databases miss comorbidities and complications that likely have less clinical impact.
Misclassification of post-operative complications was predominantly due to the challenge in differentiating a postoperative complication from a comorbidity or a preoperative in-hospital complication. For example, UC patients who underwent colectomy and were coded for pulmonary embolism were recorded as a false positive if the pulmonary embolism was diagnosed before the colectomy was performed.
Several limitations of our study should be considered. First, the chart review was retrospective and not all clinical information may have been documented in the charts. As we comprehensively reviewed only the current admission, other comorbidities may have been missed. Second, we only had access to administrative codes for the patient’s hospitalization for colectomy; searching prior admissions may have improved the sensitivity of administrative coding, particularly for comorbidities. This provides an area for future study that may be explored in other datasets. Third, variation between reviewers was unavoidable although we attempted to limit inter-observer variability. Fourth, a small portion (2.6%) of UC patients coded for a flare but not colectomy actually underwent colectomy when the chart was reviewed. Conceivably, UC patients who underwent colectomy may not have been coded for either UC or colectomy, though this misclassification error is likely far less than 2.6%. Fifth, our sample size was sufficient to evaluate the overall validity of administrative data, but uncommon comorbidities and complications could not be validated. Similarly, large administrative database studies have the power to stratify comorbidity as a categorical variable (i.e. 0, 1, 2, or ≥3 comorbidities), but the prevalence of multiple comorbidities in our cohort was too low to accurately perform this subgroup analysis. Finally, the administrative database reflects the quality associated with Calgary’s DAD and thus, may not be generalized to other hospitalization databases. However, Calgary’s DAD is comprehensive, has been widely used and validated for health service research[
41], and has demonstrated generalizability in different settings. For example, a recent study demonstrated that Charlson comorbidities predicted in-hospital mortality similarly in Calgary’s hospital DAD as compared to hospitalization databases in France, New Zealand, Japan, Switzerland, and Australia [
48]. Thus, the data from this study should reflect practices and outcomes of other administrative databases and at minimum should motivate others to test the validity of local administrative databases.
Conclusions
Administrative data identified the same risk factors (advanced age, emergency admission, and comorbidities) for postoperative complications as chart review. However, the risk estimates were biased away from the null by the administrative database. The discrepancy in risk estimates may be explained by inaccuracies in defining the study population, complications, and comorbidities. Administrative data more accurately identified severe postoperative complications and comorbidities actively managed during the admission. Thus, despite the imperfect validity of administrative data, identified comorbidities and complications were likely the most clinically meaningful. Administrative databases are valid tools for IBD research, but the general inferences drawn from risk estimates should be interpreted in the context of limitations associated in identifying the study population, risk factors, and postoperative complications.
Competing interests
The authors declare that they have no competing interests.
Author contributions
All authors have reviewed and approved this manuscript. Study design and planning, data interpretation, manuscript drafting and approval: Dr. GK. Study planning, data collection and interpretation, manuscript drafting and approval: CMa. Study planning, data collection, manuscript approval: MC, M-CP, and MP. Data analysis and manuscript approval: JH. Study planning, data collection and interpretation, manuscript approval: Dr. SDS. Manuscript drafting and approval: Dr. RP, Dr. SG, Dr. RM, and Dr. HQ.