Development and validation of data quality rules in administrative health data using association rule mining

Administrative health data are generated at every encounter with the health care system, whether through a visit to a physician’s office or emergency department, or an admission to hospital [1, 2]. These data contain rich clinical and health service utilization information and have been widely used in epidemiological studies, disease surveillance, and health services research [3‐5]. However, since the data are primarily collected for administrative or billing purposes, there are underlying concerns about whether they are suitable for other secondary purposes. For example, an international comparison of health system performance using a set of validated patient safety indicators on administrative health databases across 15 Organization for Economic Co-Operation and Development (OECD) countries showed inconsistent effect estimates due to issues of coding inconsistency and incompleteness across countries [6].

In administrative health data, patients’ specific conditions are often identified using a list of International Classification of Diseases (ICD) codes. The validity of ICD codes for identifying specific conditions depends on whether the condition contributes to health service use, and on where, when, and how data are collected [7, 8]. Therefore, data quality is an important issue for administrative health data and there is an urgent need for data quality assessment tools. Various data quality frameworks, such as the information quality framework from the Canadian Institute for Health Information (CIHI), have been developed to describe and assess data quality [9].

A practical definition for data quality is whether the data can be used for their intended purpose [10]. A dataset with high data quality should allow results observed in other studies to be reproduced. For example, Lewis et al. compared data quality between two electronic medical record (EMR) databases by evaluating whether the associations between diseases or between diseases and drugs can be reproduced as expected [11]. If the analysis results for the same study are consistent from two databases, the two databases are comparable in that respect. In drug safety surveillance, a set of drug-outcome pairs with negative (no effect) and positive (increased effect) associations was developed to check whether a dataset can reproduce drug-outcome associations as expected [12]. Similarly, data quality can be assessed by checking whether a set of expected associations can be observed in a dataset, and the degree of conformity to the expected associations reflects the data quality.

Our previous study proposed to use association rule mining to find coding associations in administrative health data and demonstrated that those association rules are useful for checking coding completeness and consistency [13]. The main objective of this study was to develop and validate a subset of clinically relevant coding association rules. In this study, we systematically applied the association rule mining method in Canadian re-abstracted inpatient data to develop a manageable set of coding association rules to be reviewed for coding and clinical validity through expert panel review. Our study does not aim to obtain a complete list of clinically valid association rules; rather, this work provides a data quality assessment tool for monitoring and comparing data quality of inpatient administrative healthcare databases using a particular set of clinically relevant association rules.

Association rule mining was applied to the 30,628 re-abstracted records with a median of three (interquartile range (IQR): two to four) ICD codes. After removing the nested and non-significant rules, there were 974 rules at the three-digit level and 821 rules at the four-digit level for panel review. After the panel review, there were 388 rules at the three-digit level and 275 rules at the four-digit level (see Table 1). The plurality of the rules was for the age group ≥65 years (46.4% of the three-digit rules and 51.3% of the four-digit rules). Three-digit rules included 82 unique codes on the right-hand side and 107 on the left-hand side, while four-digit rules included 93 unique codes on the right-hand side and 84 on the left-hand side. Overall confidence and support for the three-digit rules were slightly higher than for the four-digit rules (see further details in Table 1).

Rules captured relevant clinical associations at their corresponding age groups. For the age group 0 to 4 (Fig. 2 (a) and Fig. 3 (a)), all the codes in the four-digit rules were from the chapter for conditions originating in the perinatal period. The codes that appeared in the most rules (highest degree centrality) were P07.1/P07.3: disorders related to short gestation and low birth weight, P28.4: apnoea of newborn/prematurity and P59.0: neonatal jaundice associated with preterm delivery. The three-digit rules also identified a few associations with codes from other chapters, such as H35 (retinal disorders) and Q21/Q25 (related to the circulatory system).

For the age group 20 to 44 (Fig. 2 (b) and Fig. 3 (b)), all the codes in the four-digit rules were from the chapter about pregnancy, childbirth, and puerperium. The most commonly used codes were related to complications of labour and delivery. Those included O60.1: preterm spontaneous labour with preterm delivery, O68.0: labour and delivery complicated by fetal heart rate anomaly, etc. Three-digit rules included codes from ischaemic heart disease (I21: acute myocardial infarction and I25: chronic ischaemic heart disease), E87: disorders of fluid, electrolyte and acid-base balance, and N17: acute renal failure.

For the age group 45 to 64 (Fig. 2 (c) and Fig. 3 (c)), the majority of the codes used in the rules were from chapters about diseases of the circulatory system, endocrine/nutritional/metabolic diseases, and diseases of the genitourinary system. Rules in the age group ≥65 (Fig. 2 (d) and Fig. 3 (d)) included codes from all other chapters except for the two chapters whose codes appeared in the rules for age groups 0 to 4 and 20 to 44. Rules for the age group ≥65 included codes from more different chapters than the other age groups, and displayed a more complex association structure. This was likely due to high prevalence of conditions and complexity of patients’ conditions.

Rules were then applied to assess the quality of 2013 Alberta DAD data. For the top 5 rules with high bias at the three and four digits (Fig. 4), Alberta data generally had lower rule confidences than CIHI re-abstracted data except for three-digit rules involving the codes I10: essential hypertension or E78: disorders of lipoprotein metabolism and other lipidaemias on the right hand side, and four-digits rules with codes O75.6: delayed delivery after spontaneous or unspecified rupture of membranes, I25.1: atherosclerotic heart disease, and I10. Alberta mandatorily codes I10 for surveillance purpose, which leads to more complete capture of hypertension and high confidence for the I10 related rules than other provinces in Canada. Some codes, such as I12: hypertensive renal disease for the age groups 45 to 64 and ≥ 65, and E11.2: type 2 diabetes with renal complications, were systematically under-coded in Alberta as indicated by very low confidence across all hospitals. A variance plot (Fig. 5) shows the differences of rules confidences between Alberta hospitals. The bias and variance measures identified different sets of deviations from expected associations that could potentially indicate data quality issues.

Through the analysis of the nationwide re-abstracted hospital discharge data and an extensive modified Delphi process, this study has developed a comprehensive set of ICD-10 diagnosis code rules that can be used for data quality assessment at the three-digit and four-digit levels for four different age groups. Three-digit rules covered more codes and displayed more complex association networks compared to four-digit rules. Rules for age groups 0 to 4 and 20 to 44 captured conditions concerning the perinatal period and conditions occurring during pregnancy, childbirth, and the postpartum period, respectively. Rules for age groups 45 to 64 and ≥ 65 captured a wider range of conditions. Finally, the application of the association rules was demonstrated by assessing data quality across 27 Alberta hospitals. These rules can be used in other contexts for comparisons between hospitals, provinces and countries.

Hospital administrative health data captures rich information on each hospitalization using standardized terminologies. However, the quality of coded hospital administrative data for research and reporting purposes has been widely debated since it is primarily collected for administrative/billing purposes and aims to capture clinically significant diagnoses that require treatment or the use of clinical resources [15]. Validation studies have shown substantial variations in validities of ICD codes. Among the 32 conditions included in the Charlson and Elixhauser comorbidities, sensitivity ranged from 0.127 for weight loss to 0.808 for metastatic cancer and positive predictive value ranged from 0.32 for blood loss anemia to 1 for AIDS/HIV [8]. It should be emphasized that the specificities of ICD codes were generally very good, with most conditions having specificity close to 0.99.

The coding process is very complex, with many possible opportunities for inconsistency and incompleteness [22]. Current coding standards only stipulate mandatory coding of clinically significant conditions [15]. Other barriers to high quality of coding include clinical documentation, variability in interpretation, and high quota expectations [23, 24]. Differing documentation practices between physicians and between hospitals result in varying levels of details in patient charts and variability in the documents available for data abstraction, which might cause issues of incompleteness [25]. Variability in chart interpretation and coding knowledge between coders can also result in coding inconsistency [23]. Pressure arising from high coding quotas can also lead to focus on productivity over quality, resulting in variations in coding completeness. Therefore, deciding how to assess coding consistency and completeness is a high priority for quality assessment of coded administrative health data.

Several conceptual data quality frameworks have been developed in Canada and other countries [26, 27]. For example, CIHI created an information quality framework and used five dimensions (relevance, accuracy and reliability, comparability and coherence, timeliness and punctuality, accessibility and clarity) to describe and assess quality relative to users’ needs [9]. Other researchers have harmonized data quality assessment terminologies and frameworks to enable data owners/users, patients, and policy makers to evaluate and communicate data quality findings in a well-defined manner with a shared vocabulary [28]. However, there is always a gap that remains between data quality terminology/frameworks and actionable tools for data quality assessment and reporting. Our study provides comprehensive lists of actionable data quality rules specifically designed for ICD coded data. Application of the rules can be viewed as a “slice and dice” process of breaking data into blocks defined by age groups, and examining the information from different clinical scenarios. Each rule examines a particular clinical scenario within a specific age group.

This tool has several advantages. First, rules were developed from a data-driven perspective and reflect the real practice of coding. As association rule mining may identify spurious rules resulting from randomly occurring correlations between codes, another strength of our study was that the clinical validity of the rules was guaranteed through the expert panel review process. The rules included over 100 different ICD codes at the three- and four-digit levels respectively, and covered over one third of diagnosis codes being used in practice. Furthermore, the rules provide a cost-effective means for conducting data quality assessments. Re-abstraction studies involve the recruitment of trained coding specialists to recapture the data following the same coding standard. The high cost restricts the number of records being re-abstracted and limits its scale of application. Detailed discussion about how to potentially apply rules for data quality assessment can be found in our previous publication [13].

We proposed and defined measures of bias and variance to identify potential data quality issues. Bias reflects the degree of difference of a data source compared to a reference standard. High bias might indicate high levels of under-coding or over-coding for one data source as compared to the reference standard. Variance reflects the degree of variability between different data sources. High variance indicates high levels of coding inconsistency across data sources. Bias and variance are weighted sums of absolute confidence difference. They assign high weights if the confidence interval is narrow. Bias and variance plots could be an intuitive measure to assess coding incompleteness and inconsistency and a potential measure indicator for data quality. Use of bias and variance measures could provide directions for data quality improvement.

Our study also has certain limitations. First, association rule mining is a type of unsupervised learning with several tuning parameters (e.g. threshold of support and confidence) for rule development. Although we have used the Apriori algorithm for our association rules implemented in implemented in arules package, other researchers with large datasets may consider using other, potentially more efficient association rule mining algorithms for rule development [29, 30]. We have implemented several methods to filter rules to ensure the uniqueness and robustness of rules to minimize the number of rules for panel review based on domain knowledge and statistical assessment, and may therefore have excluded useful rules during the rule cleaning process. Second, there is a need for further validation of the association rules against chart review or prospective case reviews. A more complete version of coded data is needed to establish the reference standard of rule confidences. We expect the rule confidences would generally increase due to common issues of under-coding in coded data. Third, rules were developed using a data-driven approach, and their completeness depends on the quality of re-abstracted data. Rules involving certain conditions might be missed due to errors in the re-abstracted data, as well as because the data reflects Canadian coding practices that aim to balance completeness with efficiency, rather than focusing solely on comprehensiveness. Fourth, rules only assess data quality at the aggregate level. Deviation from a rule by a single institution does not necessarily indicate an error. We did not provide a specific acceptability threshold for monitoring the quality of data for use because we have only selected a subset of acute care facilities in Alberta with certain characteristics. The association rules are soft, probabilistic rules that may have different appropriate thresholds for different sets of health care system characteristics. The variances between the re-abstracted data of the DAD from the original data from each of the institutions in Alberta, may be due to differences in specific coding standards and requirements of that institution. Thus, further investigation of rule confidence discrepancies and external validation are still required to gain a better understanding on how to set appropriate thresholds for data quality association rules.

In conclusion, we have developed a set of data quality rules through the association rule mining. The clinical and coding validity of rules were guaranteed through the panel review process. Rules covered interesting clinical scenarios at the different age groups. Bias and variance can be used to assess coding incompleteness and inconsistency between hospitals. The resulting work has great potential to monitor and compare ICD coded data across institutions, provinces and countries.