Validation of administrative health data for the pediatric population: a scoping review

Administrative health data, which are generated through the routine delivery of health care programs [1], are rich sources of population-based information for research about population health and health services. However, these data were not originally intended for research, leading to many questions about their validity for this purpose. In particular, the use of diagnostic codes in these data, which are typically recorded using the World Health Organization’s International Classification of Diseases (ICD), to accurately identify patient populations with acute or chronic diseases has been the focus of multiple validation studies. These studies compare individual diagnostic codes or more complex case-finding algorithms based on combinations of diagnosis codes and other criteria in administrative health data to an external data source, such as survey data, medical charts, or laboratory test results [2, 3]. Validation studies and systematic reviews of validation studies [4‐6] have primarily focused on adult populations; there have been few validation studies conducted in pediatric populations. A recent review of the quality of validation studies underscored the importance of population-specific studies, because validity may be heterogeneous across populations [7].

Age may be particularly important in the assessment of diagnostic validity because pediatric and adult diseases often differ [7]. For example, only five percent of pediatric patients with juvenile idiopathic arthritis (formerly called juvenile rheumatoid arthritis) have a disease pattern similar to the pattern observed in adult patients with rheumatoid arthritis [8], which may result in discrepant diagnostic validity estimates between the two populations.

The purpose of this study was to synthesize the published literature on the validity of diagnoses recorded in administrative health data for the pediatric population (≤20 years). This study was conducted to identify gaps in the literature and opportunities for future research.

A total of 1204 abstracts were identified by the literature search (Figure 1). After removing duplicates, 817 unique abstracts were screened for study inclusion. Fifteen were excluded based only on the title and abstract (1.8%). Thus, a total of 802 articles (98.2%) underwent full text review. Of this number, 765 (95.4%) were excluded for the following reasons (reasons are not mutually exclusive): 608 (75.8%) were not validation studies, 466 (58.1%) did not use administrative health data, and 216 (26.9%) did not conduct separate validation analyses for pediatric patients. Thirty-six articles met criteria for further analysis. A hand search of the reference lists of included studies identified one additional article, yielding a final sample of 37 articles. Cohen’s κ for study inclusion/exclusion and data abstraction was 0.88 (95% CI 0.72, 1.00) and 0.97 (95% CI 0.94, 0.99), respectively.

The characteristics of included studies are summarized in Table 2. Increasing numbers of validation studies were published over time, with 11 (29.7%) published between 2006 and 2010 and a further 12 (32.4%) identified between 2011 and 2012 (up to the end of the search period). Just over 40% of the studies (16 studies, 43.2%) were conducted using administrative health data from the United States, followed by Canada (9 studies, 24.3%). All validation studies included both males and females. There was a trend of smaller numbers of validation studies as age increased, with fewer studies (17 studies, 45.9%) including individuals aged 16 to 20 years.

Slightly more than one-third of studies (14 studies, 37.8%) validated diagnosis codes, while more than half evaluated case-finding algorithms (21 studies, 56.8%), which use a combination of diagnosis codes and other criteria (e.g., procedure codes) to identify cases with the condition of interest. Two studies (5.4%) validated both diagnosis codes and case-finding algorithms.

Only five conditions were investigated in more than one study: diabetes (10.6%) [13‐16], inflammatory bowel disease (5.4%) [17, 18], asthma (5.4%) [19, 20], rotavirus infection (5.4%) [21, 22], and tuberculosis (5.4%) [23, 24]. However, a diverse range of conditions were investigated in single studies, including obesity [25], vaccine-related illness [26], injuries [27], autism [28], febrile neutropenia in oncology patients [29], high risk conditions [30], dermatologic conditions [31‐33], congenital anomalies [34], cardiac defects [35], respiratory illnesses excluding asthma [36‐38], neurologic conditions [39], other gastrointestinal conditions [40‐43], genitourinary conditions [44, 45], serum sickness [46], thrombosis [47], maternal/perinatal conditions [48], and drug-related anaphylaxis [49].

Administrative health data sources that were validated consisted of inpatient (25 studies, 67.6%), outpatient (9 studies, 24.3%), and emergency room records (2 studies, 5.4%), as well as pharmacy data (2 studies, 5.4%). Studies that validated individual diagnoses or case-finding algorithms in a single database were most frequent (25 studies, 67.6%). Eighty percent (20 studies) of the 25 studies that validated data from a single administrative database used inpatient administrative data (54.1% of 37 studies), followed by outpatient administrative data (4 studies, 10.8% of 37 studies) and emergency department data (1 study, 2.7% of 37 studies). Twelve studies (32.4% of 37 studies) validated information from multiple, linked administrative data sources; five of these studies included inpatient records (13.5% of 37 studies) and five included outpatient records as one of the databases.

The main diagnosis coding systems that were validated included ICD-9 or ICD-9 CM (28 studies, 75.7%) and ICD-10 or ICD-10-CA (8 studies, 21.6%). The most frequent external data sources used to validate administrative data were medical charts (23 studies, 62.2%) and disease-specific registry data (6 studies, 16.2%). Other validation sources used included clinical databases, laboratory records, and survey data. Validation measures reported include sensitivity (24 studies, 64.8%), specificity (20 studies, 54.0%), positive predictive value (17 studies, 45.9%), and negative predictive value (8 studies, 21.6%).

The prevalence of chronic pediatric conditions has increased over recent decades; it is estimated that between 16% and 51% of children have at least one chronic condition [50‐52]. Medical advances have improved survival for conditions that were once fatal, resulting in an increasing number of children with special healthcare needs [50‐52]. In order to allow for resource planning and optimization of care, the long-term outcomes of children and youth with chronic conditions need to be determined, as does their healthcare utilization [52]. Administrative health data are an appropriate source to conduct long-term follow-up studies, but validation studies are important to ensure that true cases of disease can be ascertained in these data.

To the best of our knowledge, this is the first scoping review to describe validation studies of administrative health data in the pediatric population. Only a small number of pediatric validation studies were identified, whereas a recent systematic review about the quality of reporting of administrative data validation studies that included all age groups, identified 271 studies published prior to June 2009 [7], most of which focused on the adult population. The increasing number of pediatric validation studies in recent years suggests that there is growing recognition that pediatric populations are important to consider separately from adult populations when validating administrative health data. The vast majority of studies were conducted in North America, reflecting a general trend for validation studies [4‐6]. Most of the 37 studies included in this scoping review validated complex case-finding algorithms that use a variety of information found in administrative health data to ascertain disease cases.

Case-finding algorithms typically take advantage of linked administrative health databases, whereas validation of individual diagnoses may only take place in a single administrative data source. In many administrative data systems, data linkage creates the opportunity to evaluate case-finding algorithms that will have sensitivity or specificity that is greater than what can be observed by examining a diagnosis in an unlinked database.

It is surprising that no validation studies were identified for common chronic pediatric conditions such as attention deficit/hyperactivity disorder and obesity [50]. Only diabetes, inflammatory bowel disease, asthma, tuberculosis, and rotavirus infection were validated in more than one setting. It is well known that diabetes can be ascertained from administrative health data with high specificity and sensitivity for adult populations, which may have contributed to increased interest in performing validation studies for this diagnosis in the pediatric population. In addition, with the growth in rates of juvenile diabetes, this is an important condition for chronic disease research and surveillance [50]. There is a gap in the literature for conditions validated in the adults but not in the pediatric setting. Rheumatoid arthritis is one example of a chronic condition for which several validation studies have been published in the adult population [53‐55] but similar validation studies are lacking in the pediatric age group. In fact, no validation studies were found for chronic inflammatory arthritis in the pediatric population at the time of this scoping review.

While this scoping review has several strengths, including the breadth of citation databases investigated, the multiple health conditions that were included, and the range of characteristics of the studies that were examined, it does have some limitations. Only English language publications were included. Conference proceedings and articles that were not published in peer-review journals were excluded. Publication bias may affect the generalizability of the scoping review results. Nevertheless, these factors taken together are not likely to result in a large number of missing research studies, and hence cannot account for the relative dearth of pediatric administrative data validation studies that were identified.

For conditions with several published validation studies in the pediatric population, such as diabetes, disease specific systematic reviews evaluating the quality of studies should be examined, but only once more studies have been published; at present, there are too few validation studies in pediatric publications to warrant systematic reviews. Many pediatric conditions are treated primarily in an outpatient setting, and almost all chronic diseases in this population require at least some outpatient care, yet validation studies in this setting are lacking. Patients with milder disease or better access to outpatient-based services may never need hospitalization, and validation studies primarily based on inpatient data likely do not capture the true spectrum of chronic disease severity. Validation studies in the outpatient setting can be challenging to conduct due to small patient numbers in individual centres, lack of standardized charting, and difficulties accessing medical records. As electronic medical records become more widely available, this could potentially facilitate validation studies in the outpatient setting.

Numerous studies about the diagnostic validity of administrative health data for the adult population have been published [7], but studies about the pediatric population have been limited in number and scope, despite the fact that diagnoses may not be equally valid in both populations. An increasing number of children are living with chronic conditions. Administrative health data can be used to estimate the burden of these conditions and provide long-term outcomes data for studies about mortality, health care utilization, and comorbid conditions. In order for administrative data to serve these purposes, their validity must be established. Our scoping review of published literature on diagnostic validity of administrative health data in the pediatric population revealed multiple gaps in the pediatric literature. Common chronic pediatric conditions have not been validated in a multiple settings, the number of validation studies decreased with increasing age within the pediatric population, and although many pediatric conditions are treated primarily in an outpatient setting, validation studies in this setting are lacking. Further studies are needed to examine validity for a broad spectrum of pediatric health conditions, in outpatient populations, and in both younger and older age groups.