Comparison of Military Health System Data Repository and American College of Surgeons National Surgical Quality Improvement Program-Pediatric

Analysis of large medical databases holds substantial promise for improving healthcare delivery, including potential contributions to predictive modeling, surveillance, and other health improvement initiatives [1]. Likewise, national databases aggregate factors that otherwise might be too rare for meaningful analysis. However, the usefulness of such databases depends on generalizability and suitability as a representative sample [1], which can be evaluated in part by comparison against an existing validated resource.

One such validated database for pediatric surgical research is the National Surgical Quality Improvement Program-Pediatric (NSQIP-P) database, overseen by the American College of Surgeons in conjunction with the American Pediatric Surgery Association. This database collects 94 data points on surgical patients under 18 years of age, with additional data points for neonates under 30 days of age, and follows patient outcomes for 30 days post-operatively [2].

Also potentially useful for pediatric surgical research is the claims database maintained by the United States Military Health System (MHS) known as the MHS Data Repository (MDR). The MHS includes a universally-insured population of 9.4 million military and civilian beneficiaries [3] including 3 million children and has been called “America’s ‘undiscovered’ laboratory for health services research” [4]. The beneficiary population is considered to be demographically representative of the adult U.S. population from age 18–64 years [5‐7], however, to date, the pediatric generalizability has not been evaluated. Therefore, the aim of this research is to compare demographics and select outcomes (mortality, length of stay, and readmission) of the MHS Data Repository (MDR) to a validated resource, the NSQIP-P database, for five common pediatric surgical procedures to highlight the relative advantages and disadvantages of each resource. Findings of this research will help to evaluate the utility of the MDR as a tool for population-level research in pediatric procedures.

A total of five procedures were assessed: appendectomy, pyeloplasty, pyloromyotomy, scoliosis operations, and cleft lip/palate repair. Overall, NSQIP-P had a greater number of patients undergoing each procedure. Among both data sources, overall mortality was low, with 1 death reported following 24,965 appendectomies, 9 following 5838 scoliosis operations, and 1 following 6951 cleft lip/palate repairs. There were no mortalities following pyloromyotomy (n = 4054) or pyeloplasty (n = 898). Results stratified by operation are described in detail below.

In this study comparing basic characteristics and outcomes of pediatric patients undergoing common procedures in the NSQIP-P and MDR, we report that the age and sex of patients were overall similar as were mortality, length of hospital stay, and readmission rate. Race distribution for each procedure was similar between the two databases, with the MDR containing a generally slightly higher proportion of white patients, although the distribution varied between procedures. Based on these comparisons, we review the advantages and disadvantages of each database, in order to help guide researchers toward the most suitable resource for a given analysis.

Two important advantages of the MDR relative to NSQIP-P are follow-up duration and patient demographics. First, the MDR does not limit outcomes to 30 days, whereas the NSQIP-P limits follow-up to 30 days post-operatively. For some outcomes, 30 days may be adequate follow-up; for others, it may be advantageous to study longer term endpoints. For example, 90-day readmission rates were often notably higher than 30-day readmission rates in the MDR. Finally, as (non-sampled) claims data, the MDR captures 100% of operations. In contrast, the NSQIP-P intentionally captures only a select number of operations in order to maximize sampling efficiency.

There are also important disadvantages to the MDR, compared with the NSQIP-P. As a clinical database, NSQIP-P captures specific post-operative outcomes, including surgical site infections, need for mechanical ventilation, pneumonia, and others. The NSQIP-P is specifically designed as a quality improvement initiative, whereas the data in the MDR are captured as claims. These outcomes have been validated in the NSQIP and may not be as reliably captured in the MDR, unless a specific reimbursement code was applied to the event. As such, it is important to choose objective outcomes whenever able if the MDR is to be used; whereas, if the question of interest involves clinical endpoints captured by the NSQIP, the latter dataset may be preferable.

Additionally, NSQIP-P tracks infants under 30 days old, whereas the MDR poses a challenge in tracking infants under 1 year of age, due to the lag time in establishing them in the database as new patients with unique identifiers. Furthermore, age is documented in integer years in the MDR, whereas the NSQIP-P records age in days for neonatal patients. Given these differences, for neonatal patients, the NSQIP-P may be a preferable resource.

Finally, although both NSQIP-P and the MDR are updated on a yearly basis, NSQIP-P data are more readily accessible by the medical community, being provided at no charge to researchers and participating hospitals. In contrast, access to the MDR is free but controlled by Federal oversight and requires a more extensive application process.

This study has several strengths, such as the inclusion of surgical procedures with markedly different rates of use and occurring in different body systems. Both databases draw from sufficiently large populations so as to capture infrequent procedures, and both have demonstrated suitability as tools for health services research. Therefore, comparing the MDR and the NSQIP-P provides useful information on the scope of these databases as well as the difference between them.

There are also important limitations to consider. As described above, the MDR is a claims database which is limited in its capture of outcomes, and may not address subtler clinical findings. Similarly, the number of operations captured differs between the databases with NSQIP containing many-fold more operations than the MDR. This may occur because of the different inclusion criteria for each database. Specifically, the NSQIP systematically samples patients who undergo selected operations from participating institutions. Conversely, the MDR contains a large cohort of children insured via TRICARE, and can be interrogated using claims codes to identify all patients who have undergone a given operation over time. Finally, this study assessed only a few notable outcomes (demographics, length of stay, mortality, and readmissions), which together serve as a strong foundation for comparison of the two databases, but which cannot constitute a full validation. Further research is needed to address each of these issues.

In conclusion, the MDR was found to be comparable to the NSQIP-P in several areas including patient demographics and several clinical outcomes following five common pediatric surgical operations. Additional comparison to other standard databases with other populations will further establish the MDR as a tool for robust health services research relevant to the general United States population.