Application of Machine Learning Models to Evaluate Hypoglycemia Risk in Type 2 Diabetes

The data source for this study was Optum Clinformatics Data Mart (CDM) (Eden Prairie, MN, USA). Optum’s CDM is a closed system of de-identified health claims data that includes over 13 million lives annually, and contains patients’ medical, prescription drug, laboratory, and eligibility information beginning in year 2000. The data comes from a large, national US health insurer and is certified as de-identified. For this study, a data extract between 1 January 2014 and 30 September 2017 was received from Optum.

This extract was used to construct a population of patients with T2D following a retrospective cohort study design. Patients were eligible for inclusion if they met the following criteria: at least one medical claim with a diagnosis of T2D (International Classification of Diseases, 9th revision (ICD-9) codes 250.x0, 250.x2; International Classification of Diseases, 10th revision (ICD-10) code E11) anytime during patients’ available claims history, at least one prescription claim for a qualifying antidiabetic medication (Generic Product Identifier (GPI) 27) anytime during patients’ available claims history, at least 18 years old as of the index date, and continuously enrolled with a medical and pharmacy benefit during the 12-month pre-index and 12-month post-index periods. The index date was the date of first prescription claim for an antidiabetic medication.

Six unique outcomes were chosen for modeling, all captured in a 12-month post-index period; these included hypoglycemic events, antidiabetic medication class persistence, HbA1c target attainment, HbA1c change from baseline, T2D-related inpatient admissions, and T2D-related medical costs.

Hypoglycemia was defined by a combination of relevant ICD-9 and ICD-10 codes for hypoglycemia (Table A1 in the supplementary material) and available blood glucose measurements as follows: if a patient had at least one medical claim with any of the ICD-9 or ICD-10 codes associated with hypoglycemia as listed in Table A1 during the post-index period or if a patient had a least one blood glucose measurement of 70 mg/dL or below, we inferred that the patient experienced a hypoglycemic event. The outcome was coded as a binary variable (i.e., 1 if at least one hypoglycemic event in the post-index period; 0 otherwise). Definitions for remaining study outcomes are detailed in the supplementary material.

A unified set of covariates was built for all outcomes using patients’ available pre-index medical, pharmacy, and laboratory claims information. Covariates were not necessarily diabetes-related; rather, patients’ complete claims history was extracted leaving variable selection to the models. Furthermore, in order to extract a meaningful signal from each variable, and to ensure variables were defined consistently across varying time periods, claims codes were aggregated together using a variety of widely used code sets. Covariate structures are briefly described below:

A machine learning analytic platform, Reverse Engineering and Forward Simulation (REFS™) (GNS Healthcare, Cambridge, MA, USA), was used for all modeling. Briefly, each REFS model was an ensemble consisting of 128 generalized linear models, constructed using Markov chain Monte Carlo sampling of the full Bayesian posterior distribution of models, given the available data—i.e., P(model|data) [12‐16]. In high-dimensional data sets, attempts to identify a single best model will inevitably lead to overfitting; thus, our approach identifies an ensemble of models, each scored by both its goodness-of-fit to the observed data and its complexity. This approach allows incorporation of previous knowledge regarding the different types of data (e.g., medical claims, pharmacy claims, laboratory data, etc.), including the expected relative contribution of each type, into the final models. Therefore, examination of large numbers of variables with low signal (i.e., medications and diagnoses) is possible without overwhelming the signal of the fewer—although potentially more directly informative—variables (i.e., demographics, indexing antidiabetic class, etc.).

The 128 constituent models within the ensemble looked at combinations of the available parameters, including linear additive and quadratic terms, and up to second-order interactions to accommodate non-linear effects against outcomes. To prevent overfitting, the complexity (i.e., the number of terms) of each model in the ensemble was penalized by specifying a maximum entropy prior over the number of unique predictors selected per given variable class.

From the ensemble for each study outcome, predictors can be ranked and evaluated by their relative selection frequency (proportion of models in the ensemble in which the variable was selected) and distributions of effect estimates. A high selection frequency for a given predictor represents an increased probability of a true predictive association with the outcome [14].

REFS models were trained on 80% of the final study cohorts sampled at random, while the remaining 20% were reserved for model validation and performance assessment. Out-of-sample validation provides a more realistic estimate of true model performance and supports generalized use of models in new patient samples.

Model prediction performance was measured via the area under the receiver operating characteristic curve (AUC) metric for all outcomes except T2D-related total medical costs. The AUC metric illustrates the performance of a binary classifier system and is used to examine the overall specificity and sensitivity of the classification approach, i.e., the trade-off between false positive and negative predictions of a model. AUC values range between 0 and 1 where greater AUC values indicate better model performance (an AUC of 0.5 is associated with a random predictor). For the T2D-related total medical costs outcome, which is continuous, Pearson R² (predicted vs. observed costs) was used to assess performance.

In addition, credible intervals (CIs) for performance metrics were calculated for each REFS ensemble by generating a distribution of performance metrics across all 128 models in each ensemble. This is a unique characteristic of ensemble modeling—granting insight into model consensus across the ensemble—as well as suggesting best- and worst-case scenarios for potential deployment of these models in new patient samples.

Finally, the utility of the hypoglycemia ensemble was estimated by evaluating the concentration of costs and visits, as well as other study outcomes within 20 equally distributed bins of predicted hypoglycemic event risk (henceforth referred to as “ventiles”), as applied in similar contexts [17]. More specifically, each ventile contained 5% of patients in the study cohort (N = 22,672) and patients were assigned to a ventile according to his or her predicted probability of hypoglycemia. Patients with the lowest probability of hypoglycemia—in the bottom 5th percentile—were assigned to the first ventile, whereas patients with the highest probability of hypoglycemia were assigned to the 20th ventile. The proportions of visits, costs, or other study outcomes within each ventile out of the total number of visits, costs, or other study outcomes were calculated as a measure of concentration of risk.

The Optum CDM data used in this study was compliant with the Health Insurance Portability and Accountability Act. Fully anonymized retrospective data were obtained from Optum via license agreement, and no other data were used in this study. As such, this study was deemed exempt from ethical approval.