Development and validation of prediction models for gestational diabetes treatment modality using supervised machine learning: a population-based cohort study

The study population was drawn from the membership of Kaiser Permanente Northern California (KPNC), an integrated health care delivery system serving 4.5 million members. The KPNC membership accounts for approximately 30% of the underlying population and is socio-demographically representative of the population residing in the geographic areas served [11, 12]. The integrated information system permits quantifying predictors and outcomes across the continuum of pregnancy. Individuals with GDM are identified by searching the KPNC Pregnancy Glucose Tolerance and GDM Registry, which is an active surveillance registry that downloads laboratory data to determine screening and diagnosis for GDM, where preexisting type 1 or 2 diabetes is automatically excluded. Specifically, pregnant individuals at KPNC receive universal screening (98%) for GDM with the 50-g, 1-h glucose challenge test (GCT) at 24–28 weeks’ gestation [1]. If the screening test is abnormal, a diagnostic 100-g, 3-h oral glucose tolerance test (OGTT) is performed after an 8–12-h fast. GDM is ascertained by meeting any of the following criteria: (1) ≥ 2 OGTT plasma glucose values meeting or exceeding the Carpenter-Coustan thresholds: 1-h 180 mg/dL, 2-h 155 mg/dL, and 3-h 140 mg/dL; or (2) 1-h GCT ≥ 180 mg/dL and a fasting glucose ≥ 95 mg/dL performed alone or during the OGTT [13, 14]. Plasma glucose measurements were performed using the hexokinase method at the KPNC regional laboratory, which participated in the College of American Pathologists’ accreditation and monitoring program [15]. This data-only project was approved by the KPNC Institutional Review Board, which waived the requirement for informed consent from participants.

Among 405,557 pregnancies with a gestational age at delivery < 24 weeks’ gestation delivered at 21 KPNC hospitals from January 1, 2007, to December 31, 2017, we excluded 375,041 (92.5%) individuals without GDM. Among 30,516 GDM pregnancies, we further excluded individuals with GDM diagnosed before the universal GDM screening (n = 42), deriving an analytical sample of 30,474 GDM-complicated pregnancies. We further derived a discovery set containing 27,240 GDM-complicated pregnancies from 2007 to 2016 and a temporal/future validation set of 3234 GDM-complicated pregnancies in 2017 (Fig. 1).

Individuals diagnosed with GDM received universal referral to the KPNC Regional Perinatal Service Center for the supplemental care program beyond their standard of prenatal care. MNT was the first-line therapy. If glycemic control targets were not achieved with MNT alone, pharmacologic treatment was initiated. Based on counseling regarding risks and benefits of antidiabetic oral agents versus insulin, pharmacologic treatment was chosen via a patient-physician shared decision-making model: (1) with antidiabetic oral agents such as glyburide and metformin being added to MNT and if optimal glycemic control continued to fail, oral medication was escalated to insulin therapy, and (2) or with insulin therapy initiated directly beyond MNT (an additional table shows this in more detail [see Additional file 1]). We searched the pharmacy information management database for prescriptions for oral agents (glyburide 97.9%, metformin or other) and insulin after GDM diagnosis. Treatment modality was grouped as MNT only and pharmacologic treatment (oral agents and/or insulin) beyond MNT. Notably, despite an overall large sample size, we grouped oral agents (32.6% of the entire population) and insulin (6.2%) into pharmacologic treatment due to insufficient power to predict insulin separately as an outcome.

Based on risk factors associated with GDM treatment modality and input from clinicians, we selected 176 (64 continuous and 112 categorical) sociodemographic, behavioral, and clinical candidate predictors obtained from electronic health records for model development. Candidate predictors were divided into four levels based on availability at varied stages of pregnancy (an additional table shows this in more detail [see Additional file 2]): Level 1 predictors (n = 68) were available at the initiation of pregnancy and dated back to 1 year prior to the index pregnancy; level 2 predictors (n = 26) were measured from the last menstrual period to before GDM diagnosis; level 3 predictors (n = 12) were available at the time of GDM diagnosis; and level 4 (n = 70) included self-monitoring of blood glucose (SMBG) levels, as the primary measure of glycemic control during pregnancy as recommended by the American Diabetes Association [5], measured the first week after the GDM diagnosis. All predictors, levels 1–4, were measured prior to the outcome of interest (i.e., final line of GDM treatment). Pregnant individuals with GDM in our study population had on average, 11.8 weeks (standard deviation: 6.6 weeks), of SMBG measurements between GDM diagnosis and delivery. We included data 1 week after GDM diagnosis to allow earlier prediction since it takes on average 5.6 weeks between GDM diagnosis and the optimal treatment is offered. Of note, individuals with GDM were universally offered enrollment to a supplemental GDM care program managed by nurses and dietitians via telemedicine from the KPNC Regional Perinatal Service Center [16]. All individuals with GDM were instructed to self-monitor and record glucose measurements four times per day: fasting before breakfast and 1 h after the start of each meal. Measurements of SMBG were then reported to the nurses or registered dieticians during weekly telephone counseling calls from enrollment until delivery and data were recorded in the Patient Reported Capillary Glucose Clinical Database.

Across the four timings, the tenfold cross-validated AUCs in the discovery and validation sets were overall lowest with level 1 predictors from 1-year preconception to LMP, regardless of prediction methods (Table 2). The models adding level 2 predictors performed slightly better than those at level 1 (an additional table shows this in more detail [see Additional file 5]). For models at levels 1–2, the complex SL (an additional table shows this in more detail [see Additional file 6]) outperformed the simple SL (an additional table shows this in more detail [see Additional file 7]), followed by LASSO regression and CART (tenfold cross-validated AUCs in the discovery set: 0.761, 0.688, 0.685, 0.618, respectively; Table 2). Adding level 3 predictors at GDM diagnosis increased the AUCs by approximately 0.10 across all models in both the discovery and validation sets; the highest AUC (95% CI) was observed for the complex SL [tenfold cross-validated AUC in the discovery set: 0.869 (0.865–0.873); validation set: 0.754 (0.739–0.772)]. The addition of level 4 predictors, 1 week after GDM diagnosis, further increased AUCs by approximately 0.05–0.07 across models, with the highest performance by the complex SL [discovery: 0.934 (0.931–0.936); validation: 0.815 (0.800–0.829)].

For level 1 predictors at the initiation of pregnancy (Fig. 2A; an additional figure shows this in more detail [see Additional file 8]), the top three contributors to the prediction based on variable importance were the same across prediction methods: pre-pregnancy obesity, prediabetes before pregnancy, and history of GDM. For predictors at levels 1–2 (Fig. 2B; an additional figure shows this in more detail [see Additional file 8]), the top four features in CART and simple and complex SL included the top three at level 1 with the addition of glucose levels at GCT for GDM screening (≥ 200 mg/dL).

At the timing of GDM diagnosis with the addition of level 3 predictors, OGTT fasting glucose value (per 1 mg/dL increase), gestational week at GDM diagnosis (continuous), and GDM diagnosis by Carpenter-Coustan criteria (versus by fasting hyperglycemia) were consistently the top three features across prediction methods (Fig. 2C; an additional figure shows this in more detail [see Additional file 8]). Adding level 4 predictors (Fig. 2D; an additional figure shows this in more detail [see Additional file 8]), the top four consistently selected by LASSO regression and the simple and complex SL were self-monitored glycemic control status at fasting, gestational week at GDM diagnosis (continuous), OGTT fasting glucose value (per 1 mg/dL increase), and number of fasting self-monitored blood glucose measurements. In the complex SL, the difference between importance measures of the fourth and fifth (i.e., history of GDM) top features was substantial (approximately by 40%); thus, the fifth-ranked and below predictors were not included as top features for the development of simpler models (see below).

We constructed a series of simpler logistic regression models with a minimum set, as opposed to the full set, of predictors at each timing to balance the model predicative performance and feasibility and easiness of clinical implementation and uptake. The main predictors of the simpler models were selected via the complex SL, and we further included interaction terms selected from the optimal stepwise logistic regression (an additional table shows this in more detail [see Additional file 9]). At pregnancy initiation, the simpler model using level 1 predictors identified history of GDM, pre-pregnancy obesity, and prediabetes before pregnancy [tenfold cross-validated AUC in the discovery set: 0.632, 95% CI (0.623–0.640); validation set AUC: 0.609 (0.587–0.632); Table 3]. The simpler model at levels 1–2 included history of GDM, pre-pregnancy obesity, glucose levels at GCT for GDM screening (≥ 200 mg/dL), and prediabetes before pregnancy, in addition to three pairwise interactions between the first three predictors, with similar AUC (95% CI) to that at level 1 [discovery set AUC: 0.648 (0.640–0.656); validation set AUC: 0.621 (0.599–0.643)]. At GDM diagnosis, the model using levels 1–3 predictors included OGTT fasting glucose value, gestational week at GDM diagnosis, and GDM diagnosis by Carpenter-Coustan criteria [discovery set AUC: 0.770 (0.764–0.775); validation set AUC: 0.746 (0.730–0.763)]. One week after GDM diagnosis, the simpler model included gestational week at GDM diagnosis, OGTT fasting glucose value, self-monitored glycemic control status at fasting, number of fasting self-monitored blood glucose measurements, and an interaction term between last two variables [discovery set AUC: 0.825 (0.820–0.830); validation set AUC: 0.798 (0.783–0.813)]. The simpler logistic regression models used for prediction is shown in Table 4.

We further evaluated calibration performance of simpler models at varied levels in the temporal/future validation set, which indicated a slight difference between the predicted risk and the estimated true probability (integrated calibration index: 0.073, 0.074, 0.072, and 0.038 at each level, respectively; Table 3). In the pre-calibrated plot of the simpler model using level 1–4 predictors (Fig. 3A), the predicted probability was slightly underestimating the estimated true probability. After the isotonic regression calibration (Fig. 3B), the cross-validated AUC in the validation set increased slightly (0.802, 95% CI: 0.786–0.818). The performance of calibrated simpler models was comparable to that of the complex SL on the validation set (Fig. 4).

Despite studies that identified independent risk factors for GDM treatment (mostly insulin versus MNT alone) [21‐27], studies focusing on the development of risk prediction models are limited, which further suffered from methodological limitations and collectively limited the generalizability of developed models. The low sensitivity (36%) of the model developed by Souza et al. may fail to adequately identify the group of patients requiring medication [28]. Barnes et al. developed a prediction model that yielded sensitivity of 86–93% for insulin therapy with the presence of 6–7 predictors [29]; however, given the wide span of the study period (1992–2015), the impact of different diagnostic criteria applied to future clinical practice is unknown [29]. Among other studies, model validation was lacking [28‐32]. Further, for the two models which failed to predict insulin versus MNT alone [29, 30], the predictive ability may have been confined due to limited clinical variables (mostly glucose levels at diagnosis) and missing data on other potentially important socio-demographic, behavioral, and medical factors [22‐24, 27].

In our study, the prediction methods employed were not able to achieve high discriminative performance using variables prior to pregnancy and before GDM diagnosis, suggesting the difficulty of assessing the risk of receiving pharmacologic treatment beyond MNT prior to diagnosis. Although identified as significant risk factors for GDM by previous studies [24, 28, 29] and our study, high pre-pregnancy BMI, prediabetes, and history of GDM did not present high predictive AUC results (best AUC performance: 0.634). Similarly, Pertot and colleagues reported the lack of predictive power for insulin therapy for GDM using maternal characteristics including race/ethnicity, BMI, family history of diabetes, hemoglobin A1c, and glucose levels [31]. Instead, the addition of week of gestational age at the diagnosis of GDM and glucose values at the diagnostic OGTT defined by the Carpenter-Coustan thresholds provide acceptably high AUC (0.752) using the calibrated simpler logistic regression model. Glycemic control measures during 1 week after diagnosis further increased the predictability of optimal treatment with a higher AUC (0.802); both prediction estimates were higher than those in previous studies with AUC around 0.700 [29, 30].

Physicians could use prediction models developed from GDM diagnosis and 1 week after GDM diagnosis to inform the patient-physician shared decisions regarding GDM treatment modality by balancing the 1-week difference in the timing of prediction (i.e., at the time of GDM diagnosis versus 1-week post diagnosis) against the slightly different predictive performance (difference in AUC: 0.05). The prediction models generated at the time of GDM diagnosis and 1 week after could be used to evaluate the likelihood of an eventual need of pharmacologic treatment. These tools could help facilitate the physician–patient conversation about the potential benefit, risk, and concerns of initiating pharmacologic treatment. The risk prediction models can play an essential role to improve efficiency of GDM care and management through the physician–patient joint discussion and decision making regarding GDM treatment modality. For patients requiring eventual pharmacologic treatment, reducing the waiting time between the first-line MNT and the optimal or last-line pharmacologic treatment (mean 5.6 weeks as observed herein within a window of 12–16 weeks for potential treatment) may result in a more effective intervention.

Our study has several notable strengths. To our best knowledge, this is the largest population-based study to date which developed and validated clinically oriented risk prediction models for GDM treatment modalities in a multi-racial/ethnic population. This is a sizeable increase compared to previous studies, with sample sizes ranging from 294 to 3317 [28‐32]. We developed predictive models using supervised machine learning algorithms based on real-world data available in an integrated clinical setting at KPNC. The predictive models could be programmed into electronic health records of a health care delivery system to allow for automated risk stratification. Universal screening and diagnostic criteria for GDM applied uniformly throughout the study period (2007–2017) minimized misclassification bias. Further, pregnant individuals with GDM received universal referral to the KPNC Regional Perinatal Center for the supplemental care program beyond their standard of care. Both procedures minimized clinical practice variations, which was a major methodological limitation in previous studies [23, 24, 29]. Importantly, we performed rigorous tenfold cross-validation in the discovery and temporal/future validation sets to minimize the impact of data overfitting and bias selection.

Some potential limitations of our study merit discussion. The oracle properties of SL may only apply to the best algorithm within its selected library. Though the complex SL had a wide variety of adaptive and smooth learners, it is possible that there could be algorithms that could perform better outside of the current selection. Further, practical barriers may complicate the implementation of our predictive models in horizontally integrated or non-integrated health care systems, where data on glycemic control via self-monitoring of blood glucose collected following GDM diagnosis may not be readily available. Nonetheless, our calibrated simpler model using predictors at levels 1–3 (from 1-year prior to pregnancy to the time of GDM diagnosis; validation set AUC: 0.752) provided a slightly lower but still acceptably high AUC compared to prediction at levels 1–4 (up to 1 week post GDM diagnosis: validation set AUC: 0.802). Finally, our findings need further validation in populations from other health care delivery systems.