Development of machine learning models to predict gestational diabetes risk in the first half of pregnancy

With this objective in mind, several models have been developed to diagnose GDM during the early stages of gestation [20‐35]. Some of these models use simple variables, such as age, previous GDM, a first-degree relative with a family history of diabetes, multiple pregnancies, fasting plasma glucose (FPG), glycated hemoglobin (HBA_1c) and triglyceride [20]. A rapidly growing body of evidence shows that the application of machine learning (ML) to analyze data of this nature, and more general biophysical and socio-economic metrics (i.e., easily obtained from a patient history early in pregnancy) may allow a new means by which early and accurate predictions of GDM risk may be made [36]. Critically, such predictions may be able to be scaled to a population level as they do not require the taking of liquid biopsies, the administration of screening or diagnostic tests, and convey comparably little per-test cost. ML approaches have shown success in the prediction of preeclampsia [37], GDM from electronic health records [22], and pattern recognition [38]. In GDM prediction, various models have been used including Deep Neural Network (DNN) [20], Logistic Regression (LR) [21], Gradient Boosting [22], a LR and Extreme Gradient Boosting (XGBoost) [23], and Random Forest (RF) with LR [24]. A recent review [36] of ML-based models for the prediction of GDM before 24–28 weeks of pregnancy reported the viability of this approach to make predictions from general patient data, and emphasized the use of generic clinical variables. The best results of previously published models using similar input variables and GDM criterion are summarized in Table 8. Although several studies focusing on the prediction of GDM have been presented, a model that can reach high sensitivity and specificity for early prediction of GDM, and with the least number of variables, is still clinically needed. Additionally, variables that are widely available for screening examinations during pregnancy will allow a massive application of the prediction model, including low-income areas where more complex tests are not available, or may not be able to be executed in a highly standardized fashion (i.e., rigorous pre-analytical sample processing).

The main objective of our ML models is to predict the risk of developing GDM early in pregnancy in order to facilitate preventive treatment and reduce the risk of adverse maternal and fetal outcomes. As this was a retrospective study, all patients had OGTT data available for validation of the GDM diagnosis. It is worth noting that the OGTT was not used to develop the models but rather to validate the diagnosis of GDM. In the present submission we report the development of twelve different ML models, and the optimizing of their hyperparameters for the prediction of GDM, to achieve the highest classification performance, and the application of a variable selection process. Redundant data was eliminated to improve model performance.

The dataset used in this study was obtained from patients attending the Obstetrics and Fetal Medicine Unit of the Hospital Parroquial de San Bernardo, Santiago, Chile. The dataset included registries from 1,611 different pregnant patients, from 2019 to 2022. The patients included in the dataset have all the available variables/completed; patients with missing data are not included. A diagnosis of GDM was made using the IADPSG/HAPO criteria for gestational diabetes [13, 39], i.e., oral glucose tolerance test (75 g) fasting glycemia ≥ 92 mg/dl, or 2 h glycemia ≥ 153 mg/dl in the second trimester. Patients with Diabetes Mellitus that had been diagnosed before pregnancy were excluded from the dataset. Data was obtained during regular maternal visits at up to the 20th week of gestation. The third column of Table 1 shows the information on the variables and the gestational week at which the information was collected. Most of the data was obtained during the first maternal visit that happened anytime between the 4th and 20th weeks of pregnancy. We also added a histogram (Fig. 1) showing the number of patients per gestational week for the first maternal visit. As in previous work [20, 22, 24, 27, 28, 30, 32, 35], our study was retrospective and therefore the dataset was available as described. Patients with Diabetes Mellitus diagnosed before pregnancy were excluded from the dataset. The data for the input to the model of each continuous variable was normalized (by subtracting the average and dividing it by the standard deviation), e.g., age, weight, height, and Body Mass Index (BMI) at the first visit, and the first trimester fasting glucose level. The database was divided into three partitions: training set (70%), validation set (10%), and testing set (20%).

Data augmentation (DA) is a common method used in ML to improve training results [40, 41]. We generated a DA method on the training set adapted to the diagnosis of GDM by restricting the data values within physiological ranges for each input. The ranges for the creation of new data were given by a specialist in Obstetrics/Gynecology. The DA approach was used to create new patients for training the models based on the original patients, changing some input values slightly as follows: i) Age: Newly created patients must be in a range of ± 2 years compared to the original ones; ii) First Trimester Glycemia Test: New created patients must be in a range of ± 5 mg/dL only if the original patient has a result between 66 and 94 mg/dL, or over 105 mg/dL in this test; iii) Height: Newly created patients must be in a range of ± 3 cm compared to the original ones; iv) Weight: Newly created patients must be in a range of ± 5 kg compared to the original ones; and v) BMI: The BMI was adapted to the changes of height and weight in the newly created patients. A new patient should not be created if the new BMI classification was different from that of the original patient. We used the BMI classification proposed by the WHO [42].

For the experiments we also considered a limited range for the DA range of values provided by a medical specialist. The original and the limited range values are shown in Table 2. Several cases for DA were determined by increasing the number of cases in the training set to generate a total number of cases reaching values of 120%, 140%, 160%, 180% and 200%, relative to the original number of cases, which was 100%.

Twelve different ML models and their hyperparameters were optimized to achieve the highest prediction performance. Gaussian Naïve Bayes (GNB) and Bernoulli Naïve Bayes (BNB), Decision Trees (DT), Support Vector Machines (SVMs), Multi-Layer Perceptron (MLP), K-Nearest Neighbors (KNN), Logistic Regression (LR), Random Forest (RF), Extra Trees (ET) [43, 44], Balanced Random Forest (BRF) [45], Gradient Boosting (GB), implemented in Extreme Gradient Boosting (XGB) [46], and Light Gradient Boosting Machine (LGBM) [47] approaches were used. All the models were trained with the training set computing over 3,000 combinations of hyperparameters. For example, for the SVM, various types of kernels were used; for the MLP, different combinations of layers and solver were used; for the models based on Trees, various types of “criteria” were used; and for ensemble, different numbers of estimators were employed, among many other hyperparameters.

The models were implemented in Python 3.9.12 using Scikit-Learn [43], Imbalanced-Learn [45], XGBoost [46], and LGBM [47] libraries. The main hyperparameters used for each model are: GNB “var_smoothing” [43]; BNB”alpha” [44]; DT”criterion”,”max_depth”, “max_leaf_nodes”, “splitter” [43]; SVM “kernel”, “degree”, “decision_function_shape”, “C” [43]; MLP “solver”, “hidden_layer_sizes”, “activation”, “learning_rate_init”, “max_iter”, “learning_rate”, “early_stopping” [43]; KNN “algorithm”, “leaf_size”, “p”, “n_neighbors” [43]; LR”C”,”solver” [43], RF, ET and BRF “n_estimators”, “criterion” [43, 45]; XGB “n_estimators”, “eta”, “booster”, “gamma”, “max_depth” [46]; LGBM “n_estimators”, “boosting_type”, “learning_rate” [47].

Table 3 shows all the hyperparameters that were used in the Grid Search, and the range of values analyzed.

The results obtained with the combination of hyperparameters values were assessed in a fivefold cross validation (CV) [48] using data from the training set and performing a grid search on the hyperparameter values. Grid search allows finding near optimal values for the hyperparameters via multiple evaluations of various combinations for each one. An input selection [49] was performed to select the best variables to be used in the prediction task to improve the model results and reduce input redundant variables to each model. The input variable selection was performed using 3 methods: F-test of ANOVA (Analysis of Variance), Chi-Square Test, and Mutual Information (also known as Information Gain) [43]. The models were trained, evaluated, and tested with various combinations of input variables selected by these 3 methods. After adjustment with the training set, the top 15% of the models ranked with the highest area under the ROC curve, AUCROC [50], were selected and assessed on the validation set. Models with the best results on the validation set were selected to obtain a good balance between high Sensitivity and good Specificity [50]. Finally, the selected models were assessed in the test set as a measure of generalization performance. The test set was not used in any previous step involving training or selection of the best models. Models were also trained using DA on the training and validation sets, but no DA was performed on the test set. The best results were chosen using sensitivity and specificity as the main metrics of performance. The accuracy, sensitivity, specificity and recall macro are measured with a specific decision threshold, calculated by using the validation dataset to determine this threshold. The ROC curve is created based on the different decision thresholds that modify sensitivity, also known as True Positive Rate (TPR), as a function of the false positive (FP). The formulas are the following: Accuracy = (TP + TN)/(TP + FP + TN + FN), Sensitivity = TP/(TP + FN), Specificity = TN/(TN + FP), Recall Macro = (Sensitivity + Specificity)/2.

A total of 1,611 pregnant women were included in this study. The database was partitioned into 1,127 cases for the training set, 161 in the validation set, and 323 (39 positive of GDM) were part of the test set. The prevalence of GDM was 14.21% (229/1,611). The input variables to the models are described in Table 1.

The most relevant 12 variables selected by the 3 methods: F-Test ANOVA, Chi-Square, and Mutual Information, are displayed on Table 4.

We selected the most important variables (features) in the dataset by removing irrelevant or redundant variables. This allows us to have a small number of variables which is useful for a clinical application. The methods used for this purpose are commonly employed in ML (F-test of ANOVA, Chi-Square Test, and Mutual Information). This variable selection also avoids the overfitting problem and achieves improved performance compared to that of using all the features [49]. For example, variables such as Pregnancy Type or Stillbirth are not selected by the variable selection methods, but may decrease the performance of models such as Multi-Layer Perceptron. Additionally, one of the models used to select variables was the BRF (see Table 4). The ranking obtained with a nonlinear model, BRF, is similar to those obtained with statistical methods, confirming that these are the relevant variables.

Table 5 shows the model type, number of input variables, whether or not DA was used, with “w/o DA” meaning that Data Augmentation is not used in this model, “DA LE”, meaning Data Augmentation w/Limited Expert range, “DA EO”, meaning Data Augmentation w/Expert original range, and the results of the following: Accuracy, Sensitivity, Specificity, Recall Macro, AUCROC, False Positives (FP), False Negatives (FN), and FP + FN. Table 5 show the top 4 models for each sensitivity level with the model that has the highest AUCROC in bold type, for models with up to 12 variables. All these metrics were computed for each model in the test set. As mentioned in the Methods section, the test set was only used to test the generalization capacity of the models. The test set was not used to train or to select the hyperparameters of the models. On Table 5 we show the results of models that reached a sensitivity above 0.9231 in the test set (model numbers 1 to 16), while model numbers 17 to 36 show the results of models with sensitivity above 0.7949 but below 0.9231 in the test set. Models with high sensitivity allow minimizing FN when screening patients. Sensitivity is important since the main goal is to prevent the serious consequences of GDM that may occur in mothers and babies even several years after pregnancy. Our method allows identifying many possible models for various levels of sensitivity and specificity. For example, model numbers 29–32 on Table 5 all have a high sensitivity of 0.82 and a specificity in the range 0.72–0.74, with accuracy between 0.73–0.75; AUCROC of 0.81; and Recall Macro between 0.77 and 0.78. A model could be selected from these ranges to have a good compromise between low numbers of FN and FP as is shown in the last column of Table 5.

Another possible choice of model could be model 17 (Table 5) with sensitivity of 0.89 that requires just 5 variables (1TFG, Age, BMI, Maternal Weight, and Gravidity). This model reaches an accuracy of 0.65, a specificity of 0.62, Recall Macro of 0.76, and AUCROC of 0.82. Models 17–20 reach the same sensitivity of 0.89 with small changes in accuracy, specificity, Recall Macro and AUCROC. The best models for sensitivity 0.89 are all MLPs. It can be seen on Table 5, and on Fig. 2 that there are several choices of models for predicting various levels of sensitivity, with a trade-off on specificity.

Figure 2 shows two different views of the same surface plotting the model results for various values of the total number of errors (FP + FN), True Positives, and number of input variables for each model. Several choices of the models are available for reaching high sensitivity (low FN), and high specificity (low FP) with a small number of input variables. On the surface plotted in Fig. 2 the red dots represent the best models in bold type from Table 5 with sensitivity above 0.92 (model numbers 1, 5, 9, and 13), and the yellow dots represent the best models from Table 5 with sensitivity above 0.79 but below 0.92 (model numbers 17, 21, 25, 29, and 33).

Figure 3 shows the ROC curves for each of the 9 best models with a fixed sensitivity starting at sensitivity of 1 (a), up to a sensitivity 0.79 (d). These best models for each sensitivity level appear in bold type in Table 5. Figure 3(a) shows the ROC curves for the best models with sensitivities of 1, 0.9744 and 0.9487. Figure 3(b) shows the ROC curves of the best models with sensitivities of 0.9231, 0.8974 and 0.8718. Figure 3(c) shows the ROC curves of the best models with sensitivities of 0.84, 0.82, and 0.79. Finally, Fig. 3(d) shows the ROC curves for model number 29 in Table 5 with the best recall macro (gray), and a comparison with the same model having DA (cyan), and the same model with a lower number of variables (pink). This model, number 29, has the lowest number of FP + FN.

Table 6 shows the best models for different sensitivity levels, with more than 12 input variables. It can be observed that models 38, 42, 43 and 45 reached a slightly better FP + FN than our best selected models shown on Table 5. Nevertheless, the number of required input variables is more than doubled. For example, model 25 requires 6 input variables while model 43 requires 15 input variables for the same sensitivity. A much larger number of input variables would be more difficult to implement in clinical practice.

In the clinical context, one of the main focuses of the GDM specialists is the balance between sensitivity and specificity. High sensitivity avoids errors in detecting patients with the illness (low FN), while high specificity decreases the FP number. Tables 5 and 6 show a trade-off between sensitivity and specificity in our results, yielding a high, but not maximum, AUCROC. The models are ordered on Table 5, first by a sensitivity level, and then other selected metrics, such as specificity and AUCROC. The main metrics used in the final selection of our models were sensitivity and specificity. We also used a Balanced Random Forest (BRF) model that had good performance on imbalanced datasets, that achieved good performance, although not better than that of the models presented on Tables 5 and 6.

On Table S1 (Additional file 1), we show the Mean AUCROC, 95% confidence interval, and standard deviation (STD) of the different models presented on Tables 5 and 6, calculated by using ten different seeds for the initialization of the models.

Table 7 presents performance comparisons among the models with Data Augmentation (w/DA), and without Data Augmentation (w/o DA). The comparisons include the same models.

In the present study, the best performing models (i.e., SVM 12; Table 5) using data collected prior to 20 weeks of gestation had a sensitivity of 82% and specificity of 74%, coming quite close to that of the two-step protocol widely used in the United States at later gestations. In our study, we developed a group of 12 models for early diagnosis of GDM, with data that are commonly acquired at the early stages of pregnancy during prenatal care visits to gynecologists/obstetricians. The ease of data collection should facilitate the future of these models in clinical practice. Another important consideration is that sensitivity is crucial since the main goal is to prevent serious consequences of GDM for mothers and babies, many of which will impact them for several years after pregnancy. In cases of lower specificity (higher FP), additional tests could be used to improve diagnosis, although this would come with additional cost, inconvenience, and risk. Also, in many cases the main treatment involves diet and exercise which are not harmful. From our variable selection methods, the most important variables for GDM diagnosis were related to glucose metabolism (first trimester fasting glycemia), physical status (weight and BMI), age, and hypertension. The use of DA had a positive effect in most models, improving specificity up to 51.43% and AUCROC up to 3.70% with the same sensitivity. The best model results, for each sensitivity level, was reached in 7/9 cases with DA and in 2/9 with no DA.

The limited public availability of datasets for informing previously published work makes direct comparisons of model performance difficult [20‐35]. Nevertheless, a general assessment can be undertaken by comparing the result ranges from different metrics obtained on various datasets. However, there are important aspects, such as characteristics of the population, and diagnostic criteria, that vary between countries/regions in the different studies analyzed, and therefore, these aspects should be considered when comparing the different datasets. Table 8 shows a comparison of model results from the present study against those of recent studies assessing ML-driven diagnosis of GDM risk. In general, our models performed better in AUCROC than comparable models generated with similar input variables and the same or similar GDM diagnosis criteria [20‐22, 25‐28, 30, 31]. As explained previously, sensitivity is important due to the possible adverse effects of GDM on the mother and baby later in life. Other models [20, 22‐24, 29, 32‐35] that required additional complex data are not listed in Table 8. In some cases, such as those presented in the meta-analysis [52], more complex variables were employed on the models such as ultrasound screening data, or biochemical data of liver/renal/coagulation function at the prenatal visit. For example, a comparison between our model 33 SVM 7 Variables DA LE (Table 8), and the work of Wu and colleagues [20] (Table 8) yielded a higher sensitivity (13.55%), and a higher specificity (6.14%). Our model 17 MLP 5 Variables DA EO (Table 8) vs. Pintaudi et al. [28] (Table 8), reached a similar sensitivity but had an improved specificity (56.70%). A different criterion for GDM diagnosis was by Kumar and coworkers [31] (WHO, 1999), with which GDM was diagnosed if fasting OGTT ≥ 126 mg/dL and/or 140 mg/dL in a 2 h OGTT. Another model was implemented by them [31] using the same GDM diagnosis criterion as ours, IADPSG/HAPO, reaching an AUCROC of 0.73, with a fivefold stratified CV. ML models have also been applied for predicting Diabetes Mellitus [53].

Table 9 shows a list of the input variables used in each of the best models selected, including those used for comparison, and those developed and selected by the authors. It can be observed that some of the best solutions require only five input variables. When choosing these models for a clinical application, only 5–7 variables will need to be measured in each patient to diagnose GDM with these models. This will facilitate the possible application of these models in clinical practice. Developing accurate ML models for predicting GDM is an important step towards implementing early prediction and treatment strategies for patients. The next step should be to prospectively apply them in a clinical setting to validate and evaluate their performance.

In the present study, twelve ML models and their hyperparameters were optimized for early (20 weeks of gestation or earlier) GDM with high sensitivity, specificity, AUCROC, and Recall Macro. The models could predict GDM with a good degree of accuracy before 20 weeks of gestation, and with variables that are widely available from screening examinations The variables required by most of the models were age, weight, BMI, and FPG which is consistent with previous publications [20‐22, 25‐28, 30, 31]. Variable selection was performed by three methods and results show that several models reached good performance with as few as 5–7 input variables, while other models required more, including up to 12 variables. Choosing models with high GDM prediction performance, a low number of input variables, and widely available variables will facilitate the possible application of these models in low income settings. Although patient data from previous publications are often not available, comparing the results obtained for various metrics show that, in general, our models performed favorably in comparison with the existing literature. In conclusion, our data demonstrate that ML-analysis of patient data sets from early pregnancy may serve as a cost-effective and efficacious means of detecting GDM risk early in pregnancy.

We described all steps required to implement, train and test the models. In particular, we used a test partition that is different from the training and validation partitions, to improve the generalization capacity of the models. Many of the previous reported work did not state explicitly using an independent partition for testing [20, 21, 24‐32, 35]. This study provides a valuable contribution by utilizing and comparing a broad range of ML models (12), which differs from many other studies that often use only one type of model, such as Logistic Regression. Additionally, various metrics have been employed to compare the performance of each model, including a wide range of variables that could potentially be selected for clinical implementation. This approach allows for a more comprehensive assessment of the potential utility of different ML models in predicting GDM and facilitates the identification of the most effective models for future clinical implementation.

As with any study of this nature, the findings need to be assessed in light of the ground-truth data set from which they were drawn. For the present study, we used a single center population drawn from a socio-economically vulnerable medical center in Santiago, Chile. Accordingly, a cautious approach should be taken in extrapolating these findings to a wider socio-economic grouping, and to the maternal situation in other regions. The strengths of this study include a well-characterized pregnancy cohort and robust data collection. Future iterations of this work will involve the cross-population analysis of GDM risk and the comparison of predictive outcomes from different populations to assess the broad applicability of model performance. While the variables used in the different ML models show promising predictive capacity for GDM, the addition of other inputs such as biomarkers could potentially further improve their performance. As such, future studies may consider incorporating additional data sources to enhance the accuracy of GDM prediction models.

These findings are of particular importance given the increasing prevalence of GDM in the maternal population and the significant impacts (both patient well-being and financial) that derive from poorly controlled glucose levels in pregnancy. For example, recent modeling from the United States suggests that, in 2014, the short-term costs of GDM were $1.8 billion [54]. The cost of treatment for T2DM is routinely around $3,500 per year [55]. Given estimates that one in six pregnancies are impacted by GDM, even a small improvement in outcomes deriving from early risk identification and timely intervention would yield profound public health benefits and health system cost savings.