An evaluation of cervical maturity for Chinese women with labor induction by machine learning and ultrasound images

Data of pregnant women who were examined and had their delivery in Peking University Third Hospital between December 2019and -January 2021 were collected. The study population was Chinese women.

Patients with cervical Bishop score of > 6, patients aged ≤ 18 years or ≥ 50 years, and patients who did not agree to participate in this study either by themselves or because of their families were excluded. A flow diagram for the population selection is shown in Fig. 1.

For each puerperal woman, we collected data by the following parameters: age, primiparity/multiparity, cervical length, angle, induced labor time (ILT), measurement time (MT), the time from measurement to induced labor (MTLILT), Bishop score, method of induced labor, and the time from induced labor to labor (ILTLT). From ultrasound scanning, cervical length and angle were determined. Ultrasound images were obtained by standardized transvaginal ultrasound with a frequency of 5.0–9.5 MHz. The cervical length was measured in centimeter. The angle was measured in radian. Angle refers to the included angle of the uterine wall at the cervical opening, which is shown by the two red lines in Fig. 2.

Owing to the lack of data and inconsistent format of the original data, data cleaning and preprocessing are needed. After eliminating the missing data, the data of 101 puerperal women were available for evaluation. The subjects of our study were all Chinese Han pregnant women. All pregnant women had successful labor. The proportion of vaginal delivery was 87.13% (88 women), the proportion of caesarean delivery was 12.87% (13 women), and the proportion of multiparity was 20.79%. Three women delivered with caesarean delivery because cervical dilation arrested for 6 h at 4 cm, and other 10 caesarean deliveries because of active phase arrest. Some basic biological characteristics of 101 pregnant women participating in the research are shown in Table 1.

Bishop scoring method was used as the control group of this study. To evaluate the prediction ability of the control group and make the traditional Bishop evaluation method comparable with the machine learning method proposed by us, we processed the data of the control group as follows (taking a group with Bishop score of 6 as an example): The root mean square error (RMSE) was calculated from the real value of the time from induced labor to labor for each patient in this group and then the average value was taken. Then, in this group, a total of 31 mean square error values can be obtained. The groups with Bishop scores of 0, 1, 2, 3, 4, 6, and 7 could generate a total of 101 RMSE values similar to the procedure mentioned above. This processing method, which takes the mean value from induced labor to labor of each Bishop score group as the predicted value of each group, is the most fair data processing method of the control group. In addition, we also conducted experiments to fit bishop score and the time from induction of labor to labor with linear regression model.

We finally used cervical length, Bishop score, angle, age, induced labor time (ILT), measurement time (MT), the time from measurement to induced labor (MTILT), method of induced labor, and primiparity/multiparity as the input of three machine learning algorithms in the experimental group. The output of machine learning algorithm is the predicted time from induced labor to labor. The explanation of each feature is described in Table 2. Among them, the relationship between the three time variables is shown in Fig. 3. In Fig. 3, the three variables below in green are the input variables of the model, while the one above in yellow is the result that the model aims to predict, which is the output variable of the model. The induced labor time and measurement time is expressed in weeks, and the time from measurement to induced labor is expressed in days.

Method of induced labor and primiparity/multiparity are category features, which cannot be directly used as the input of the machine learning model. These two features need to be processed further. The categorical features are not numerical features but discrete sets, such as method of induced labor, that include misoprostol, oxytocin, amniotomy, Propess (PGE2), or none. When dealing with the two category features of primiparity/multiparity and method of induced labor, we used one-hot coding method to convert them into numerical characteristics. The specific process is to use two values to indicate whether the puerperal woman had a primiparity or multiparity delivery. If the value is 10, it means primiparity, and if the value is 01, it means multiparity. Five numerical values were used to represent the methods of induced labor of pregnant women. We selected three real pregnant women in the data set and showed the one-hot coding of their method of induced labor as presented in Table 3. The first Puerperal woman used one method to induce labor: oxytocin. The second Puerperal woman was induced by two methods: oxytocin and miso. The third woman was induced by three methods: misoprostol, oxytocin and amniotomy.

Ensemble learning is considered to be the most advanced solution to solve machine learning problems (Fig. 4). In recent years, a large number of ensemble learning models have been created and that showed strong learning performance [24]. By combining multiple learners, ensemble learning can usually obtain significantly superior generalization performance than a single learner.

Therefore, we used the three ensemble learning models, that are commonly used and perform well in most cases to predict the cervical maturity. They are XGBoost (eXtreme Gradient Boosting) [25], CatBoost (an implementation of Gradient Boosted Decision Trees) [26], and Random forest (RF) [27]. Among them, XGBoost and CatBoost models are serialization methods, whereas RF model is parallelization method.

XGBoost, CatBoost, and RF are often used for regression tasks, but they are different from each other. RF is a more advanced algorithm based on decision tree. RF is a forest constructed in a random way, and this forest is composed of many unrelated decision trees. Its working principle is to generate multiple decision trees to learn and predict independently. The predictions generated by these decision trees are finally combined into a single prediction, so they are better than any single base learner. XGBoost has a higher precision and flexibility. It supports not only Classification and Regression Tree (CART, a sort of decision tree) but also linear classifier. XGBoost also adds a regularization term to control the complexity of the model. It draws lessons from the practice of RF and supports column sampling, which not only reduces over fitting but also reduces calculation. CatBoost loss function is the same as the XGBoost loss function. The specialty of CatBoost is to deal with category feature efficiently and reasonably. If a category feature has low-cardinality features, that is, the number of set elements formed by the de-duplication of all values of the feature is relatively small, then the advantage of CatBoost cannot be brought into play. In this case, we generally use the one-hot coding method to convert the category feature into numerical type, similar to the processing of the two category features of primiparity/multiparity and method of induced labor introduced in the data preprocessing section. In addition, CatBoost also solves the problems of gradient bias and prediction shift, hence, reduces the occurrence of over fitting and improves the accuracy and generalization ability of the algorithm.

In machine learning tasks, it is usually necessary to make further feature selection, that is, select the features that have a great impact on the prediction results. Then, train the model again with the selected features. Feature selection eliminates some data that have a little impact on the results, which helps to alleviate the problem of less data and improve the accuracy. In addition, feature selection can also reduce the computational overhead and the time of model training. For the three machine learning methods of XGBoost, CatBoost, and RF, we analyze the importance of each feature in different models and make feature selection. For the method of feature selection, sum the value of feature importance, and then calculate the proportion for each feature. The features with a proportion of < 0.033 are discarded, and the remaining features are used as the input of the machine learning model to re-train the model.

We use the methods embedded in these three machine learning models to estimate the importance of features. If a feature is particularly helpful to improve the accuracy of the model, it is considered important. In the model we use, the importance of features is measured by Gini index. The calculation formula of Gini index is where, K means there are K categories,\({p}_{mk}\) represents the proportion of category k in node m. The importance of feature j in node m is the variation of Gini index before and after node m splitting.

Owing to the small amount of data collected, to ensure the higher credibility of the results, we have conducted a five-fold cross-validation for each machine learning method. In each fold, 80% of the data are used as the training set, and 20% of the data are used as the test set. 80%/20% segmentation is also commonly used in machine learning algorithm that deals with medium or small samples. This ratio helps to ensure that there are enough training samples to build a robust model and enough test samples to evaluate that model.

We adjusted the following parameters to make the performance of several machine learning models better. The number of base learners in XGBoost is set to 40. In general, the more the number of spanning trees, the more accurate the model is, but has longer model training. Learning_rate is a parameter used to control the step size of each gradient descent during the model training, which is set to 0.1; max_depth is the maximum depth of each tree, which can be used to prevent over fitting, and that is set to 3. In CatBoost, the number of base learners is set to 70, and the loss function is specified as the RMSE. The number of base learners in the RF is set to 165, and the minimum sample number of leaf nodes is set to 6. When the CART tree of the base learner is divided, the evaluation standard of the feature is set as Gini index.

To evaluate the accuracy of the model, we use RMSE and MAE The calculation formula is shown in formulas (2) and (3), where \({y}_{i}\) is the true value and \(\widehat{{y}_{i}}\) is the predicted value.

The p-value of model significance test is calculated by stats.ttest_ind() in Python’s standard scientific calculation library SciPy. If p < 0.05, it is considered to be statistically significant difference; if p < 0.01, it is considered to be prominent statistically significant difference; if p < 0.001, it is considered to be very prominent statistically significant difference.

As described in the “Methods” section, we gave the prediction value of the time from induced labor to labor by the traditional Bishop scoring method. The processing method of taking the mean value as the predicted value is fair to minimize the prediction error. The prediction results are detailed in Table 4. The table presents the sample number, predicted value, MAE of each group, RMSE of each group, maximum time difference of each group, overall MAE, and overall RMSE. MAE and RMSE were calculated using the formulas described above. The predicted value and the maximum time difference of each group were measured in hours. The results showed that MAE and RMSE decreased with the increase of Bishop score. This also shows that Bishop score has some value as a standard of cervical maturity.

At the same time, the experimental results shown in Table 4 also exposed the limitations of Bishop score. Taken the group with a Bishop score of 5 as an example: the number of samples is 42, and the predicted value is 30.88, but the MAE of this group is 18.80, the RMSE is 24.84, and the maximum time difference is 120. This shows that although Bishop score has some significance for cervical maturity evaluation on the whole, there are great differences among different pregnant women with the same Bishop score. The overall MAE of Bishop scoring method is 19.45, and the overall RMSE is 24.55, indicating that Bishop scoring system has low accuracy and large deviation in evaluating cervical maturity. The MAE and RMSE results of the bishop scoring method that obtained by linear regression model in Table 5. The results in Table 5 showed that MAE and RMSE of linear regression prediction results are 23.17 and 29.60. It is worse than our proposed method, which takes the mean ILTLT of each bishop score group as the predictive value.

After inputting all features into three machine models, the prediction results are presented in Table 5. The MAE and the RMSE in the training set and the MAE and the RMSE in the test set of the three models are closely matching, indicating that the model has not been over fitted. Among them, the XGBoost model performs the best with the MAE of 13.68 and the RMSE of 17.16. The MAE of the control group was 19.45, and the RMSE was 24.55. Our machine learning model has improved by 5.77 and 7.39, respectively, in two error indicators for evaluating prediction accuracy. Even the worst performing RF model has an improvement of 5.24 and 6.5, respectively. The p-values of the three machine learning models are 0.0023, 0.0039, and 0.0071, respectively, which are < 0.01 with a significant statistical difference. It shows that the three machine learning models are significantly better than the traditional Bishop scoring method.

Figure 5 shows the importance of each feature in the XGBoost model (a), CatBoost model (b) and RF model (c). The feature importance of the output of the three models shows that the cervical length, Bishop score, angle, age, ILT, measurement time (MT), MTILT, and amniotomy or misoprostol that are used in the method of induced labor have an important impact on the accuracy of the model. The importance of ILT, MT, and amniotomy used in the method of induced labor is much greater than that of cervical length, Bishop score, angle, age, MTILT, and misoprostol that are used in the method of induced labor.

Method introduced in the previous section was used to evaluate the importance of each feature and make feature selection. After feature selection, the remaining features of XGBoost are cervical length, Bishop score, angle, age, ILT, MT, MTILT, misoprostol, and amniotomy; the remaining features of CatBoost are cervical length, Bishop score, angle, age, ILT, MT, MTILT, and amniotomy; and the remaining features of RF selection are cervical length, Bishop score, angle, age, ILT, MT, and amniotomy.

The results of each model after feature selection are presented in Table 5. The results showed that after feature selection, the p-values of the three models are still < 0.01, indicating that the model is still significantly better than the traditional Bishop scoring method. Moreover, the prediction accuracy of the three models after feature selection is slightly improved.

During feature selection, the feature of Bishop score was selected. In the XGBoost model, the feature of Bishop score ranked 7th in importance. In the CatBoost model, the feature of Bishop score ranked 8th in importance. In the RF model, the feature of Bishop score ranked 7th in importance. After removing the Bishop score, we re-train the model, and the results are presented in Table 5. After the Bishop score is removed, the three models become less significant (p-values increased). The MAE of XGBoost increased from 13.49 h to 13.82 h, with an increase of 2.4%; the RMSE increased from 16.98 h to 17.31 h, with an increase of 1.9%. The MAE of CatBoost increased from 13.74 h to 13.82 h, with an increase of 0.5%; the RMSE increased from 17.25 h to 17.40 h, with an increase of 0.8%. The prediction accuracy of RF (i.e., MAE and RMSE) did not change significantly. The experimental results in Table 5 show that the Bishop score also has some contribution to the accuracy of the model, but its contribution is limited. The machine learning model /excluding Bishop score is still significantly better than the traditional simple Bishop score system.