Comparison and development of machine learning tools in the prediction of chronic kidney disease progression

A total of 551 pathologically confirmed CKD patients with 24-h urinary protein were recruited from August 2015 to September 2018 at the Department of Nephrology in the Shanghai Huadong Hospital Affiliated to Fudan University. None of the patients were diagnosed with METS, cancers or cardio- and cerebrovascular diseases. The detailed demographic characteristics of the cohort are listed in Table 1. In this study, urine protein > 1 g/24 h was used as the outcome variable to classify the progress and severity of proteinuria in patients with kidney disease. Our study was approved by the Clinical Ethics Review Committee of the Shanghai Huadong Hospital Affiliated to Fudan University, and clinical consent was obtained from all patients. We first cleaned and formatted the data before model fitting. Then, in the pre-processing stage, we transformed categorical variables into binary dummy variables. Finally, we scaled the data as most models are affected by the difference in the scale of the variables. We performed power analysis over urinary protein values to determine if the sample size was suitable for further statistical process (alpha = 0.05). All values were normalized to reduce the dimension-introduced bias using Z-score standardization as previously described [9‐13]: (Eq. 1).where μ is the average of the features across all samples, and α is the standard deviation.

In this study, nine predictive models were established to predict the progression of urinary protein in patients with chronic kidney disease, and model selection was based on several currently and frequently adopted predictive model types. For the linear model, the logistic regression model (LR) [14, 15], the elastic network model (Elastic Net) [16‐18], the lasso regression model (Lasso) [19], and the ridge regression model (Ridge) were selected [20‐22]. The neural network model (NN) [23] was chosen because it is an important class of nonlinear prediction models [24] and has been reported to predict CKD. For the kernel-based model, a support vector machine (SVM) with a Gaussian kernel (RBF) has been widely adopted in many clinical applications, such as coronary artery disease prediction [25, 26]. For the decision tree approach, the random forest (RF) model [27‐29] and the XGBoost model [30‐32] have also been used in clinical research. Finally, a basic prediction technique [33], k-nearest neighbor algorithm (k-NN) was built [34]. The model was fitted using the method described above for each set of parameters, which were adjusted to obtain the average performance index. The log-loss was calculated to indicate the confidence of the model. The lower the log-loss value is, the more confident the model is for its classification results [35, 36]. The technical parameters of the selected prediction models are listed for the optimization of the equations (Table 2). Model establishment and brief illustrations are described in Additional file 1.

In this study, we have improved the method of the data resampling technique [37] considering the overfitting problem caused by the empirical risk minimization algorithm of the optimization model. First, the candidate values of the model parameters were defined, and the patients were randomly allocated into a training set (80%) and a validation set (20%), where the two class proportions in each set were the same. In the training set, k-fold cross-validation (k = 10) was used, and various parameter combinations were exhausted by grid search. For each set of parameters, 9/10 of data were used for fitting the model in turn, and 1/10 of data was used for validation. AU-ROC was selected as the performance index, which was calculated 10 times, and its average performance was calculated as the parameter score of the current parameter combination. The average value of the parameter value grid was selected as the best performance adjustment parameter of the current iteration and was finally executed on the test set. A forecast flow chart is shown in Fig. 1. This step was repeated 20 times randomly, i.e., 20 resampling iterations were defined. This study used the same resampling to evaluate different models. For each model, the evaluation indicators used were the confusion matrix, area under the curve (AUC), sensitivity (recall), specificity, accuracy, log-loss, AP, F1, false positive rate (FPR), and precision. Each evaluation used the same data segmentation and repetition to ensure a fair comparison of the models. Additionally, we carried out hierarchical clustering analysis over methods based on false positive (FP) and false negative (FN) values. In this study, Python (version 3.7.0) and R (version 3.5.1) were used to build and evaluate the models.

To facilitate the predictive function in clinical practices, we designed and developed a CKD Prediction System for the above models whose predictive precision, sensitivity and specificity were highest. The proteinuria predictor was embedded in the web tool. User data interaction and visualization of analysis results were displayed using HTML5, JavaScript, and PHP. Source codes for model establishment by Python and web tools by PHP are provided in Additional file 1.

This study recruited 551 patients with CKD from the Department of Nephrology, Huadong Hospital, Shanghai Fudan University Affiliated Hospital who had pathologically confirmed 24-h urine protein. The training dataset included 330 mild CKD patients (urinary protein ≤ 1 g/24 h) and 221 moderate/severe CKD patients (urinary protein ≥ 1 g/24 h). Through statistical power analysis of the urinary protein values, the sample size in our study was competent for further procedures with power at 1. The following non-urine indicators of 13 outpatient blood biochemistry tests and 5 demographic features were used as predictive variables: CRP, ALB, TC, TG, BG, BUN, EGFR, Scr, SUA, SK, Sna, LDL, HDL, sex, age, height, weight, and BMI. Urine protein (g/24 h) was considered an outcome variable to judge the status of CKD patients.

The average AU-ROC for different models and their parameters are listed (Fig. 2). The SVM was not sensitive to cost choice C, and the kernel smoothing parameter σ of 0.01 was optimal. For k-NN, a relatively large number of k = 24 was optimal; for RF, a relatively large number of randomly selected 61 subtrees provided the best performance. The maximum depth (max_depth) of the XGBoost tree was 3, and the minimum leaf node sample weight (min_child_weight) of 1 achieved optimal performance.

The average ROC curves and PR curves during the 20-fold data resampling process are shown in Fig. 3a, b. Most models had AUC values above 0.85, but the value of k-NN was lower (0.80). We used the AP value as the criterion for the PR curve [38]. The APs of the Elastic Net, Lasso, LR, Ridge, SVM and XGBoost models were all above 0.82. The confusion matrix (rounding) was also calculated for the nine models (Table 3). As shown in Table 3, k-NN generated a large amount of FNs (= 12) and FPs (= 17) during the prediction process, while the other models had the same number of FNs, which could be controlled within 10, where the Lasso and Elastic Net models produced the least amount of FNs (= 7). The model XGBoost produced the minimum number of FPs (= 11).

The nine methods were clustered based on hierarchical clustering analysis using the FP and FN values from one random sampling (Additional file 1: Figure S1). Similar models drew similar results; for example, the decision tree models XGBoost and random forest were clustered closely. Table 4 shows the AUC, sensitivity (recall), specificity, accuracy, log-loss, FP rate, precision, f1, and AP of each model evaluation result.

There were significant performance differences between the different models (Fig. 3c and Table 4). The linear models LR, Elastic Net, Lasso and Ridge had excellent performance, and the accuracy rate was up to 0.80. Among them, LR obtained the highest AUC value of 0.873, and the tree model XGBoost had an AU-ROC value of 0.868 and an accuracy rate of 0.83. K-NN obtained the lowest AUC value of 0.802. The best performance of sensitivity was the Elastic Net model, which is suitable for the early diagnosis of proteinuria progression in patients with chronic kidney disease. The best particularity was XGBoost and LR, which are suitable for the early stage of proteinuria in patients with chronic kidney disease. The sensitivity and specificity of the LR, Elastic Net, SVM, XGBoost and Lasso models both reached over 0.80. The XGBoost model had the lowest log-loss value (5.87), indicating that Lasso is more useful for its classification results, while the k-NN model had the highest log-loss value of 8.91. LR and XGBoost performed best regarding FP rate and precision, while XGBoost showed the highest AP values.

We further compared each model using the AU-ROC mean and paired t-test. Compared to the other models, LR, Elastic Net, Lasso, and XGBoost showed no statistical significance, implying that these models were similar in terms of their predictive power. In our study, k-NN provided the lowest predictive performance (Table 5).

The importance features, as shown in the effect sizes, were calculated (Fig. 4). For most of the models, the importance could be divided into two groups. The first group included ALB, Scr, TG, LDL, age, EGFR, and TC, which had important influences on the predictability of the models. The second group included BMI, height, weight and CRP, which showed less impact on prediction. ALB and TG were shown with the highest frequencies in the top predictors in all nine models, while Scr, TC, age and LDL were also shown with a high effect size in more than half of the models.

In this study, we developed a Web tool (CKD Prediction System) for clinical practice that can be widely used in the evaluation of proteinuria progress in nephrology and during follow-up examinations (Fig. 5). Clinicians can visit the system website (http://​www.​ckdprediction.​com) and use the desired clinical model by entering the 13 clinical biochemical indicators and 5 demographic features from follow-up CKD patients. The calculated probability of CKD progression will be predicted and obtained by the system. For example, after we input the features into the CKD Prediction System, the tool will feed back the prediction of the patient’s current status with “mild” or “moderate/severe”.