Development and testing of a random forest-based machine learning model for predicting events among breast cancer patients with a poor response to neoadjuvant chemotherapy

The patients diagnosed as BC in the First Affiliated Hospital of Chongqing Medical University from January 2013 to December 2019 were fully reviewed. This study was approved by the ethics committee of the First Affiliated Hospital of Chongqing Medical University (ID: No. 2020–59). Patients submitted to NAC were selected for the following analysis. According to the Recist guidelines, version 1.1, the response to the NAC included (1) complete response (CR, the tumor disappeared totally by imaging examination and lasted for at least 4 weeks), (2) partial response (PR, decrease no less than 30% in the sum of lesions and last for at least 4 weeks), progressive disease (PD, more than 20% increase in the sum of diameters of lesions), stable disease (SD, neither met the criteria of PR nor did PD) [22, 23]. In this study, patients with SD or PD were included. Of these patients, we first recorded their clinicopathological features, as well as their physical condition post-surgery in 1 and 5 years, respectively. The patients with tumor recurrence or death were divided into the event group, otherwise were into the non-event group. Then, we compared the clinicopathological characteristics between the two groups. Next, based on the Cox regression in both univariate and multivariate analysis, we screened out the variables for constructing the random forest model. For comparison, a logistic model was developed. Finally, we validated the performance of the model. Figure 1 displays the study process.

NAC was performed according to the guidelines of the Chinese Society of Clinical Oncology (CSCO). The CSCO has a longstanding dedication to conducting Continuing Medical Education and facilitating multi-center collaborative research in clinical oncology. Their efforts aim to promote the standardization of tumor diagnosis and treatment, ultimately enhancing the academic excellence of clinical oncology in China. If the patients met the requirement, they would be submitted to the NAC in a week. The treatment protocol was TEC (docetaxel 75 mg/m², epirubicin 75 mg/m², and cyclophosphamide 500 mg/m²) or EC (epirubicin 75 mg/m², and cyclophosphamide 500 mg/m²), drugs were administered every 21 days. After 4–6 cycles of NAC, the response was evaluated by both the clinicians and the pathologists [24]. Mastectomy or breast-conserving surgery plus axillary lymphadenectomy was performed subsequently. Furthermore, the systemic therapy procedures were as follows: First, patients with an IHC core of 3 + or 2 + with positive FISH was defined as HER2-positive need receive anti-HER2 targeted therapy. Second, patients with estrogen receptor positive or progesterone receptor positive require hormone therapy. Third, the indications of chemotherapy included: (1) high level of Ki67 index; (2) triple negative breast cancer; (3) HER2 positive; (3) regional lymph node positive; (4) Histological grade III or higher; (5) Genetic testing indicates a high risk of recurrence; (6) a relatively larger tumor size. Fourth, for node-positive disease, all the patients receive PMRT to the chest wall. For the node-negative, tumor less than or equal to 5 cm, and clear margins (≥ 1 mm), PMRT is generally not needed. However, if patients with high-risk, including (1) central tumors; (2) T3; (3) tumors not less than 2 cm with fewer than 10 axillary nodes removed; (4) at least one of (a) grade 3, (b) ER-negative, or (c) lymphovascular invasion, PMRT should be considered [25, 26].

The immunohistochemistry (IHC) index was assessed based on the core needle biopsy of the tumor. Hormone receptor (HR) positive was defined as estrogen receptor (ER) or progesterone receptors (PR) expressing cells percentage > 1% by IHC. HER2 status was classified according to the IHC score and the result of fluorescence in situ hybridization (FISH). A score of 0 was HER2 negative, a score of 1 + or 2 + without amplification of ERBB2 gene was HER2-low expression, and a score of 3 + or 2 + with ERBB2 gene amplified was HER2 positive. A cut-off value of 1% was used for defining the P53 positive expression [27, 28]. These were evaluated by two pathologists independently and blindly.

All patients were interviewed by telephone from discharge to February 1st, 2021. Disease-free survival (DFS) was selected for assessing patients’ prognoses. DFS was defined as the period between surgery to the event that first recorded tumor relapse or metastasis. The patients with follow-up time longer than 1 year, or who had an event within 1 year were divided into the 1-year cohort. The patients with follow-up time longer than 5 years, or who had an event within 5 years were divided into the 5-year cohort.

In this study, we first established an RF model in the 1-year cohort, and the outcome variable was whether the patients had an event (defined as tumor relapse or metastasis). Based on the result of the chi-square test and the Cox regression, multiple factors that might influence the outcomes were chosen as explanatory variables, which were used to classify and predict the outcome variables. The data were randomly and independently divided into the training set and the test set according to the ratio of 8:2.

Then, we used the training set for developing optimal models. The importance of each explanatory variable was assessed by evaluating the Mean decrease accuracy and the Mean decrease gini. The higher the value, which means the more important the explanatory variable in the model. The RF model was tested in the test set, and the receiver operating characteristic (ROC) curve and the area under the curve (AUC) were used to measure the accuracy of the models.

Next, the same explanatory variables in the RF model were used to develop a logistic regression to classify and predict the outcome variable in the 1-year cohort. Finally, with the same method as the above mentioned, an RF model and a logistic regression were developed in the 5-year cohort. The performance was compared between the RF model and the logistic regression based on specificity, sensitivity, and AUC.

We screened patients from the Surveillance, Epidemiology, and End Results (SEER) research data, 8 Registries, Nov 2022 Sub (1975–2022). The breast cancer patients—who were stated as no response to NAC and had comprehensive data—were selected. These data were used to validate the previously RF model and logistic regression model which constructed based on the 5-year cohort.

A total of 980 BC patients received NAC. Of them, 317 patients were with no response to NAC (297 of SD and 20 of PD). The mean age was 49.3 years old (ranging from 20 to 72). 2 patients refused to receive follow-up. Of the rest 315 patients, the mean follow-up time was 40.4 months (range from 2 to 101). Totally, 82 cases had an event during the follow-up time. 251 patients were with a follow-up time of more than 1 year, and 30 patients had an event within 1 year. Stage T (P = 0.017) and N (P = 0.031), HR (P = 0.046) and HER2 (P = 0.013) were significant with the event. 165 patients underwent follow-up for more than 5 years and 80 patients had an event within 5 years. HR and HER2 were significant in the 5-year group (Table 1). The Cox regression revealed that stage T, HR, HER2, and P53 were risk predictors for DFS in the univariate analysis, while HR, HER2, and P53 were independent predictive factors in the multivariate analysis (Table 2).

Based on the R package “randomForest”, RF models for predicting the probability of the event in the 1-year cohort and 5-year cohort were developed, respectively. Firstly, in the 1-year cohort, 225 patients were divided into the training set while 56 were into the test set. After measurements, Stage T, stage N, HR, HER2, and P53 were selected as explanatory variables. When mtry = 5 (Fig. 2A), ntree = 1200 (Fig. 2B), the model would have the lowest error. The model was significant at P = 0.01 and with an out-of-bag (OOB) error of 14.67%. Figure 3A, B shows the mean decrease accuracy and mean decrease gini index of each variable.

Secondly, the model was tested in the test set. The ROC of the RF model was performed and the specificity, sensitivity, and AUC were 0.800 (95% CI was 0.659–0.895), 0.833 (95% CI was 0.365–0.991), and 0.810 (95% CI was 0.640–0.980, Fig. 4B) in the test set, respectively. The positive predictive value (PPV), negative predictive value (NPV) and F1 score were 0.333, 0.976 and 0.476, respectively. Thirdly, logistic regression was conducted with the same explanatory variables. The specificity, sensitivity, and AUC were 0.520 (95% CI was 0.376–0.661), 0.833 (95% CI was 0.365–0.991), and 0.653 (95% CI was 0.363–0.944, Fig. 5B). The PPV, NPV and F1 score were 0.172, 0.963 and 0.285 respectively.

Next, we used the same method to establish the RF model and logistic regression in the 5-year cohort, 132 patients were divided into the training set while 33 were into the test set. cT, HR, and HER2 were selected as explanatory variables. Figure 3C, D shows the mean decrease accuracy and mean decrease gini index of each variable. Here in the RF model, we chose mtry = 3 (Fig. 2C) and ntree = 4500 (Fig. 2D). The RF model was significant at P < 0.01 and with an OOB error of 30.30%. The specificity, sensitivity, and AUC were 0.882 (95% CI was 0.623–0.979), 0.750 (95% CI was 0.474–0.917), and 0.829 (95% CI was 0.676–0.982) in the test set (Fig. 4D). The PPV, NPV and F1 score were 0.857, 0.789 and 0.80, respectively. Of the logistic regression, the specificity, sensitivity, and AUC were 0.882 (95% CI was 0.623–0.979), 0.688 (95% CI was 0.415–0.879), and 0.752 (95% CI was 0.575–0.829) in the test set (Fig. 5D). The PPV, NPV and F1 score were 0.846, 0.75 and 0.759 respectively. Relative information is displayed in Figs. 2, 5 and Table 3.

Finally, in the 5-year cohort test set, we categorized patients into the low risk group if the probability of no event occurring exceeds 50%, and into the high risk group otherwise. Afterwards, we proceeded to plot survival curves for both groups and observed a clear distinction: the low-risk group exhibited significantly longer survival times in comparison to the high-risk group (Fig. 6, P = 0.04).

Totally, 514 patients were identified as no response to NAC from the SEER database, Table 4 shows their clinical features. Out of them, 250 patients had events in the follow-up period. Of the RF model, the specificity, sensitivity, and AUC were 0.765 (95% CI was 0.708–0.814), 0.812 (95% CI was 0.757–0.857), and 0.779 (95% CI was 0.613–0.945), Fig. 7A. The PPV, NPV and F1 score were 0.766, 0.811 and 0.788, respectively. Of the logistics regression model, the specificity, sensitivity, and AUC were 0.833 (95% CI was 0.782–0.875), 0.376 (95% CI was 0.316–0.440), and 0.619 (95% CI was 0.572–0.666), Fig. 7B. The PPV, NPV and F1 score were 0.681, 0.585 and 0.474, respectively. Table 5 presents the relevant information.

The performance of the RF model and the logistic regression model were compared in the 1 year and the 5-year group. The AUC of the RF model was higher than that of logistic regression in the 1-year cohort, the 5-year cohort, as well as in the external validation set. The RF model can reach a satisfactory performance.