Retrospective validation study of an artificial neural network-based preoperative decision-support tool for noninvasive lymph node staging (NILS) in women with primary breast cancer (ISRCTN14341750)

The study population for the development of the ANN model was based on a prospectively maintained pathological database on consecutive breast cancer patients with clinically node negative status scheduled for primary surgery at Skåne University Hospital, Lund, Sweden. The original ANN model presented in 2019 was based on a combination of preoperatively and postoperatively available data [18]. The characteristics of the development cohort is thoroughly described in the provided Supplementary File 1.

In short, ANN is a machine learning method that has the ability to explore nonlinear associations in a dataset. The ANN within the NILS prediction model contains ensembles of multilayer perceptrons (MLP). Each MLP contains three layers: 1) the input layer (clinicopathological variables), 2) the hidden layer, and 3) the output layer. To learn the association to nodal status (output), the MLPs were trained by a standard back-propagation technique. Four-fold cross-validation, repeated five times, was used as the internal model validation strategy to obtain optimal model parameters.

In this study, the NILS model for the prediction of nodal status in cN0 BC patients was based on 10 preoperatively available features: patient age at diagnosis and eight features that are easily and reliably accessible from preoperative imaging and CNBs: tumor size, multifocality, estrogen receptor (ER) status, progesterone receptor (PR) status, histological type, mode of detection, tumor localization in the breast, and Ki-67 positivity. Vascular invasion, the tenth feature of the NILS model, was difficult to determine preoperatively (Fig. 1). Therefore, a separate ANN model was developed to impute this feature, using the other nine features of the NILS model as predictors. The model reflects routine diagnostic BC workup and was trained to handle missing histopathological input variables. NILS handles combinations of unknown LVI status and missing values for ER/PR status and the proliferation index Ki67; however, the six remaining variables (age, mode of detection, tumor localization in the breast, tumor size, multifocality, and histological type) were mandatory. A user-friendly web implementation of the NILS model was tested in this study. Data from patient records were extracted and used as inputs to predict the axillary nodal status. For each patient, the output from the calculator displays the estimated probability of healthy lymph nodes as well as the malignant or benign categorization of the nodes [20]. Although strictly preoperative variables were used for the estimation of the probability of healthy lymph nodes using the NILS web interface, supplementary postoperative variables (10 input variables) were acquired, enabling a batch-mode validation of the original model (thus not using the web interface).

A total of 601 patients who underwent surgery for primary BC at two sites were included: 401 at site 1 (Malmö, Skane University Hospital, Sweden in 2020), constituting a temporal validation cohort, and 200 at site 2 (Helsingborg Regional Hospital, Sweden, between 2019 and 2020), constituting a geographical validation cohort. Patients were identified through the national registry of cancer diagnoses, treatments, and outcomes (the Swedish National Quality Registry for Breast Cancer [23]). Digital medical charts were reviewed by three experienced research nurses and two physicians (n = 5 in total). The inclusion and exclusion criteria strictly followed the study protocol [21]. Fifteen patients were excluded because they did not meet the inclusion criteria (Fig. 2).

cN0 was defined as no axillary nodal involvement on clinical or radiological examination. pN0 was defined as no invasive cell clusters of > 0.2 mm at the largest diameter in the lymph node (i.e., the presence of isolated tumor cells or cell clusters that are ≤ 0.2 mm at the largest diameter are considered pN0).

In this validation study, we used the Research Electronic Data Capture (REDCap) module using audit trail for data management [24]. Thorough data monitoring was performed by an independent researcher not involved in data entry, according to a predefined flow chart and quality control plan (Supplementary Material S1 [21]). In addition, the study statistician (POB) checked the entered data to identify inconsistencies not captured by the data entry rules, as defined in the REDCap. The low number of missing data points was due to meticulous data management.

Prior to study initiation, sample size calculation was performed as described in the study protocol [21]. Descriptive statistics were presented for the whole cohort and split according to the study site. Chi-squared tests and t-tests were used, where appropriate, to test for homogeneity across sites. The performance of the NILS model was assessed in terms of discrimination with the area under the receiver operating characteristic (ROC) curve (AUC) and calibration, that is, comparison of the observed and predicted event rates of axillary disease using Hosmer–Lemeshow graphics summarized by calibration slope and intercept [25]. Perfect calibration corresponds to a slope of 1 and an intercept of 0. Locally weighted scatterplot smoothing (LOWESS) was used to capture the calibration performance for high probabilities of N0, the range of interest for a de-escalation strategy.

The primary endpoint was axillary nodal status (discrimination, N0 vs. N+) determined by the NILS model (test result) in comparison with nodal status by definitive pathological diagnosis after SLNB. As specified in the study plan, subgroup analyses were performed according to: 1) study site; 2) complete data on all CNB features in NILS (vascular invasion was excluded because of the routine unavailability of this variable preoperatively) or records with imputed values; 3) mastectomy or breast-conserving surgery; and 4) separately for ER-positive/HER2-negative, T1-2, and age ≥ 70 (subgroup specified by Choosing Wisely and adopted by ASCO guidelines on axillary staging [2]). Test characteristics in terms of sensitivity, specificity, false positive rate (FPR), and false negative rate (FNR) were computed. NILS indicating nodal disease (N+) was defined as ”positive” [1] in coherence with N+ as assessed by pathology (Supplementary File 2). The FNR in the NILS model was calculated as the number of false N0 cases predicted by NILS divided by the number of cases with pathology-verified axillary nodal metastasis. Finally, clinical utility was evaluated as the proportion of patients for whom SLNB could have been avoided and by decision curve analysis as a net benefit [26] for NILS compared to SLNB for all patients in the target population.

In addition, the original ANN model, which included both preoperatively and postoperatively available variables, was validated in batch-mode, thus bypassing the web interface (preoperative variables: patient age at diagnosis, mode of detection, and tumor localization in the breast; postoperative variables: tumor size, multifocality, ER status, PR status, histological type, Ki67 value, and vascular invasion).

All analyses were performed using the Statistical Package for the Social Sciences Statistics for Windows, version 26 (IBM Corp., Armonk, NY, USA) and StataCorp. 2021. Stata Statistical Software: Release 17. Decision curve analysis was performed in Stata using dca.ado and custom-made software written in C (gcc version 7.5.0) and Perl (version 5.26.1).

All procedures performed in this study involving human participants were in accordance with the ethical standards of the institutional or national research committee and with the 1964 Declaration of Helsinki and its later amendments or comparable ethical standards. This study was approved by the Swedish Ethical Review Authority (Ethical Review Board, Stockholm Department 3 Medicine, committee reference number: 2021–00174). Previously treated patients received study information through advertisements in the local press and were allowed to opt out by using the provided contact information. The ethics committee (Ethical Review Board, Stockholm Department 3 Medicine, committee reference number: 2021–00174) waived the requirement for informed consent and consent for publication. As per Swedish law, all patients receiving treatment for BC at hospitals are to provide consent for registration in a national registry of cancer. The local department for personal data admission at the hospital (KVB Samråd, Region Skåne, Sweden) granted researchers access, with digital logging, to patients’ digital medical charts. Only users authorized by the principal investigator had access to the NILS web calculator.

The mean age of the patients in the cohort at diagnosis was 65 years; thus, the majority of the women were postmenopausal (85%) (Table 1). Most patients had unifocal (91% on mammography), ER-positive (93%), PR-positive (82%), and HER2-negative (95%) tumors (according to preoperative CNB). The mean tumor size on mammography was 19 mm, with slightly larger tumors at site 2 than at site 1 (mean 17 mm vs. 22 mm, p-value: < 0.001). Data on ultrasound variables are shown in Supplementary File 3. Approximately three-fourths of the patients had no metastases in SLNB (N0 74%). Of the patients with metastases on SLNB (N+), the majority had macrometastases (Table 2). The SLNB reduction rate was 26% (Table 3). A comparison between the current validation and original cohorts, in which the NILS model was first developed, is presented in Supplementary File 1.

The discriminatory ability of the NILS model was assessed with ROC analysis: the AUC for the entire cohort was 0.6741 (95% confidence interval [CI]: 0.6255–0.7227) (Fig. 3.) In the subset of women aged ≥ 70 years with ER-positive/HER2-negative tumors (n = 113), the AUC was 0.6006 (95% CI: 0.4888–0.7124) (Supplementary Fig. 1). The NILS model showed the best calibration in patients with a high estimated probability of healthy axilla (lower left corner of Fig. 3 B). In the entire cohort, using the predefined cut-off, the sensitivity of the NILS model was 88% (136/154), corresponding to a FNR of 12% (18/154). The specificity was 31% (133/432), and 151 of the 586 patients (26%) could potentially have been spared SLNB if the NILS model had been used. The net benefit [26] of the NILS model compared to the SLNB strategies for all and none is shown in Fig. 4. The vertical line at the threshold of 20% of being N+ corresponds to accepting four false-positive tests for each true-positive test. The difference in net benefit at this cut-off between using the NILS model and the strategy SLNB to all was 0.026, which should be interpreted as 2.6 more true positives in favor of the NILS model for every 100 patients in the target population. Furthermore, it is clear from the Fig. 4 that, as measured by net benefit, NILS is superior to SLNB for all patients in a wide range of cut-offs, approximately 20%. The cut-off value of 20% highlighted in Fig. 4 is a compromise between the predefined separate thresholds for patients with and without imputed biomarker data, which are slightly above and below 20%, respectively.

The discriminatory performance was similar for sites 1 and 2 (Fig. 5). Likewise, a similar discriminatory performance was noted in patients with complete data on biomarkers (except for vascular invasion) and in those with imputed biomarkers (Supplementary Fig. 2). The batch-mode validation, utilizing a combination of preoperative and postoperative variables, showed an AUC of 0.7362 (95% CI: 0.6917–0.7807) for discrimination of N0 vs N+ (Fig. 6). The discrimination in terms of AUC for nodal prediction in the breast surgery-based prespecified subgroup analyses based is provided in Supplementary Fig. 3.

Our model performed equally well when biomarkers and vascular invasion were imputed as when these data were available. The AUC was marginally higher when data were imputed; however, the calibration slope deviated more for the optimal value of 1.0. This shows the robustness of the NILS model and broadens the area of application to sites in which biomarkers are not routinely analyzed. In addition, a significant advantage was the previously published study protocol, which includes a predefined data quality and statistical plan.

The NILS prediction model is based on an ANN algorithm. However, there are other machine-learning models in addition to logistic regression models for the same purpose. In the original publication, the ANN model performed slightly better than logistic regression upon internal validation with cross-validation, and thus selected for further application and development. Variation in the proportions of node positive breast cancer in different target population may impact the performance of the NILS prediction model.

It is interesting to briefly consider the variable “screening-detected.” In Sweden, women aged 40–74 years are offered mammographic screening; thus, only women in this age group have the prospect of having screening-detected BC. However, the NILS model was developed independently of these age conditions. In addition, poorer discrimination of NILS in the population of women aged ≥ 70 years with ER-positive/HER2-negative tumors was a limitation of the present study. In future work with the NILS decision-support tool, it is possible to adjust the screening/age discrepancy and develop a model specific to the target population of women aged ≥ 70 years with ER-positive/HER2-negative tumors to better cohere with plausible future clinical settings.

We performed a thorough retrospective validation of the NILS model. Concurrently, we are optimizing the web interface to improve its usability before clinical implementation.

To provide patient-centered care on the management of axilla in early breast cancer, the ASCO guideline states that patients should be evaluated on a case-by-case basis. We present a user-friendly decision-support tool for personalized pre-operative prediction of nodal status, with the potential to identify one-quarter of patients as eligible for abstaining SLNB. The adoption of prevailing guidelines can be enhanced by providing clinicians and patients with such tools. Furthermore, by conducting a health–economic analysis, the implementation of the NILS prediction model was shown to be cost-effective and associated with health gain [19].