Ultrasound-based radiomics machine learning models for diagnosing cervical lymph node metastasis in patients with non-small cell lung cancer: a multicentre study

Patients with NSCLC who underwent neck ultrasound examination followed by cervical LN biopsy at Affiliated Hospital of North Sichuan Medical College, Nanchong Central Hospital, and Suining Central Hospital between January 2019 and January 2022 were initially recruited. The inclusion criteria were as follows: (1) cervical LN ultrasound examination followed by biopsy; (2) primary NSCLC; and (3) available detailed descriptive semantic characteristics. Patients with poor ultrasound image quality were excluded from the current study. A total of 807 patients who underwent neck ultrasound examination followed by cervical LN biopsy were initially included in the study, out of whom 319 with primary NSCLC were identified. After 6 patients were excluded due to poor-quality ultrasound images, the study finally enrolled 313 patients. The flow chart of participant inclusion and exclusion is shown in Fig. 1. The Ethics Committees and Review Board of the Nanchong Central Hospital, Affiliated Hospital of North Sichuan Medical College, and Suining Central Hospital approved this study. Patient consent was waived due to the retrospective nature of the study.

All enrolled patients underwent a routine pretreatment ultrasound examination with a Mindray Rezona 7 T (Shenzhen Mindray Bio-Medical Electronics Co, Ltd, China) or a GE Vivid E9 and E20 (General Electric Co, USA) ultrasound system. In addition, all ultrasound devices used a 6–15 L linear array probe with frequencies ranging from 5 to 12 MHz during the examination. We selected images containing the maximum LN cross-sections, which is considered the most useful radiologic criterion for assessing cervical LNM, for further radiomics analysis [15]. Descriptive semantic features were evaluated by two radiologists (X.L.L and R.M.W) and included nine parameters: the site (left and right), long diameter (largest diameter of the cervical LN), short diameter (vertical to the long diameter), shape (irregular and regular), boundary (unclear and clear), calcification (with calcification, without calcification), liquidation (with liquidation, without liquidation), corticomedullary boundary (unclear and clear), and lymph node location of the cervical LNs (according to the guidelines of the American Head and Neck Society and the American Academy of Otolaryngology in 2002) [16]. Demographic characteristics such as the sex and age of the patients were collected (Fig. 2).

3D Slicer software version 4.10.2 (https://​www.​slicer.​org) was used to segment ultrasound images manually. Regions of interest (ROIs) were independently segmented by two radiologists with 13 (X.L.L) and 10 (R.M.W.) years of experience in ultrasound. A final determination of the ROIs for each segmentation required the agreement of the two readers (X.L.L and R.M.W). Radiomic features were extracted from the ROIs on the ultrasound images using the PyRadiomics package version 1.2.0 (Computational Imaging and Bioinformatics Lab, Harvard Medical School) in Python version 3.7 (Python Software Foundation) [17]. A total of 474 radiomic features were extracted from each ultrasound image, including first-order, shape, textural, and wavelet features. The z score was determined d to normalize all radiomic features (Fig. 2).

Univariate and multivariate analyses were used to select descriptive semantic features and demographic characteristics. Univariate logistic regression (LR) was used to screen risk factors for cervical LNM, with features having a P < 0.10 included in the subsequent multivariate LR to identify independent risk factors (P < 0.05); odds ratios (OR) with 95% confidence intervals (Cl) were calculated. For radiomic features, a three-step feature selection method was employed. Unstable and low-reproducibility features were excluded by calculating the interclass correlation coefficient (ICC), and features with an ICC < 0.75 were removed. The nonparametric Mann‒Whitney U test was used to identify features significantly associated with cervical LNM (P < 0.05). Subsequently, the least absolute shrinkage and selection operator (LASSO) was used for the LR model to select the modeling features. The LASSO method decreases feature dimensionality while avoiding collinearity and overfitting [18]. We used fivefold cross-validation to determine the best-fitting λ, allowing the selection of features with nonzero coefficients for fitting the LR models. Feature selection for the random forests (RF) model was performed using the mean decrease accuracy (MDA), a measure for ranking feature importance in the RF algorithm. It directly measures the impact of each feature on the prediction or classification accuracy of the model; the higher the value, the more important the feature is [19]. We selected the 15 top-ranked features of the MDA sum for fitting the RF model. Model performance was assessed using ROC curve analysis, and the areas under the curve (AUCs) were calculated. Five-fold cross-validation was used to validate the model performance internally. In addition, the hold-out method was used to test the performance of models externally. It could evaluate the reproducibility of models and determine whether the models have similar performance in new data. The training set comprised patients from two centers, while patients from another were designated as the test set. All statistical analyses were performed using R (version 3.6.3, R Foundation for Statistical Computing). The AUCs and accuracies were described using two methods: the cross-validation method presents the average value with range, and the hold-out method presents a specific value along with its 95% CI. Quantitative, continuous, and categorical variables are expressed as medians and ranges, mean ± SD, and frequencies with proportions, respectively. A two-tailed P value < 0.05 indicated statistical significance. The workflow of the radiomics analysis is shown in Fig. 2.

In total, 313 patients were included, with a median age of 64 (range, 29–87) years, and 216 (69.0%) were male. Of the 313 patients, 276 (88.18%) had cervical LNM identified by biopsy. The long diameter (mean ± SD, 22.9 ± 0.6 vs. 13.8 ± 0.9, P < 0.001), short diameter (mean ± SD, 14.5 ± 0.4 vs. 8.3 ± 1.5, P < 0.001), shape regularity (P < 0.001), and corticomedullary boundary clarity (P < 0.001) were significantly different between the cervical LNM-positive group and cervical LNM-negative group. No statistically significant differences in age, sex, calcification, liquidation, lymph node site, or location were noted between the groups (P > 0.05). The number of cervical LNs that metastasized to the left and right sides of the neck was approximately equal (48.9% vs. 51.1%, P = 0.370). The most common cervical sites of LNM were lower jugular nodes (IV, 50.4%), spinal accessory nodes and supraclavicular nodes (V, 38.0%). Other descriptive semantic features are summarized in Table 1.

The results of univariate and multivariate analyses are displayed in Table 2. Univariate analysis showed that one demographic feature and four descriptive semantic features, namely sex, long diameter, short diameter, shape, and corticomedullary boundary, had a P value < 0.10. After the multivariate analysis, three statistically significant factors were selected: long diameter (OR = 1.210, 95% Cl, 1.081–1.354, P = 0.001), shape (OR = 2.709, 95% Cl, 1.176–6.238, P = 0.019), and corticomedullary boundary (OR = 9.478, 95% Cl, 1.371–65.529, P = 0.023). Following the ICC evaluation and Mann‒Whitney U test, 104 of 474 radiomic features were selected for subsequent analyses. After the above two steps, among the remaining 104 features, nine modeling radiomic features were selected by the LASSO algorithm for fitting the LR model (Fig. 3), including two shape features, one texture feature, and 6 wavelet features (Table 3). For fitting the RF model, the 15 top-ranked features of the MDA sum were selected, including four shape features, two texture features, and nine wavelet features (Fig. 4). Furthermore, the original_shape2D_Elongation was the most critical feature among those selected by both LASSO and MDA-top15.

The LR and RF algorithms were used to develop the semantic, radiomics, and semantic-radiomics combined models. The models validated using fivefold cross-validation demonstrated satisfactory performance. Both the semantic and radiomics LR models achieved high AUC values, with averages of 0.876 (range: 0.781–0.961) and 0.883 (range: 0.798–0.966) and the best performance, respectively. Additionally, they exhibited average accuracies of 0.878 (range: 0.839–0.952) and 0.901 (range: 0.825–0.968), respectively. The semantic-radiomics combined LR model outperformed the individual LR models, with an average AUC and accuracy of 0.901 (range: 0.862–0.927) and 0.907 (range: 0.871–0.952), respectively. When applying the RF algorithm, the radiomics model achieved an even higher average AUC value of 0.908 (range: 0.837–0.966), with an average accuracy of 0.894 (range: 0.825–0.952). The semantic-radiomics RF model had the highest average AUC of 0.922 (range: 0.872–0.982), exhibiting an average accuracy of 0.898 (range: 0.855–0.968) (Table 4). The hold-out method showed similar results; the radiomics RF model (AUC, 0.877; 95% CI, 0.861–0.970) had a greater AUC value than the semantic LR model (AUC, 0.809; 95% CI, 0.772–0.973), radiomics LR model (AUC, 0.840; 95% CI, 0.750–0.954), and semantic-radiomics combined LR model (AUC, 0.833; 95% CI, 0.725–0.957). The semantic-radiomics combined RF model exhibited the highest AUC value of 0.901 (95% CI, 0.886–0.968), with an accuracy of 0.947 (95% CI, 0.842–0.983) (Table 5). The AUC values of the RF models were generally higher than those of the LR models (Fig. 5, Figure E1).