Transfer learning–based PET/CT three-dimensional convolutional neural network fusion of image and clinical information for prediction of EGFR mutation in lung adenocarcinoma

This study adhered to the CLEAR checklist [27] for the reporting of our radiomics research. The completed CLEAR checklist has been listed in Table S1. To ensure the transparency and reproducibility of our study, we have made all the raw data and analysis code publicly available. For detailed links, please refer to Availability of data and materials.

This retrospective single-center cohort study used privately sourced data, obtaining information from consecutive patients at the Third Affiliated Hospital of Soochow University. Patients with histologically confirmed lung cancer underwent pretreatment ¹⁸F-FDG PET/CT scans in our department between January 2018 and April 2022. The Institutional Review Board approved this study (No. [2022] KD 087) and waived the need for informed consent from the patients. All patient data used in this study were fully de-identified to ensure privacy and confidentiality, in accordance with international data protection guidelines and standards. The sample size was determined based on the consecutive patients available during the study period. Inclusion criteria: (1) the patient was confirmed with lung cancer by surgery or pathological biopsy; (2) the patient underwent ¹⁸F-FDG PET/CT examination before surgery, and the interval between surgery and examination was less than 30 days; (3) the patient had definite EGFR test results; and (4) the patient had no history of other malignant tumors. Exclusion criteria: (1) other pathological subtypes except for ADC, (2) lesions with poor image quality or difficulty in measurement, (3) absence of routine chest CT imaging, and (4) severe liver disease or diabetes. Using an internal testing technique, we designated 404 patients from January 2018 to April 2021 as the training set, while the independent test set was formed from 112 patients spanning May 2021 to April 2022. The process of patient enrollment is outlined in Fig. 1. All data used in this study have not been published or used in any other previous publications.

In this study, we also collected clinical characteristics of the patients, which included: age, gender, smoking history, type of nodules, location of nodules, tumor size, clinical stage (I-IV), carcinoembryonic antigen (CEA), and maximum standardized uptake value (SUVmax).

The detection of EGFR mutations was conducted in tissue samples obtained by surgical resection or puncture. Mutations in exons 18–21 of the EGFR gene were detected using real-time fluorescent PCR. Real-time fluorescent PCR was conducted per the instructions of the Shanghai Yuanqi EGFR gene mutation detection kit, and the details are described in the Supplementary Material EGFR mutation detection method. EGFR mutation was defined if a mutation was detected in any of those exons; otherwise, wild-type EGFR was defined.

The image acquisition protocol was described according to an acquisition protocol based on the Imaging Biomarker Standardization Initiative (IBSI) reporting guidelines [28]. The image acquisition parameters are listed in Table S2. The patient underwent non-contrast chest CT imaging and ¹⁸F-FDG PET/CT (Biograph mCT 64, Siemens, Erlangen, Germany) within one month before surgical treatment. Before the administration of ¹⁸F-FDG, the blood glucose levels of the patients were checked to ensure they were within the acceptable range (< 150 mg/dL). According to the European Association of Nuclear Medicine (EANM) guidelines 1.0 (version 2.0, published in February 2015) [29], ¹⁸F-FDG PET/CT images were acquired 60 ± 5 min after ¹⁸F-FDG injection. All PET/CT images were reconstructed on a processing workstation (TrueD software, Siemens Healthcare).

A nuclear medicine physician with over 10 years of experience selected regions of interest on PET and CT images, and all images were segmented using 3D-Slicer (version 4.11.20200930, www.​slicer.​org). For CT images (3 mm), we utilized a semi-automatic method with NVIDIA AI-assisted annotation (3D-Slicer built-in) and a boundary-based CT segmentation model to process the lung nodule images. For PET images, 3D masks were generated using a semi-automatic segmentation method developed by Beichel et al. [30]. Please refer to Supplementary Material-PET/CT image pre-processing for deep learning for details.

The overall approach to developing the deep learning model is summarized in Fig. 2. To initialize the encoder for the target classification task, we employed Models Genesis, an openly available deep learning model on GitHub (https://​github.​com/​MrGiovanni/​ModelsGenesis.​git). We subsequently fine-tuned it in accordance with the specific requirements of our target task. The structure diagrams of four 3D CNNs, namely CT TL (CT_TL), PET TL (PET_TL), dual-stream TL (DS_TL, fusing PET and CT), and three-stream TL (TS_TL, adding clinical data to PET and CT) networks, are depicted in Fig. S1. Additionally, we trained two models from scratch (CT_origin and PET_origin). Details of the model training and tuning are described in the Supplementary Material-Training of deep learning models.

A visualization method, Grad-CAM, was adopted to explain the prediction process of the deep learning models [31]. High-reaction regions (predicted tumor-associated areas) were retained with a cutoff value of 0.5. When the deep learning model predicts the EGFR mutation status, it informs the clinician which areas attracted the model's attention (divided into wild and mutant type-associated activation areas). We selected only 3D input cross-sectional intermediate layer images for visualization.

Statistical analysis of clinical data and routine PET/CT metabolic parameters was performed using R software (version 3.4.3; http://​www.​R-project.​org/​). The model's performance was assessed using receiver operating characteristic (ROC) curves and quantified by calculating area under the receiver operating characteristic curve (AUC) and 95% confidence intervals (CI). In addition, we further evaluated the model by calculating accuracy, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) to obtain comprehensive quantitative performance metrics. Due to the presence of class imbalance in the data, we chose AUC as the primary performance metric for the model. A pairwise comparison of the AUC values of the models was performed using the method proposed by Delong et al. [32]. All statistical tests were two-sided, and statistical significance was interpreted as p < 0.05. In this study, no missing data was encountered.

Table 1 describes the clinical characteristics of 516 patients, of which 202 (39.1%) were with wild-type EGFR and 314 (60.9%) exhibited EGFR mutations. During our analysis, we observed significant differences in clinical characteristics such as gender, smoking history, type of nodules, tumor long axis, and tumor short axis between patients with EGFR mutations and those with wild-type EGFR (with p < 0.05 in both the training and test sets). Based on these observations and informed by clinical prior knowledge [33, 34], we decided to incorporate these characteristics as clinical information in constructing TS_TL based on PET/CT images. During this process, we normalized the quantitative indicators to ensure they fall within the range of 0 to 1, aligning with the scale of the normalized PET/CT image data. For categorical variables, we retained their original binary format, allowing for the consideration of clinical variable diversity in the construction of the TS_TL model.

The predictive performance of several deep learning models in the test set is listed in Table 2 (see Table S3 for training set). In the training set, TS_TL showed the best predictive performance (AUC = 0.883), further confirmed in the independent test set (AUC = 0.730). CT_TL and PET_TL outperformed CT_origin and PET_origin in both training and test sets, with significant improvement in test set for CT_TL (AUC = 0.701 vs. 0.544, p = 0.027) and in training set for PET_TL (AUC = 0.770 vs. 0.619, p < 0.001; Fig. 3 A, B). Also, CT_TL showed high sensitivity and low specificity in both sets, while PET_TL showed low sensitivity and high specificity (Table S3 and Table 2).

The tumor long axis, tumor short axis, tumor stage, and SUVmax showed statistical differences between the two sets (all p < 0.05), which might be attributed to the different compositions of patients at different periods in our center (Table S4). To eliminate this discrepancy, we performed a hierarchical analysis to validate the diagnostic performance of the four TL models for different tumor stages (Table S5). For stage I-II tumors, TS_TL exhibited significant overfitting (AUC of training set vs. test set: 0.903 vs. 0.667), while DS_TL had the optimal performance in the test set (AUC of training set vs. test set: 0.862 vs. 0.708); for stage III-IV tumors, TS_TL showed the best performance in both training and test sets (AUC of training set vs. test set: 0.871 vs. 0.760).

We established four radiomics models of CT, PET, PET/CT, and PET/CT combined with clinical features (CT_RS, PET_RS, DS_RS, TS_RS) and compared them with the proposed TL models (Table 3 and Table S6). The specific methods for the development of the radiomics model are described in the Supplementary Material-Development of four radiomics models, Table S7 and Table S8. ROC curves of the four radiomics models and SUVmax are shown in Figure S2. First, PET_RS outperformed SUVmax in both training and test sets but only showed significance in the training set (training set: AUC = 0.651 vs. 0.582, p = 0.014). It was subsequently validated that PET_TL performed significantly better in the training set but slightly inferior in the test set than the PET_RS model (training set: AUC = 0.770 vs. 0.651, p < 0.001; test set: AUC = 0.645 vs. 0.661, p = 0.763). CT_TL performed better than CT_RS in both training and test sets, with a significant improvement in the training set only (training set: AUC = 0.739 vs. 0.655, p < 0.001).

Compared to single-modal radiomics models CT_RS and PET_RS, DS_RS performed slightly better in the training set and worse in the test set, but the differences were not significant (training set: AUC = 0.662 vs. 0.655 vs. 0.651, both p > 0.05; test set: AUC = 0.620 vs. 0.639 vs. 0.661, both p > 0.05). DS_TL performed better than DS_RS in both training and test sets (training set: AUC = 0.823 vs. 0.662, p < 0.001; test set: AUC = 0.722 vs. 0.620, p = 0.033).

By combining DS_RS with clinical features, TS_RS outperformed CT_RS, PET_RS, and DS_RS in both training and test sets, while statistical significance was only noted in the training set (training set: AUC = 0.771 vs. 0.655 vs. 0.651 vs. 0.662, all p < 0.001). Similarly, TS_TL outperformed TS_RS in both training and test sets, with statistical significance in training set only (training set: AUC = 0.883 vs. 0.771, p < 0.001). When DS_TL was combined with clinical features, TS_TL outperformed DS_TL in both training and test sets, with significance shown only in the training set (training set: AUC = 0.883 vs. 0.823, p < 0.001).

Figure 4 displays six representative solid lesions (three wild-type and three mutant) along with TS_TL's wild and mutant type-associated activation areas. In solid lesions, TS_TL consistently focuses on the local and peripheral areas of the lesion in CT images (Fig. 4a, c, e, g, i, k) and most metabolic areas in PET images (Fig. 4b, d, f, h, j, l), regardless of wild-type or mutant status.

Figure 5 displays six representative subsolid lesions (three wild-type and three mutant) along with TS_TL's wild and mutant type-associated activation areas. For subsolid wild-type lesions, the wild type-associated activation areas in CT and PET images are similar to those in solid lesions (Fig. 5a-f). For subsolid mutant lesions, the mutant type-associated activation areas in CT images still robustly capture the local and peripheral areas of the lesion (Fig. 5g, i, k), while the mutant type-associated activation areas in PET images do not effectively capture the metabolic areas of the lesion (Fig. 5, j, l).

Figure 6 shows the changes in the mutant type-associated activation areas and the predicted mutation probability (P⁺) before and after TKI treatment in three EGFR-mutant patients with effective TKI treatment. Both case 1 and case 2 are solid lesions with non-ideal model predictions before TKI treatment (P⁺ values in Fig. 6a, b and c, d are both less than 0.5, indicating that the model has not yet captured a clear image mutation pattern). The P⁺ values of both cases significantly increase after TKI treatment (from 0.462 to 0.679 and from 0.299 to 0.485). Looking at the mutant type-associated activation areas after treatment (Fig. 6g, h and i, j), they are also similar to the previously mentioned subsolid mutant lesions (refer to Fig. 5g-l), indicating that the treatment has changed the image pattern captured by the model. For subsolid lesion case 3, there are no obvious mutant type-associated activation areas in either CT or PET pathways before and after treatment (Fig. 6e, f and k, l). However, the P⁺ value increases significantly after treatment (from 0.373 to 0.620). The changes in classical methods and P⁺ values before and after TKI treatment in 3 cases can be seen in Table S9.