Predicting conversion of brain β-amyloid positivity in amyloid-negative individuals

Data used in this study were obtained from the Alzheimer’s Disease Neuroimaging Initiative (ADNI) cohort. ADNI was launched in 2003 to test whether serial magnetic resonance imaging (MRI), positron emission tomography (PET), other biological markers, and clinical and neuropsychological assessments could be combined to measure the progression of mild cognitive impairment and the early onset of AD. Inclusion and exclusion criteria, clinical and neuroimaging protocols, and other information about ADNI can be found at www.​adni-info.​org. Demographic information, raw neuroimaging scan data, APOE ε4 genotype, and clinical information are publicly available and can be downloaded from the ADNI data repository (www.​loni.​usc.​edu/​ADNI/​). To develop our prediction models, subjects who underwent longitudinal ¹⁸F-florbetapir (AV45) PET tests with a total follow-up duration longer than 6 months were selected (N = 824). Among these subjects, (1) initially amyloid-positive subjects (N = 373) and (2) subjects with a follow-up duration of less than 5 years without conversion to amyloid-positive (N = 222) were excluded. We excluded subjects with a short follow-up duration to avoid false-negative. The cutoff for follow-up duration was determined considering the reported mean follow-up time to conversion from Aβ (−) to Aβ (+) [16]. Finally, among the remaining 229 subjects (135 CN, 92 mild cognitive impairment (MCI), 2 dementia), we defined amyloid-negative subjects who converted positive within 5 years as converters (N = 53) and subjects who remained amyloid-negative for more than 5 years as non-converters (N = 176) (Fig. 1a).

For validation of the prediction model, we recruited subjects who had longitudinal amyloid PET results from the in-house amyloid PET registry of Samsung Medical Center (SMC). A total of 356 subjects had longitudinal Centiloid (CL) data with follow-up duration longer than 6 months, of which 135 were initially Aβ (−). We excluded a subject with a single follow-up visit 8.7 years from the baseline because the long gap between the visits made it difficult to assume the conversion time. After excluding 94 subjects with a follow-up duration of less than 5 years without conversion of Aβ positivity, 40 subjects (6 CN, 28 MCI, 6 dementia) consisting of 10 converters and 30 non-converters were included in the final validation set (Fig. 1b). Because quantification of Aβ burden using the CL method is currently validated and applied only for global uptake data, we validated the prediction model using global SUVR. The institutional review board at SMC approved this study, and informed consent was obtained from the patients and caregivers.

For the analysis of ADNI data, we obtained global and regional ¹⁸F-florbetapir SUVR values from the UCBERKELEYAV45_11_16_21.csv table downloaded from the ADNI website (http://​adni.​loni.​usc.​edu/​). ADNI PET acquisition and processing protocols are described elsewhere (www.​adni-info.​org). Briefly, ¹⁸F-florbetapir images were co-registered to the MRI image of the subject using SPM8. Following co-registration, images were processed using a FreeSurfer pipeline, which includes skull stripping, segmentation, and delineation of cortical and subcortical regions. Then, the volume-weighted florbetapir mean was extracted from each region and the resulting values were intensity normalized with respect to the whole cerebellum. We used 40 cortical regions (18 frontal, 8 cingulate, 8 lateral parietal, and 6 lateral temporal regions) comprising cortical summary regions according to AV45 processing methods available from the ADNI website for model development. To determine amyloid positivity, we used a whole cerebellum-referenced global SUVR cutoff of 1.11 [17].

SMC participants underwent ¹¹C-Pittsburg compound B (PiB), ¹⁸F-Florbetaben (FBB), or ¹⁸F-Fluetemetamol (FMM) PET at Samsung Medical Center using a Discovery STe PET/CT scanner (GE Medical Systems, Milwaukee, WI, USA). Following the protocols proposed by the ligand manufacturers, a 30-min emission PET scan 60 min after the injection of a mean dose of 420MBq of PiB or a 20-min emission PET scan 90 min after the injection of a mean dose of 311.5 MBq of FBB or 185 MBq of FMM was performed. To harmonize uptake values across tracers, we calculated CL values based on previous studies regarding SUVR to CL conversion [18‐20]. We followed the CL pipelines using SPM8, including co-registration and normalization steps using the cortical target region (CTX-VOI) and the whole cerebellum mask for the reference region. The in-house implementation of the standard CL analysis was validated using the GAAIN PiB data website (http://​www.​gaain.​org). We found excellent correlation between CL values (CL_SMC = 1.00 × CL_GAAIN − 0.08, R2 = 0.99) [21], showing our pipeline is valid within the acceptance criteria defined by Klunk et al. [18]. After extracting the SUVR values in CTX-VOI, we converted SUVR to CL using each tracer conversion equation (PiB: CL = 100 × (SUVR_PiB – 1.009)/1.067, florbetaben: CL = 153.4 × SUVR_FBB − 154.9, flutemetamol: CL = 121.42 × SUVR_FMM − 121.16). We used a cutoff value of 20 CL, which was reported to be equivalent to FBP SUVR 1.11 [22], to determine the Aβ positivity of SMC subjects.

Artificial neural network-based models were developed using the PyTorch framework. For the development of the ANN model, we trained the model using the Adam optimizer [23], mean squared error loss function, and ReLU activation function; we also applied batch normalization to prevent internal covariate shift [24]. The grid search approach was used to tune hyperparameters including learning rate, hidden node size, batch size, dropout rate, and weight decay. Stratified 10-fold cross-validation was performed for each model by repeating the random train-validation set splitting 10 times. The models were trained for 100 epochs on graphical processing units (GPUs; NVIDIA GTX 1080Ti).

The performances of the developed classifiers were assessed based on six different metrics: (1) the area under the curve of the receiver operating characteristic (AUROC) curve reflecting the sensitivity and specificity of model predictions; (2) the area under the precision-recall curve (AUPRC), which is a useful performance metric for imbalanced data; (3) sensitivity; (4) specificity; (5) positive predictive value (PPV); and (6) negative predictive value (NPV). The 95% confidence intervals (CIs) of AUROCs and AUPRCs were calculated as well. The averaged values of AUROC and AUPRC were estimated by concatenating all prediction results of each test fold from total 10-fold cross-validation. Sensitivity, specificity, PPV, and NPV values were determined using a classifier threshold of 0.5. After finalizing the model training with the ADNI dataset, external validation with the SMC dataset was performed by loading Model 2. Graphs for the results were plotted using the matplotlib package in Python 3.8.

Efforts to interpret machine learning models have been continued, and model-agnostic methods have been suggested [25‐27]. The feature importance in Model 3 was analyzed regarding model interpretability using the Captum Python library, which can describe internals in the PyTorch-based model [27]. Specifically, we applied the integrated gradient method to estimate the attributes of the prediction of an ANN with respect to certain inputs [28]. We averaged the attribution scores across the test sets to derive a representative value of the feature importance for comparison. For visualization of feature importance, we used the ggseg R package, which can visualize the Desikan-Killiany ROI-wise values.

We compared the characteristics of converters and non-converters using Student’s t-test for continuous variables and the chi-square test for categorical variables. All tests were two-sided and considered statistically significant at p < 0.05. Statistical analyses were performed using the scipy package of Python 3.8.

Table 1 describes the characteristics of the clinical and PET results of converters and non-converters. A total of 229 participants from the ADNI dataset were included in model development. A total of 53 participants (23.1%) were converted to beta-amyloid-positive within 5 years, while 176 remained negative. Statistical tests underlined significant group differences in APOE ɛ4 carrier status and global SUVR (p < 0.05). With respect to SMC participants, a total of 10 out of 40 subjects (25.0%) were converted to Aβ (+) within 5 years, while 30 subjects remained negative. Statistical tests highlighted significant group differences in age and global CL (p < 0.05).

Table 2 reports the performances of each model. The AUROC and AUPRC after 10 times repeated 10-fold cross-validation were described in the mean and 95% CI. Model 3 with features of age, gender, APOE ɛ4 carrier, global SUVR, and regional SUVR demonstrated the highest mean performance of all models after repeated cross-validation: The mean AUROC was 0.841 (95% CI 0.832–0.849) and the mean AUPRC was 0.627 (95% CI 0.610–0.645). Sensitivity of 0.600 (95% CI 0.581–0.619), specificity of 0.869 (95% CI 0.862–0.875), PPV of 0.579 (95% CI 0.562–0.597), and NPV of 0.878 (95% CI 0.873–0.884) were obtained. The fitted hyperparameters at the highest performance were as follows: a first hidden layer of 256 nodes, a second hidden layer of 32 nodes, a batch size of 8, a learning rate of 0.0003, a dropout rate of 0.3, and a weight decay of 0.0001. The inclusion of family history as a predictor did not increase the model performances (results not shown).

Model 2, which was trained with demographic and SUVR data, had a mean AUROC of 0.814 (95% CI 0.806–0.821) and a mean AUPRC of 0.549 (95% CI 0.534–0.564). Model 1, trained without PET data, which included only age, gender, and APOE ɛ4, showed a mean AUROC of 0.674 (95% CI 0.666–0.683) and mean AUPRC of 0.374 (95% CI 0.364–0.384). AUROC curves of the three models are plotted in Fig. 2. DeLong test results showed that Model 3 had higher AUROC than Model 2 (p = 0.003) or Model 1 (p < 0.001). The external validation results using the SMC dataset were as follows: AUROC of 0.900, AUPRC of 0.625, sensitivity of 1.000, specificity of 0.700, PPV of 0.526, and NPV of 1.000. All performance metrics are listed in Table 2.

The feature attribution to the classifier output probability was estimated by loading the best-performing model for each fold in the experimental setup for model 3. Feature-wise averaging of 10-fold results and sorting features positively contributing to the prediction of converters were performed. Accordingly, the top 12 features among those average values are presented in Table 3. The results show features contributing to the prediction of subjects as Aβ converters. The features highly contributing included global SUVR, APOE ɛ4 carrier status, regional SUVRs of the right pars triangularis, left lateral parietal cortex, and left frontal pole. The mean attribution scores are shown in Fig. 3. Overall, the medial and lateral parietal and frontal cortices, as well as superior temporal and cingulate cortices, exhibited high attribution scores.

A few limitations of our study need to be noted. First, the sample sizes of the datasets used in our study were modest. In fact, the ADNI is the cohort with the largest number of longitudinal amyloid PET data. However, the SMC dataset had a smaller number of eligible subjects especially when limited to initially negative subjects who had longitudinal PET data. Despite the limitations of the sample size, we were able to validate our results. Accumulation of amyloid PET data, especially in amyloid-negative individuals, is needed for a more robust validation of our results. Second, while model 3 showed the best performance, the sensitivity and positive predictive value were relatively low. We can adjust the prediction score threshold of the neural network model to find a different balance between specificity and sensitivity depending on the purpose of the prediction model. In contrast to the results in the ADNI cohort, the specificity was relatively low in the SMC cohort. The difference may be attributed to the smaller number of subjects or differences between the cohorts, such as ethnicity, gender, educational attainment, and family history. Since the false positivity may pose ethical challenges in applying the model in clinical trials, the prediction score threshold of the model may have to be adjusted in favor of specificity rather than sensitivity. More importantly, incorporating additional features such as neuropsychological test results, other neuroimaging phenotypes, or genetic factors is needed to improve the overall performance of the model. Third, we could not test the model including regional SUVR with the SMC dataset, although the model showed the best performance in the ADNI cohort. SMC subjects were recruited from a PET registry comprising amyloid PET scans of three different tracers, which forced us to use the CL method for harmonization. Unfortunately, the application of the CL method for regional uptake has not yet been validated. Once the methodology regarding the regional application of the CL is validated, it needs to be tested. Fourth, we used CL values to validate a model developed using global SUVR values. While this was possible since CL values have a strong linear relationship with FBP SUVR [36], further validation studies with harmonized features are warranted. Nevertheless, it is noteworthy that we developed well-performing models for the prediction of Aβ conversion and found important features that should be considered for the selection of primary prevention of AD.