Quantitation of PET signal as an adjunct to visual interpretation of florbetapir imaging

The software packages used in this study, MIM (MIMneuro®), Siemens (Siemens syngo.PET Amyloid Plaque) and Hermes (Hermes Brain Analysis Software Suite™ BRASS, 2.0; CE 0413), are all commercially available and approved in the US and EU for visual examination and quantitation of PET images, with specific routines designed to quantitate ¹⁸F-florbetapir PET images. Although the individual packages use different proprietary algorithms to perform the quantitation, the three packages share the following features:

A total of 80 physicians participated as scan readers in this study. The study was conducted in three separate replications in different cohorts of readers using the three different software packages (MIM, Siemens, Hermes). The MIM and Siemens replications (NCT 01946243) were performed with US physicians at ACR Image Metrix, Philadelphia, PA. The Hermes replication (NCT 02107599) was performed with readers from Spain and the UK at Bioclinica, Inc. (Leiden, The Netherlands). For each replication (MIM, Siemens, Hermes), imaging physicians who had completed a florbetapir PET reader training course were contacted at random and invited to participate. Physician readers were excluded from the study if they had more than minimal experience with or had previously been trained personally to perform quantitation of amyloid PET.

For each replication, readers who met the above qualifying criteria were invited to the testing facility in cohorts of three to ten readers to complete day 1 (visual read) and day 2 (quantitative read). The testing continued in each replication until a minimum of seven readers with visual read accuracy ≤90% (below-average readers; accuracy less than the mean accuracy expected based on previous studies) and a minimum of five readers with visual read accuracy >90% (above-average readers) were recruited.

Upon arrival at the core laboratory read facility, all readers underwent a brief refresher training utilizing portions of the online (US) reader training program, highlighting the steps for visual interpretation and criteria for determining a scan as positive or negative for amyloid plaques. The core laboratory provided training on the respective software to facilitate visual reads, and readers practiced with nine image sets under supervision. The readers then independently visually interpreted a test group of 20 florbetapir PET scans (without supervision). These interpretations served as a practice exercise and were not used in the primary or secondary analyses, nor were these results used to disqualify readers from the study.

All readers then underwent training related to the use of quantitation with florbetapir PET images. Training consisted of teaching the operation of the quantitative software, and the method for generation of SUVr values. The readers were shown the validation of the research quantitation method in autopsy-verified cases [7] and the relationship between the quantitation results from the research method and the results from the respective commercial quantitation package [23] (see also Online Resource 1), which allowed them to estimate the approximate SUVr values associated with a positive scan. Readers were then taught the principles for applying quantitation as an adjunct to visual interpretation, including algorithms for comparing the quantitative results to their initial visual interpretation. The training included supervised practice of the visual with adjunct quantitation (VisQ) interpretation approach on the same nine sample cases used for the initial practice of visual interpretation.

On day 1 of the study, the readers visually interpreted 96 florbetapir scans comprising the 46 autopsy-verified scans [7], and 50 randomly selected scans from a trial with patients seeking a diagnosis for cognitive impairment [24]. The readers did not have access to quantitation tools during this reading session.

On the following day (day 2), readers in the MIM and Hermes replications were presented these same 96 florbetapir PET scans for interpretation using the VisQ approach. The readers obtained SUVr values for the predefined ROIs, as well as an overall cortical average SUVr using the respective quantitation software in accordance with the software manufacturer’s instructions. For each scan, the reader had the opportunity to review their previous interpretation based on visual assessment alone and was then asked to make a final read interpretation using the VisQ interpretation principles. In addition to the final interpretation, the SUVr values for the individual regions and the average SUVr value were recorded.

For the Siemens replication (on day 2), readers were randomized to either an experimental arm (VisQ) or a control arm (VisVis). Procedures in the experimental arm were identical to those described for the MIM and Hermes replications above. For the readers randomized to the control arm, the only difference was that they were not allowed to use the quantitative software or the VisQ approach during the second review of the 96 florbetapir PET cases; these readers had the opportunity to review their previous interpretation (Aβ+ or Aβ−) based on visual assessment alone and were then asked to make a final read interpretation using only the visual interpretation method (hence VisVis). This condition was intended to control for any learning or other benefit derived from reviewing the scans a second time. A diagram of the study design is shown in Fig. 1.

The images used in this study included florbetapir PET scans from 46 end-of-life patients recruited from hospice, long-term care facilities and community healthcare facilities who came to autopsy within 1 year of their scan in the florbetapir pivotal trial [7] and 50 scans randomly selected from a previous study of florbetapir use in patients with diagnostic uncertainty [24] (Table 1). In general, the patients seeking a diagnosis for cognitive impairment were younger, included more mildly impaired patients, and a lower proportion of patients with AD and other non-AD dementia than the end-of-life patients. Both previous studies were approved by the relevant institutional review boards and subjects or other family members of subjects contributing PET scans used in these studies gave written informed consent. All florbetapir PET scans used in these studies were acquired under standard methods described previously [7, 24]. A 10-min PET acquisition was performed approximately 50 min after administration of approximately 370 MBq (10 mCi) of ¹⁸F-florbetapir. Images were acquired and reconstructed with iterative or maximum likelihood algorithms with a postreconstruction gaussian filter. Images were displayed for visual interpretation, and quantitation was performed using the MIM, Siemens, or Hermes software in accordance with the respective replication.

The initial visual interpretation was performed in accordance with the instructions in the ¹⁸F-florbetapir package insert. Briefly, images were reviewed using a black-and-white palette (gray scale) with the maximum intensity of the scale set to the maximum intensity brain pixel. Starting at the bottom of the brain, primarily in transaxial orientation, the cerebellum (presumed amyloid-free normal tissue) was examined followed in succession by the temporal lobes and occipital cortex, the prefrontal cortex and parietal lobes. A scan was defined as positive (Aβ+) if at least two regions contained areas with reduced gray–white matter contrast, or if at least one region had an area of gray matter uptake more intense than the adjacent white matter uptake. Readers recorded an interpretation as either Aβ− (indicative of no or sparse neuritic plaques) or Aβ+ (moderate to frequent plaques). For positive scans the regions of positivity were also recorded.

Image quantitation involved spatial normalization of the ¹⁸F-florbetapir PET scan into a standard coordinate system, application of predefined ROIs, checking the quality of the normalization and application of the ROI, and refitting of the image if applicable for a software package. A series of cortical target region to whole cerebellum count ratios (SUVr) and a cortical average SUVr were then generated.

Readers were instructed to use quantitation as an adjunct to the visual read, not as an alternative. Thus, they compared the calculated cortical average SUVr to the expected range for Aβ+/Aβ− scans for the particular software package. If the quantitative result was consistent with the initial visual read, the readers were expected not to change their initial interpretation. In the event of apparent disagreement between visual interpretation and quantitation, readers were instructed to perform the following actions. First, the readers checked the spatial normalization and fit of the scan to the template. They confirmed the accuracy of the placement of the ROIs, checking for cerebrospinal fluid or bone within the ROI, and evaluated the potential impact of atrophy or ventriculomegaly on quantitation. Next they reviewed the basis for making a visual Aβ+ or Aβ− determination. They looked for loss of gray–white contrast in at least two regions or intense uptake in one region. In the case of an Aβ+ initial visual read and an apparent Aβ− quantitation, readers were instructed to consider whether the positive visual interpretation might be based on tracer retention in regions other than the six ROIs that contribute to the composite SUVr (e.g., intense tracer retention in the occipital lobe could support an Aβ+ visual determination but would not contribute to the SUVr). In the case of an Aβ− initial visual read and an Aβ+ quantitation, readers visually examined the regions corresponding to the ROIs with elevated SUVr to confirm whether there was a loss of gray–white contrast in these areas. Finally, the readers visually examined the cerebellum region, confirming the fit of the ROI (which can affect the denominator of the SUVr) and the level of gray–white contrast (which provides a standard for comparison to the cortex), and looking for possible structural anomalies (e.g., stroke) that could influence quantitation of the cerebellar region. The final interpretation was then based on a visual read augmented by quantitative information.

The prespecified primary efficacy hypothesis in all three replications was that addition of quantitative information (VisQ method) would significantly improve overall accuracy of florbetapir PET scan interpretation. The primary analysis utilized the 46 images from patients who received a florbetapir scan within 1 year of autopsy. The neuropathologist’s diagnosis, that was based on the modified Consortium to Establish a Registry for Alzheimer’s Disease (CERAD) plaque score, was used as the truth standard such that an image was considered correctly interpreted as Aβ+ when there were moderate or frequent plaques and Aβ− when there were no or sparse plaques as previously described [7]. In the MIM and Siemens replications, the primary analysis population comprised those readers with a visual read (day 1) average accuracy of 90% or less based on historical studies. Paired t tests were used to determine whether accuracy (percent agreement with the truth standard) increased between the day 1 visual interpretation and the day 2 interpretation incorporating quantitative information (VisQ). In the Hermes replication, all readers were included in the primary analysis and the net reclassification index (NRI) [25] (see Online Resource 3 for statistical description) was used to evaluate differences in accuracy between the day 1 visual interpretation and the day 2 VisQ interpretation.

Since the three replications were designed similarly, the results were also integrated to assess the possible benefit from the addition of quantitative information, across all readers. The integrated analyses were done in two ways. In the first analysis, for scans from the autopsied patients [7], the neuropathologist’s diagnosis was used as the truth standard, as specified above, and the changes in reader accuracy in the VisQ group between the day 1 visual interpretation and the day 2 interpretation incorporating quantitative information were compared with the changes between the day 1 and day 2 interpretations in the VisVis group (control group from Siemens software) that did not involve the use of quantitative information and performed on both day 1 and day 2. An analysis of covariance (ANCOVA) model was used for this comparison, adjusting for readers’ day 1 accuracy, and replication. Secondary analyses were also performed looking at sensitivity and specificity relative to the truth standard.

In the second analysis, because an autopsy-based truth standard was not available for the scans from patients seeking a diagnosis for cognitive impairment [24], the majority VisQ interpretation of the three readers with the best interpretation accuracy on the autopsy-verified scans was used as the reference standard. The majority interpretation from these readers (coincidentally all from the Siemens replication) was 100% accurate relative to the neuropathologist’s diagnosis of the autopsied patients. All three readers agreed on 45 of 50 scans from patients lacking autopsy truth standard. A sensitivity analysis was performed excluding these five cases to ensure that they did not influence the results. Thus, the best three readers’ majority interpretation was considered a reasonable reference standard for the scans from the non-autopsy patients.

The interreader reliability of scan interpretation by the VisQ and VisVis methods was also assessed using Fleiss’ kappa statistics for both day 1 and day 2. In addition, for each read method, a change in interreader reliability was calculated as the kappa value based on day 2 interpretations minus the kappa value for the day 1 interpretations. The 95% confidence interval around this difference was calculated using a bootstrap method [26]. If the lower bound of this 95% confidence interval was greater than 0, then a statistically significant improvement on interreader consistency for day 2 interpretations over day 1 interpretations was demonstrated. Percent agreement is also provided to assess the interreader reliability, calculated as the number of reader pairs who agreed when interpreting the same scan, divided by all possible pairs of readers for the same scan. A logistic regression model with robust variance estimation by a generalized estimated equation was used to compare the change in percent agreement between day 1 and day 2 for the VisQ and VisVis methods. Finally, reader confidence (low, medium, high) was recorded on day 1 and day 2 and the changes from day 1 to day 2 in the VisQ and VisVis groups were compared using the Wald chi-squared test from a proportional odds model.