Deep learning-based attenuation correction in the absence of structural information for whole-body positron emission tomography imaging

Figures 1 and 2 outline the schematic workflow for the training and correction stages of the proposed DL-AC method, respectively. For a given pair of NAC PET and corresponding AC PET, 3D patches of NAC PET and AC PET were extracted from NAC PET and AC PET images as training pairs. The 3D patches were extracted by sliding a window (with voxel size of 72  ×  72  ×  32) from NAC PET or AC PET image with an overlap (with voxel size of 60  ×  60  ×  24) between each two neighboring patches. The AC PET patch was used as the deep learning-based target of the NAC PET patch. The goal of DL-AC is to learn the mapping from NAC PET to AC PET to directly generate DL-AC PET images that can reach the same image quality level of an original AC PET image. Since the NAC PET image is contaminated with attenuation and scatter artifacts, training an NAC-PET-to-AC-PET (NAC-to-AC) mapping model is highly under-constrained, meaning artifacts may mislead the mapping. To cope with this issue, first, a CycleGAN architecture (Harms et al 2019, Lei et al 2019a) was applied into DL-AC to enforce the learned mapping from NAC-to-AC to be closer to one-to-one mapping via introducing an inverse AC-PET-to-NAC-PET (AC-to-NAC) mapping and using cycle consistent loss to supervise the two mappings. The two mappings were modeled by two generators. In addition, to increase the realism of the generated DL-AC PET image (synthetic AC), two discriminators were used to judge the realism of the synthetic image produced by the two generators. During correction stage, the patches of a new NAC PET image were fed into the trained NAC-to-AC generator to obtain DL-AC PET patches, and the final DL-AC PET image was reconstructed with patch fusion.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

A retrospective sample of 25 whole-body PET patients was used for training and leave-one-out cross-validation. Each patient has one whole-body PET/CT image dataset. Each dataset contains AC PET, NAC PET and CT that were acquired in one exam. Among the 25 patients, the reasons for exam of eight patients were lung cancer, four lymphoma, four head and neck cancer, 3 for skin cancer, two breast cancer and 4 for abdominal cancer. Fourteen out of the 25 patients contain head in the PET/CT images. An additional cohort of ten whole-body patients, each with three sequential whole-body PET scans separated by approximately one month (30 datasets total) were excluded from training as hold-out test to further evaluate the proposed method. All the 10 patients were lung cancer patients, and 1 out of the ten patients contains head in PET/CT images. All PET data were acquired on a Discovery 690 PET/CT scanner (General Electric, Waukesha, WI) using a clinical 18F-FDG whole-body protocol. Briefly, patients were intravenously administered between 370 (body mass index (BMI)  <  30) and 444 MBq (BMI  >= 30) followed by a 60 min uptake period. Emission data were collected at a bed duration of 1.5 min (BMI  <= 18.5), 2 min (18.5  <  BMI  <  25) or 2.5 min (BMI  >= 25). AC PET Images were reconstructed as static images with a 3D order-subset expectation maximization algorithm (three iterations and 24 subsets) with time-of-flight and all corrections (scatter, randoms, attenuation, normalization and dead time) (Iatrou et al 2004). NAC PET did not include attenuation, scatter and time-of-flight correction. The final reconstructed matrix size was 192  ×  192 with a pixel size is 3.65  ×  3.65  ×  3.27 mm³. The number of slices of PET ranges from 263 to 515.

Figure 3 shows the generator and discriminator network architectures used in the proposed method. As can be seen from figure 3, the network architecture of discriminator is a traditional FCN (Lei et al 2019b), which include several convolution layers followed by max-pooling and a sigmoid layer to obtain binary output. The generator architecture (of both NAC-to-AC and AC-to-NAC) is an end-to-end U-Net including encoding and decoding paths. The encoding path is composed of two convolution layers followed by max-pooling to reduce the feature maps size. The decoding path is composed of two deconvolution layers to obtain end-to-end mapping and a tanh layer to perform the regression. In order to combine the features extracted from encoding path and decoding path, a short connection was used to bypass the features extracted from previous hidden layer to current hidden layer. The short connection was implemented by six residual blocks, since the residual block could lead the feature maps extracted from deep hidden layer to learn the difference of source and target images' distributions. This enforces the DL-AC focus on learning image differences between the NAC PET and AC PET, which would be mainly the attenuation and scatter artifact.

Zoom In Zoom Out Reset image size

Convolutional neural networks (CNN) with residual blocks have achieved promising results in tasks where source and target images are largely similar, much like the relationship between NAC PET and AC PET images. Each residual block includes a residual connection and multiple hidden layers. Through the residual connection, an input bypasses the hidden layers of a residual block, thus these hidden layers are enforced to learn specific differences between input and output, which would be attenuation and scatter artifacts. As shown in generator architecture of figure 3, a residual block is implemented by two convolution layers within residual connection and an element-wise sum operator.

Generative adversarial network (GAN) relies on two sub-networks, a generator and a discriminator, which work in competition with each other. Given a NAC PET and an AC PET image, an initial mapping is learned to be able to generate a DL-AC (synthetic AC) PET image from a NAC PET image. The generator generates a DL-AC PET image that can fool the discriminator to misrecognize the image as an original AC PET image. Conversely, the discriminators' training objective is to decrease the judgment error of the discriminator network, and enhance the ability to differentiate DL-AC PET from AC PET. As these networks are pitted against each other, the capabilities of each improve, leading to more accurate DL-AC PET generation. CycleGAN doubles the process of GAN by enforcing inverse mapping. This doubly constrains the model and can increase accuracy of output images. As shown in figure 1, during training stage, the extracted patches of the NAC PET were fed into the generator (NAC-to-AC) to get the equal-sized DL-AC PET patches. The DL-AC PET was then fed into another generator (AC-to-NAC) to generate NAC PET, referred to as cycle NAC PET. To enforce forward-backward consistency, the extracted patches of the training AC PET are also fed into the two generators to produce a synthetic NAC PET and cycle AC PET.

To train the CycleGAN, the learnable parameters of generators and discriminators were optimized iteratively and in an alternative manner. The accuracy of both networks is directly dependent on the design of their corresponding loss functions. The generator loss consists of an adversarial loss and a cycle consistency loss. The goal of the adversarial loss is to improve the generator to produce the synthetic images that can fool the discriminators via minimizing adversarial losses, which relies on the output of the discriminators, i.e. the distribution of feeding synthetic AC image (generated from NAC-to-AC generator ${{G}_{{\rm NAC}-{\rm AC}}}$)) into the discriminator of AC and the distribution of feeding synthetic NAC image (generated from AC-to-NAC generator ${{G}_{{\rm AC}-{\rm NAC}}}$) into the discriminator of NAC. For clarity, we present only formulation for ${{G}_{{\rm NAC}-{\rm AC}}}$.

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {{L}_{{\rm adv}}}({{G}_{{\rm NAC}-{\rm AC}}},{{D}_{{\rm AC}}},{{I}_{{\rm NAC}}}) ~ ={\rm SCE}\left[{{D}_{{\rm AC}}}\left({{G}_{{\rm NAC}-{\rm AC}}}({{I}_{{\rm NAC}}}) \right),1 \right]\nonumber \end{align} \tag{ 1 }$$

where ${{I}_{{\rm NAC}}}$ denotes the NAC PET image and ${{G}_{{\rm NAC}-{\rm AC}}}({{I}_{{\rm NAC}}})$ is the output of the NAC-to-AC generator, i.e. the DL-AC (or synthetic AC). ${{D}_{{\rm AC}}}$ is the AC discriminator which is designed to return a binary value indicating whether a distribution is real (from AC) or fake (from synthetic AC). The function ${\rm SCE}\left(\cdot ,1 \right)$ is the sigmoid cross entropy between the distribution output of discriminator and a unit.

The cycle consistent loss is computed as the combination of the mean squared error (MSE) and gradient difference error (GDE) between the original images and the cycle images. The MSE loss forces the generator to synthesis AC images with accurate voxel intensity to a level of ground truth AC images. The GDE loss forces the synthetic AC images' gradient structure to a level of ground truth AC images.

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {{L}_{{\rm cyc}}}\left({{G}_{{\rm NAC}-{\rm AC}}},{{G}_{{\rm AC}-{\rm NAC}}},{{I}_{{\rm NAC}}},{{I}_{{\rm AC}}} \right)=\begin{matrix} {\rm MSE}\left[{{G}_{{\rm AC}-{\rm NAC}}}\left({{G}_{{\rm NAC}-{\rm AC}}}({{I}_{{\rm NAC}}}) \right),{{I}_{{\rm AC}}} \right] \nonumber \\ +\lambda \cdot {\rm GDE}\left[{{G}_{{\rm AC}-{\rm NAC}}}\left({{G}_{{\rm NAC}-{\rm AC}}}({{I}_{{\rm NAC}}}) \right),{{I}_{{\rm AC}}} \right] \nonumber \\ \end{matrix}\nonumber \end{align} \tag{ 2 }$$

where $\lambda$ is a parameter which control the balance of MSE and GDE loss for cycle consistency. ${{G}_{{\rm AC}-{\rm NAC}}}\left({{G}_{{\rm NAC}-{\rm AC}}}({{I}_{{\rm NAC}}}) \right)$ is the output of first feeding ${{I}_{{\rm NAC}}}$ into the generator ${{G}_{{\rm NAC}-{\rm AC}}}$ and then feeding the output into the generator ${{G}_{{\rm AC}-{\rm NAC}}}$, namely the output of this term denotes the cycle NAC. The parameter $\lambda$ was set to 10 in this work.

Finally, the optimization of generator is obtained by

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {{G}_{{\rm NAC}-{\rm AC}}},{{G}_{{\rm AC}-{\rm NAC}}}= ~ \underset{{{G}_{{\rm NAC}-{\rm AC}}},{{G}_{{\rm AC}-{\rm NAC}}}}{\mathop{{\rm arg}\,{\rm min}}}\,\left\{\begin{matrix} {{L}_{{\rm adv}}}({{G}_{{\rm NAC}-{\rm AC}}},{{D}_{{\rm AC}}},{{I}_{{\rm NAC}}})+{{L}_{{\rm adv}}}({{G}_{{\rm AC}-{\rm NAC}}},{{D}_{{\rm NAC}}},{{I}_{{\rm AC}}}) \nonumber \\ {{L}_{{\rm cyc}}}\left({{G}_{{\rm NAC}-{\rm AC}}},{{G}_{{\rm AC}-{\rm NAC}}},{{I}_{{\rm NAC}}},{{I}_{{\rm AC}}} \right)+{{L}_{{\rm cyc}}}\left({{G}_{{\rm AC}-{\rm NAC}}},{{G}_{{\rm NAC}-{\rm AC}}},{{I}_{{\rm AC}}},{{I}_{{\rm NAC}}} \right) \nonumber \\ \end{matrix} \right\}.\nonumber \end{align} \tag{ 3 }$$

The optimization of discriminator is obtained by

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {{D}_{{\rm AC}}},{{{\rm D}}_{{\rm NAC}}}=\underset{{{D}_{{\rm AC}}},{{{\rm D}}_{{\rm NAC}}}}{\mathop{{\rm arg}\,{\rm min}}}\,\left\{\left. \begin{matrix} {\rm SCE}\left[{{D}_{{\rm AC}}}\left({{G}_{{\rm NAC}-{\rm AC}}}({{I}_{{\rm NAC}}} \right)),0 \right]+ ~{\rm SCE}\left[{{D}_{{\rm AC}}}\left({{I}_{{\rm AC}}} \right),1 \right]\quad \nonumber \\ {\rm +SCE}\left[{{D}_{{\rm NAC}}}\left({{G}_{{\rm AC}-{\rm NAC}}}({{I}_{{\rm AC}}} \right)),0 \right]+ ~{\rm SCE}\left[{{D}_{{\rm NAC}}}\left({{I}_{{\rm NAC}}} \right),1 \right] \nonumber \\ \end{matrix} \right\}. \right.\nonumber \end{align} \tag{ 4 }$$

To supervise the generators and discriminators via the proposed loss functions, Adam gradient optimizer with learning rate of 2  ×  10⁻⁴ was used for optimization. The batch size was set to 20. The number of training iterations was set to 8.6  ×  10⁴. The proposed algorithm was implemented by Python 3.7 and TensorFlow as in-house software on a NVIDIA Tesla V100 GPU with 32GB of memory.

To evaluate the reliability of the proposed method on predicting the quantification changes over time, we calculate the mean error (ME), normalized mean square error (NMSE), peak signal to noise ratio (PSNR) and normalized cross correlations (NCC) metrics between DL-AC PET and AC PET on the evaluation dataset. These metrics were calculated within whole-body volume and within contoured organs, such as brain, lung, heart, left and right kidney and liver. The calculation of these metrics is as follows:

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {\rm ME}=\frac{\sum\nolimits_{i\in V}\left({{I}_{{\rm AC}}}\left(i \right)-{{I}_{{\rm DL}-{\rm AC}}}\left(i \right) \right)}{\sum\nolimits_{i\in V}{{I}_{{\rm AC}}}\left(i \right)}\nonumber \end{align} \tag{ 5 }$$

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {\rm NMSE}=\frac{\sum\nolimits_{i\in V}{{\left({{I}_{{\rm AC}}}\left(i \right)-{{I}_{{\rm DL}-{\rm AC}}}\left(i \right) \right)}^{2}}}{\sum\nolimits_{i\in V}{{\left({{I}_{{\rm AC}}}\left(i \right) \right)}^{2}}}\nonumber \end{align} \tag{ 6 }$$

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {\rm PSNR}=10{\rm lo}{{{\rm g}}_{10}}\left(\frac{N\cdot {{\max}_{i\in V}}\left({{I}_{{\rm AC}}}\left(i \right),{{I}_{{\rm DL}-{\rm AC}}}\left(i \right) \right)}{\sum\nolimits_{i\in V}{{\left({{I}_{{\rm AC}}}\left(i \right)-{{I}_{{\rm DL}-{\rm AC}}}\left(i \right) \right)}^{2}}} \right).\nonumber \end{align} \tag{ 7 }$$

ME and NMSE are averaged over all the voxels, $i$, inside the contoured organs or whole-body volume, V, with N the total number of voxels inside the volumes. ${{I}_{{\rm AC}}}\left(i \right)$ and ${{I}_{{\rm DL}-{\rm AC}}}\left(i \right)$ are PET intensities after AC and the proposed DL-AC, respectively. ${\rm ma}{{{\rm x}}_{i\in V}}(\cdot)$ is the max intensity inside the delineated volume.

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {\rm NCC}=\frac{\sum\nolimits_{i\in V}C\cdot \left({{I}_{{\rm AC}}}\left(i \right)-{\rm mean}\left({{I}_{{\rm AC}}}\left(i \right) \right) \right)\cdot \left({{I}_{{\rm DL}-{\rm AC}}}\left(i \right)-{\rm mean}\left({{I}_{{\rm DL}-{\rm AC}}}\left(i \right) \right) \right)}{{\rm std}\left({{I}_{{\rm AC}}} \right)\cdot {\rm std}\left({{I}_{{\rm DL}-{\rm AC}}} \right)}\nonumber \end{align} \tag{ 8 }$$

where ${\rm mean}\left(\cdot \right)$ and ${\rm std}\left(\cdot \right)$ calculates the mean and standard deviation (STD) intensity among selected voxels. The intensity profiles of AC PET and DL-AC PET which plot the intensities of voxels one by one along a line in the image volume, are also shown in one figure to qualitatively demonstrate their differences.

In order to quantify the lesion detectability, we calculated and compared the signal-noise-ratio (SNR) and contrast-noise-ratio (CNR) of the lesion in both AC PET and DL-AC PET (Qi 2001, Bao and Chatziioannou 2010, Schaefferkoetter et al 2017). The SNR is defined as

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {\rm SNR}=\frac{{\rm mean}\left(I\left(i \right) \right)}{{\rm std}(I\left(i \right))},\nonumber \end{align} \tag{ 9 }$$

where $I\left(i \right)$ are the voxels in the lesion of AC PET or DL-AC PET. The CNR is defined as

$$\begin{align} \newcommand{\e}{{\rm e}} \displaystyle {\rm CNR}=\frac{{\rm mean}\left(I(i) \right)-{\rm mean}\left({{I}_{b}}\left(i \right) \right)}{{\rm std}(I\left(i \right))},\nonumber \end{align} \tag{ 10 }$$

where ${{I}_{b}}\left(i \right)$ are the intensities of voxels in the 2 mm margin around the lesion which are considered as background.

Leave-one-out cross-validation experiments were performed with 25 patients, each of which has one PET/CT image dataset. For each experiment, 24 sets of images were used for training and the remaining 1 set for validation. The experiment repeated 25 times to make each set used as test data exactly once. A separate hold-out test was also used to evaluate the proposed method, we trained the model by previous 25 patients and test the model by an additional ten patients, each of which includes three sequential scans for a total of 30 datasets. The change between two sequential PET scans is of clinical interest which may indicate the treatment outcomes or tumor growth. We calculated such change using both AC PET and DL-AC PET and compared the two results in order to demonstrate the reliability of the proposed method in quantifying the change between two sequential PET scans. Volumes of interest (VOIs) were manually delineated over structures of lung, heart, liver, bilateral kidneys and brain if patient dataset contains, based on CT, and lesions based on PET as indicated in patient's clinical report. The whole-body, which is defined as the region within patient body, was also segmented based on CT as a VOI to quantify the general performance across the whole patient body. Statistic T-test was also performed to quantify the statistically difference between AC PET and DL-AC PET images in the sequential scan cohort.

In order to compare the proposed method with other state-of-art learning based method in PET AC, we implemented UNET (Van Hemmen et al 2019) and GAN (Kurz et al 2018) using the same datasets. The UNET method's architecture is a deep convolutional encoder-decoder network structure with 50% dropout, batch normalization, and max pooling (Van Hemmen et al 2019). The GAN method's architecture consists a generator architecture, which is U-shaped deep convolutional neural network (U-Net-like), to synthesize DL-AC from NAC PET, and a discriminator architecture, which is a fully convolutional network, to judge the realism of synthesized DL-AC (Kurz et al 2018). Compared with UNET and GAN, CycleGAN could enforce the mapping of NAC-to-AC to be closer to one-to-one mapping, although NAC-to-AC mapping is an ill-posed problem. Specifically, we trained both UNET and GAN using the 25 patients and tested on the ten patients with sequential scans as we did for CycleGAN. Same quality metrics and VOIs were also used in statistical analysis.

Figure 4 shows a comparison of images from the AC PET and the proposed DL-AC PET results on one patient as test dataset among the 25 patients in the leave-one-out cross-validation study. The images generated with the DL-AC method show excellent resemblance to the AC PET images. More explicit comparison is illustrated in figure 5, where the profiles of the DL-AC PET data agree well with those of AC PET. The joint histogram of voxels from the whole-body region is close to identity line indicating good intensity agreement between reference AC PET and our generated DL-AC PET. Figure 6 shows representative plans of brain, lesion in lung and kidneys of AC PET and DL-AC PET and their difference maps, in addition to those of liver shown in figure 4. Output of the validation metrics are listed in table 1. The ME on brain, lung, heart, kidneys, liver and lesion are all less than 4%, with NMSE less than 1.5%. The ME and NMSE on whole body are  −0.01%  ±  2.91% and 1.21%  ±  1.73%. The NCC is close to identity, demonstrating excellent intensity similarity between the AC PET and DL-AC PET. For comparison, the NCC between DL-AC PET and NAC PET is 0.644  ±  0.214 in whole body region. PSNR are all around or larger than 30 dB.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Figure 7 shows the side-by-side comparison of AC PET and DL-AC PET on one patient from a representative sequential PET scan and figure 8 plots the cranial-caudal profile and whole-body joint histograms from the same data. Excellent agreement is observed in both PET image and profile comparisons. The joint histogram shows an intensity distribution that is close to the line of identity, indicating good agreement between reference AC PET and the DL-AC PET from proposed method. Table 2 lists the average quantification results across all scans. Because only one set of images included head in the scan, brain was excluded from the statistical analysis. As indicated in table 2, the ME and NMSE generated with the proposed method are less than 3.5% on all contoured organs except lung. The NCCs are close to identity. For comparison, the NCC between DL-AC PET and NAC PET is 0.618  ±  0.168 in whole body region. The PSNR are all larger than 30dB. Among the 30 datasets, the lesion SNRs on AC PET and DL-AC PET are 4.188  ±  3.086 and 4.143  ±  3.047, respectively, with p -value  =  0.393, and the lesion CNRs are 1.548  ±  0.787 and 1.528  ±  0.768, respectively, with p -value  =  0.350. Thus there is no statistical significance in lesion SNR or CNR between AC PET and DL-AC PET.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Each patient received three sequential PET scans, therefore we calculated the relative PET intensity changes of the second scan over the first scan, the third over the first and the third over the second to evaluate the reliability of the proposed method. The results are summarized in table 3. Note that the STD in some VOIs such as lesion is large because the intensity change between sequential scans varies a lot from patient to patient. Figure 9 illustrates joint histogram of percentage changes calculated on the contoured organs and lesions with AC PET and DL-AC PET. The linear regressions are almost identical to the line of identity, which indicates excellent agreement of the intensity distributions captured on AC PET and DL-AC PET. The average difference of the percentage changes calculated with the two methods is less than 3% on all contoured volumes. The p -values using t-test (last column in table 3) are all larger than 0.05, indicating no statistically significant difference between the two methods.

Zoom In Zoom Out Reset image size

Table 4 lists the average quantification results across all scans among the 10 patients of 30 scans using UNET and GAN and is compared with those of CycleGAN, the proposed method, from table 2. Both UNET and GAN show inferior performance in most metrics and VOIs and larger variation among patients. They also have much large bias found in lesion than the proposed method.