NiftyPET: a High-throughput Software Platform for High Quantitative Accuracy and Precision PET Imaging and Analysis

Since the bed cannot be imaged with MR, high resolution CT images of the patient bed (table) and the MR coils are supplied with the scanner. The images of different hardware components can be found in the scanner’s file system with hardware components such as the head and neck coil, the spine coil and the patient bed. The specific part of the hardware μ-map for any given image study has to be separately generated based on additional information about the table position relative to the iso-centre of the scanner. Depending on the imaging settings, only some parts of the hardware are in the field of view and only those parts have to be included in the μ-map by appropriate image resampling of the high resolution CT-based images into the PET space.

In addition to the hardware μ-map, information about the object’s electron density is required for attenuation correction. For brain imaging, the software offers a choice between the μ-map provided by the scanner or the μ-map generated using the pseudo-CT (pCT) synthesis from T1 and T2 weighted (T1w/T2w) MR images (Burgos et al. 2015). The multi-atlas CT synthesis method provides a significant improvement in PET quantitative accuracy when compared to the ultra-short echo time (UTE)-based attenuation correction (as provided by the vendor). In this method, atlases of multiple pairs of aligned MR and CT images from different subjects are used to generate a synthetic CT image, called also a pCT image in the original, target MR space. It is possible to generate pseudo-CT images using a web application: http://​cmictig.​cs.​ucl.​ac.​uk/​niftyweb/​program.​php?​p=​PCT with the output image in NIfTI format, in the T1w/T2w image space. The CT values, which are expressed in HU, are converted to linear attenuation coefficients using a piecewise linear transformation (Burger et al. 2002).

Since it is likely that patient motion will occur between the T1w/T2w and PET acquisitions, the software allows creation of a reference PET image (an average image or any dynamic time frame), which is reconstructed without attenuation correction, and to which the MR images are then registered (Modat et al. 2014). The resulting transformation is used to resample the pCT image into the correct position and resolution of the PET image of the time frame.

To achieve high throughput processing, the projection and image data are reorganised in the device memory allowing high bandwidth (coalesced) memory access. The image and projection data are rearranged such that the fastest changing index for the image and projection data is along axial direction. Therefore, this leads to axially-driven calculations, which allow more efficient use of the L2 cache. Consider an axial image row (127 voxels for the Biograph mMR) of fixed transaxial position as shown in Fig. 3b and a pair of transaxial detectors forming a set of possible LORs intersecting the image row. The axial intersections for these voxels are calculated for all oblique and direct sinograms, after combining them with a single pre-calculated transaxial intersection length for one such axial row (cf. Fig. 3a). This process is then repeated for all voxels being intercepted by the chosen LOR, which leads to the projection data being stored consecutively along the diagonals of the Michelogram and thus ensuring coalesced load and store operations (NVIDIA 2017a) (Fig. 3c).

Since the projection paths in the cylindrical scanner geometry vary depending on the radial projection position, the computational load will vary correspondingly for each projection bin (Hong et al. 2007). Therefore, to keep threads well-balanced with similar workload, and thus maintaining high throughput by minimising idle threads, the projection data are first sampled along sinogram angles followed by the radial projection sampling. The reason for this is that the same radial projection position will have similar intersection length through a circular FOV for any sinogram angle, and hence threads with similar calculation load are bundled together and executed in parallel more efficiently.

Forward and back projections are executed in three parts: (i) the 64 direct sinograms are executed by a grid of 68516 CUDA blocks, whose number corresponds to the total number of sinogram bins without crystal gaps and each block is comprised of 64 threads (see documentation (NVIDIA 2017a) for details); (ii) the next 1024 oblique sinograms are executed by a grid of 68516 CUDA blocks, each with 1024 threads (maximum number of threads per block for NVIDIA architectures with a compute capability of 3.5); (iii) the remaining oblique sinograms are executed with the same parameters as (ii). Forward projection takes around 3 seconds on the NVIDIA K20 Tesla. The back projection takes 3 seconds more due to the resolving of race hazards using the CUDA atomic operations (race hazards are created by multiple projection bins accessing the same image voxel at the same time). Currently, no symmetries are used for speeding up the calculations as in Hong et al. (2007), but such an approach is under development.

For quantitative imaging, the random coincidences have to be accurately measured for each detector pair. For the Siemens Biograph mMR scanner, the random coincidences are measured using the delayed time window method (Meikle and Badawi 2005) (p.96), in which the true and scatter coincidences are eliminated from such a delayed acquisition, leaving only the estimate of randoms. Since such estimates can be very noisy, especially for short time frames (Hogg et al. 2002) (cf. Fig. 7b in “Results and Discussion”), we implemented a maximum likelihood method for reducing the noise of the random events estimates.

The rate of measured random events between detectors i and j within the time coincidence window 2τ can be approximated using singles rates S _i and S _j : Since the random events follow Poisson statistics, the expected values of the estimated random data are found using the maximum likelihood (ML) approach based on (3) (Panin et al. 2007): where \({\sum }_{j{\in }\mathbb {J}_{i}}d_{ij}\) are the fan sums found while processing the list-mode data for delayed events (for more details see “List-mode Data Processing” and Markiewicz et al. (2016a) and Panin et al. (2007)). The fan sums in the denominator of Eq. 4 are calculated at each iteration on the GPU exploiting inter-thread communication for very fast reductions using CUDA shuffle instructions (Luitjens 2014). The 3D fan sums are found first for axial fans as shown in Fig. 4 and then for transaxial fan sums for any given detector i. The random event sinograms in span-1 are found by applying (3) to each sinogram bin (span-11 sinograms are found by reducing span-1 sinograms accordingly).

Consider a positron emission at E giving rise to a photon pair emitted such that one photon is detected unscattered at A while the other one is incident on scattering patch S. The unscattered photon trajectory (\(\hat {\mathbf {a}}=-\hat {\mathbf {u}}\)) is defined by detector A and emission location E (shown in the sagittal plane in Fig. 5a). The probability of photons being incident on S is: where ε _A is the geometric efficiency of detecting a photon emitted at E (specified by position vector e), r _A is the distance between E and the opposing detector along path \(\hat {\mathbf {u}}\) and r _S is the distance between E and the beginning of patch S along path \(\hat {\mathbf {a}}\). Each scatter patch is represented by a single point, s = (s _x ,s _y ,s _z ), from which photons are assumed to be scattered. Therefore, the absolute probability that a photon emitted around E will scatter at S (i.e., along the length l _S of the scattering patch) while the other photon will be received unscattered at crystal A is given by The probability of photons scattering from S towards a given detector B is found using the Klein-Nishina (K-N) cross-section, dσ _e/dΩ, for unpolarised radiation and the solid angle Ω _B subtended by detector B at S, leading to the total absolute probability P _s(B S E A) that a positron emitted at E will result in an unscattered photon detected at A and the paired scattered photon at B: where σ _e is the total K-N electronic cross-section, c _B is the factor accounting for the changed photon energy after scattering towards B. The 3D scatter response to emission point E is found by accounting for all detectors receiving unscattered photons. This procedure is repeated for all the possible emission voxels to estimate the full scatter sinogram.

In this model, a sub-sample K _S of all available detectors K are considered to receive the unscattered photons and their paired scattered photons, all emitted at the vicinity of any point E. The model requires as an input the current estimate of radioactivity distribution and the μ-map images. Due to the efficient and high-throughput computational model, both images can be represented in high resolution, allowing high accuracy and precision. For ¹⁸F-florbetapir radiotracer, the μ-map was down-scaled by a factor of two (from [127 × 344 × 344] to [63 × 172 × 172] using 4 mm³ voxels) to allow high accuracy of photon attenuation calculations. The radioactivity image was down-scaled independently from the μ-map by a factor of 3, resulting in [43 × 114 × 114] images. The independent scaling allows for greater control of the trade-off between accuracy and computational time.

One of the advantages of this voxel-driven approach is its suitability for multi-level parallel GPU computing (or using other parallel computing architectures), with all the emission voxels being separated and calculated independently in the top level of parallelism. Then, in the lower level of parallelism, all the detectors receiving unscattered photons are considered separately: first axially with 8 detector rings out of all 64 rings (resulting in 1:8 axial sampling) and then transaxially with 64 detectors out of 448 (resulting in 1:7 transaxial sampling, similar to the axial sampling using the Siemens Biograph mMR geometry). The next (lower) level of parallelism is used for calculating the paths of scattered photons detected by 8 axial rings and 32 transaxial detectors, which form a 3D fan of the basic scatter distribution originating at scattering point S on trajectory \(\hat {\mathbf {a}}\) and ending on the detector ring (Fig. 5). The lowest level of parallelism is employed on each CUDA warp (a group of 32 CUDA threads scheduled and executed simultaneously) with fast reductions using shuffle instructions introduced in the NVIDIA’s Kepler architecture, facilitating rapid tracing of photon rays through the attenuating medium (NVIDIA 2012).

The last step is to scale the scatter sinogram to the prompt data to account for multiple scatter and scatter from outside the FOV. Since all the quantitative proportions between scatter sinograms are maintained within the 3D model, the scaling factors are obtained using the weighted least squares method applied to reduced sinograms for high count levels. Finally, this process is followed by scatter specific normalisation in span-1 or span-11 (cf. “Axial Normalisation Factors”).

The proposed scatter model was validated using Monte Carlo simulation toolkit SimSET (Lewellen et al. 1998) with two setups: (i) a point source within an attenuating medium and (ii) simulated amyloid ¹⁸F-florbetapir brain scan (Fig. 5b). In both cases the geometry of the Siemens Biograph mMR was used. The μ-map for both setups is taken from a real amyloid brain scan, including the patient’s head and neck, the table and the head coil. The location of the simulated point source is marked with point E in Fig. 5b, whereas the whole brain simulated radioactivity is taken from a real reconstructed ¹⁸F-florbetapir brain scan. In the case of the point source, a total of 2 × 10¹⁰ events were simulated. For the brain scan, a total of 3 × 10¹⁰ events were simulated across the whole brain.

Partial volume correction (PVC) is applied in order to improve both the qualitative and quantitative aspects of the reconstructed PET images by correcting for the degrading effects of the limited spatial resolution. While many PVC methods have been proposed (see e.g., Erlandsson et al. 2012 for review), here we have chosen to implement the “iterative Yang” (iY) method, an iterative version of a method proposed by Yang et al. (1996). This method utilises a segmented anatomical image to correct for the spill-over between different regions on a voxel-by-voxel basis, as modelled by the PSF of the scanner. There is no correction for blurring between voxels within the same region, however. For these reasons the method does not suffer from the excessive noise-amplification and ringing artefacts associated with standard de-convolution algorithms, and produces results similar to those of the RBV method (Thomas et al. 2011). The iY method can be described as follows with and where \(\hat {f}^{(k)}(\mathbf {x})\) is the corrected image after k iterations, g(x) is the original image, h(x) is the PSF of the system, I _i (x) is the indicator function for region i, N is the number of regions (which is unlimited), x is a 3D spatial coordinate, ∗ represents the convolution operator, and the integral is evaluated over the entire FOV. The procedure is initialised with: \(\hat {f}^{(0)}(\mathbf {x}) = g(\mathbf {x})\) and typically converges after approximately 10 iterations.

In practice, PVC is performed in the MRI domain with binary parcellations (Fig. 6), so the PET image first needs to be up-sampled, as the voxel-size is typically smaller in MRI than in PET. The parcellation is obtained using a multi-atlas segmentation propagation strategy (Cardoso et al. 2015) based on a T1w MR image which was parcellated into 144 regions of interest (ROI), which are then grouped into relevant ROIs for amyloid imaging, including: the cerebellar white and great matter, pons, brain stem, cingulate gyrus, hippocampus, precuneus, parietal and temporal lobes and the whole neocortex. The previously obtained global transformations (from the T1w MR to the PET space, see “Generation of the μ-map (Specific to PET/MR Scanners)”) were then used to propagate the regions of interest from the MRI space to the PET space.

The most computationally expensive operation in this PVC algorithm is the 3D convolution, which is expensive for the higher resolution input images (the PET image is upsampled to a resolution of at least that of the T1w MR). The convolution was implemented on the GPU using the method of separable kernels (see the NVIDIA’s CUDA SDK algorithm of 2D separable filters, which we extended to 3D PET images (Podlozhnyuk 2007)). The 3D kernel is decomposed into three one-dimensional filters: one for the transaxial image rows, one for the transaxial columns and one for the axial rows. Therefore, a separable kernel convolution can be divided into three consecutive one-dimensional convolution operations. It requires only U + V + W multiplications for each output voxel as opposed to the standard convolution requiring U ∗ V ∗ W multiplications (U,V,W are the sizes of the kernel in x,y,z directions, respectively). The kernel itself is based on point source measurements on the Biograph mMR scanner, followed by parametrisation of the measured kernel shape through fitting two Gaussians for each one-dimensional kernel.