Assessing deep learning reconstruction for faster prostate MRI: visual vs. diagnostic performance metrics

This study, approved by an institutional review board, waived the need for informed consent. It involved retrospective data from three medical centres: University Medical Centre Groningen (UMCG, Groningen, The Netherlands) between 2014 and 2020, Martini Hospital Groningen (MHG, Groningen, The Netherlands) between 2013 and 2020, and Radboud University Medical Centre (RUMC, Nijmegen, The Netherlands) in the year 2016. All patients were suspected of having clinically significant prostate cancer (csPCa) based on criteria such as suspicious digital rectal examinations, lower urinary tract symptoms, or elevated prostate-specific antigen levels.

The collected data comprises either biparametric or multiparametric prostate MRI examinations obtained for routine clinical care. The biparametric protocol included a Turbo Spin Echo T2-weighted (T2W) sequence (see Supplementary Material 1, Table 1), high b-value (≥ 1400 s/mm²) diffusion-weighted imaging, and an apparent diffusion coefficient map. The study comprised 357, 572, and 846 patients from the UMCG, MHG, and RUMC cohorts, respectively. Exclusions were made for studies that did not follow the PI-RADSv2 [6] MRI protocol requirements or for patients with prior prostate treatment. The external RUMC dataset was subjected to the same exclusion criteria before data transfer. After exclusions, the final sample included 1535 patients, of which 733 (~48%) had PI-RADS 4 or higher, comprising 305 from UMCG, 384 from MHG, and 846 from RUMC.

This study focused on T2W sequences for DLRecon-based prostate MRI for specific reasons. T2W imaging is the dominant sequence for transitional zone lesions and is crucial for morphological assessment, lesion localisation, and guiding procedures like biopsies. By concentrating on T2W sequences, we aim to assess DLRecon’s impact on anatomical representation without functional data from diffusion-weighted imaging, which PI-RADS designates as the primary sequence for peripheral lesions.

Expert uroradiologists with 4–25 years of experience assigned PI-RADS scores and delineated all (likely) csPCa lesions (defined as a PI-RADS score ≥ 4), serving as the reference standard [7‐9].

T2W acquisitions (DICOMs) were retrospectively undersampled. The slices were converted to the frequency domain, commonly referred to as k-space, using a fast Fourier transform (FFT). The k-space acquisition is known to be time-intensive, especially during the acquisition of phase-encoding lines. An approach to accelerate the acquisition is to acquire fewer phase-encoding lines, leading to aliasing artefacts in reconstructed images. During the processing of raw k-space data, zero-filling is commonly applied before conversion into an image to achieve higher spatial resolution and a smoother appearance. To take the zero-filling into account, we utilised a binary mask to capture the k-space regions that were acquired. Following this, zero-padding was added to the binary mask in a manner analogous to the zero-filling typically applied to acquired k-space data. Finally, the reconstruction from the undersampled data uses the inverse of the FFT shift, followed by the inverse fast Fourier transform (IFFT), to convert the masked k-space back to the image domain.

The k-space undersampling mask was designed to subsample the row-based phase-encoding (horizontal) direction (Fig. 1), making it compatible with the multicentre dataset. Research has shown that fully sampling the centre of k-space, which corresponds to low spatial frequencies, improves the signal-to-noise ratio and yields favourable ImagQMs [10]. On the other hand, sampling in the periphery of k-space helps preserve high-resolution details and sharp edges, essential for representing small anatomical structures. We used a 20% central and 80% peripheral k-space undersampling scheme for optimal balance. In practical terms, for a 4-fold (R4) undersampling where 25% of k-space is sampled, 20% of the samples would be drawn from the centre and the remaining 80% from the periphery. This approach aimed to maximise diagnostic utility by enabling the detection of small-sized lesions characterised by high-frequency information while maintaining overall image quality.

For the DLRecon model, we adopted the 3D U-Net [11] architecture, which is widely used for image-based data (see Supplementary Material 2, Fig. 1a). The dataset contains 1535 DICOM images, each with its corresponding PI-RADS segmentation. Due to the absence of multicoil k-space data, we used simulated k-space to carry out image space-based reconstruction tailored for lesion detection algorithms. Therefore, employing an image space-based U-Net is driven by the need to compute a DiagQM through the performance of a subsequent csPCa DLDetect model.

The DLRecon model inputs a centre-cropped, undersampled, and aliased image. The objective is to recover lost details and eliminate aliasing artefacts from k-space undersampling. The training is performed on 4- and 8-fold (R4 and R8) subsampling rates. Specific training parameters include SSIM for the loss function, a batch size of 12, and an Adam optimiser with a 4e–4 learning rate. Additionally, early stopping was used to reduce the risk of overfitting. During training, data augmentation included rotations (25% probability, −30 to 30 degrees), the addition of normally distributed noise (60% probability, 0–0.003 range), and mirroring (50% probability). Training completed within 48 hours on a 32-GB Tesla V100 GPU using the Keras software libraries. The code is available on GitHub (Repo: ProstateMRI-DLReconVsDiagMetrics).

The data for the DLRecon model is preprocessed in four steps: k-space synthesis combined with undersampling, resampling, centre cropping, and normalisation. First, the image is undersampled with the binary subsampling mask. Next, the image undergoes resampling using nearest neighbour interpolation to achieve 0.5 × 0.5 mm voxel spacing, leaving the z-direction unchanged. Subsequently, the image dimensions are standardised through centre cropping (256, 256, 16). Lastly, instance-wise min/max normalisation (scaled between 0 and 1) ensures uniform pixel values, improving model generalisation.

To evaluate the clinical utility of DL reconstructions, we deployed a specialised 3D U-Net [9], distinct from the reconstruction model, solely focused on locating likely csPCa (Fig. 2) in T2W scans (see Supplementary Material 2, Fig. 1c for model architecture). The model has an attention mechanism and squeeze-and-excitation layer to improve feature learning [9, 12]. Specifically, the model was tailored for T2W scans to exclude diffusion-weighted imaging, apparent diffusion coefficient, and dynamic contrast-enhanced sequences, ensuring an unbiased evaluation of the diagnostic quality of T2W sequences. The model generates a 3D heatmap, assigning each voxel a likelihood of csPCa. The training employed a weighted binary cross-entropy loss function. Class weights were set to 0.05 and 0.95 for the negative and positive classes, respectively, to address the class imbalance and to prioritise lesion features over the background tissue. We employed 5-fold cross-validation on 80% of the dataset, reserving 20% as a test set. The highest-performing models from each validation fold were stored. These models were used to evaluate the diagnostic quality of DL-reconstructed T2W MRI scans in the test set. In our methodology, we chose not to retrain the DLDetect models for each acceleration, reflecting the reality in clinical practice where radiologists do not undergo specific retraining for each new technique. This approach prevents the models from normalising hallucination effects—false positives or negatives due to reconstruction—which could mask true diagnostic accuracy.

Preprocessing data for the diagnostic DLDetect models involved resampling, centre cropping, and z-normalisation. Images were resampled to a consistent voxel spacing (0.5, 0.5, 3.0) mm across the multicentre dataset, with a crop size of (180, 180, 16) to focus on the prostate. A larger crop was used in the reconstruction model as the additional field of view could enhance its performance. Instance-wise z-score normalisation standardised voxel values to a mean of zero and standard deviation of 1. Data augmentation included a 10% probability of rotation (−30 to 30 degrees), a 30% probability of adding normally distributed noise (0–0.001 multiplier), and a 50% probability of horizontal flipping.

Our study involved two experiments that evaluated the impact of DLRecon on visual and diagnostic quality using ImagQMs and a DiagQM. See Fig. 2 for a flowchart of the reconstruction and evaluation pipelines. The first experiment employed a Wilcoxon test to statistically assess the visual quality of reconstructions, focusing on SSIM as the ImagQM. The SSIM ranges from −1 to 1, with 1 indicating identical visual quality to the reference image and lower values indicating decreased quality. We analysed reconstruction quality recovery on two subsampling factors, R4 and R8, using both naïve and U-Net-based reconstruction methods. Here, naïve reconstruction refers to inversion from subsampled k-space.

The second experiment evaluated how effectively diagnostic DLDetect models could recognise likely csPCa in reconstructed MR images. Diagnostic accuracy was measured using free-response operating characteristic (FROC) analysis, a methodology used to evaluate a system’s lesion detection sensitivity alongside the false-positive rate, as a patient can have multiple lesions. Within this framework, the diagnostic accuracy metric used was the partial area under the curve (pAUC) of the FROC curve. Specifically, the study employed pAUC values ranging between 0.1 and 2.5 false positives per patient as the DiagQM.

A permutation test compared the pAUC between R4 and R8 subsampling factors across both reconstruction methods. With true positives requiring a 10% overlap with ground truth lesions [9, 13], we aimed to demonstrate non-inferior diagnostic performance (DiagQM) and visual quality (ImagQMs) compared to fully sampled MRI (R1). From the 1535 scans, we designated 80% (N = 1229) exclusively for training and validation, employing five-fold cross-validation. The remaining 20% (N = 306) constituted a different unseen separate held-out test set for final evaluation. This data-splitting method was consistently applied for the DL reconstruction and detection models. In addition to these quantitative metrics, we also qualitatively investigated the manifestation and impact of hallucinatory effects in the reconstructed images.

We conducted a small reader study to validate the DLDetect as a proxy for assessing the diagnostic quality of DLRecon. We selected a balanced set of 30 cases from our test set based on their DLetect observed differences in csPCa likelihood. Comprising 15 cases with the highest difference in PCa detection (the ‘Inconsistent Set’) and 15 with the lowest difference (the ‘Consistent Set’).

An experienced radiologist (> 8 years) reviewed R1 vs. R4/R8 image pairs, categorising them as consistent, showing minor variation, or inconsistent. Cohen’s kappa analysis compared radiologist’s and DLDetect’s ability to agree on observed inconsistencies (hallucinations). For a detailed explanation of the study’s methodology and findings, please refer to Supplementary Materials 3.