Inter-reader agreement of the PI-QUAL score for prostate MRI quality in the NeuroSAFE PROOF trial

The development and subsequent diffusion of multiparametric magnetic resonance imaging (mpMRI) of the prostate have unavoidably resulted into variability in terms of vendors’ and scanners’ quality worldwide [1].

Suboptimal image acquisition reduces the diagnostic accuracy of mpMRI for the detection of clinically significant prostate cancer and this is why since the publication of the first version, the Prostate Imaging Reporting and Data System (PI-RADS) committee outlined the technical requirements for the acquisition of mpMRI of the prostate of adequate diagnostic quality [2].

Moreover, two boards of experts have reiterated the importance of quality criteria for the acquisition of mpMRI of the prostate [3,4], and the Prostate Imaging Quality (PI-QUAL) scoring system [5] from the multi-centre PRECISION trial [6] has represented the first attempt to address this issue.

The PI-QUAL score evaluates the quality of prostate mpMRI against objective technical criteria (as per PI-RADS guidelines) together with subjective criteria from the images [5].

PI-QUAL is centred on a 1-to-5 scale that assesses the adequacy of the diagnostic quality of mpMRI of the prostate, where 1 implies that all mpMRI sequences are below the minimum standard of diagnostic quality, 3 means that the study is of sufficient diagnostic quality (as at least two mpMRI sequences taken together are of diagnostic quality), and 5 indicates that all sequences are of optimal diagnostic quality. In more detail, a PI-QUAL score ≥ 4 implies that the quality of the study is high (i.e., all clinically significant lesions can be ruled in and ruled out).

The purpose of this study was to evaluate the inter-observer agreement of the PI-QUAL score in patients enrolled in the NeuroSAFE PROOF trial [7].

This is a retrospective analysis of men prospectively enrolled in the NeuroSAFE PROOF trial (registration number: NCT03317990), whose recruitment started in 2018 [7]. The trial received ethical approval (Regional Ethics Committee reference 17/LO/1978) and was supported by the National Institute for Healthcare Research (NIHR reference PB-PG-1216-20013). Written informed consent was obtained from all patients, who granted permission for their samples to be used for research purposes. No further ethical approval was needed for this specific audit.

The NeuroSAFE PROOF trial is an ongoing multicentre randomised controlled trial in which patients are randomised 1:1 to either NeuroSAFE (a technique that involves intraoperative fresh-frozen section analysis of the posterolateral aspect of the prostate margin to assess whether cancer extends beyond the capsule) or standard robot-assisted radical prostatectomy [7].

Among 123 patients from 13 centres who were enrolled in the study (as of February 2021), 20 (all from the same centre) had biparametric MRI, leaving 103 patients from 12 centres for final analysis (Fig. 2).

Patient characteristics are shown in Table 1.

Overall, 83/103 (81%) patients were scanned on a 1.5-T scanner and 20/103 (19%) patients on a 3-T scanner, and the distribution in terms of MRI systems and manufacturers is shown in Fig. 3. The distribution of scans for each participating centre is reported in Table 2.

The main acquisition parameters for the different MRI scanners are reported in Supplementary Table 1.

In our study, we observed strong reproducibility in the assessment of prostate MRI quality using PI-QUAL between two radiologists with high expertise in prostate MRI. The agreement was strong both for each single PI-QUAL score (κw = 0.85 and percent agreement = 84%) and when the PI-QUAL scores were clustered into three groups according to the ability (or not) to rule in and to rule out clinically significant prostate cancer (κw = 0.82 and percent agreement = 84%).

There is currently much interest in standardising high-quality prostate MRI, as this is of paramount importance especially when it comes to MRI-derived targeted biopsies used to detect clinically significant prostate cancer, since a patient with a negative scan and favourable prostate-specific antigen (PSA) kinetics should confidently avoid unnecessary immediate biopsy [13]. However, in order to be able to safely rule in and rule out prostate cancer, images with good spatial resolution, high signal-to-noise ratio and no artefacts are necessary.

The level of reproducibility found in our study compares favourably with that reported for other scoring systems of the prostate [14,15] including PI-RADS [16‐21], and similar results have been reported in the arterial hyperenhancement for the diagnosis of hepatocellular carcinoma using a 1-to-5 scoring system [22] and for the Bosniak classification of cystic renal masses [23].

In addition to this, Brembilla and colleagues have recently reported that interobserver studies in prostate MRI research should mirror clinical practice as closely as possible to increase the generalisability of the results [24].

At present, little is known about the effect of imaging quality on interobserver agreement [16] and a recent international consensus meeting has reiterated the importance of studies focusing on reporting MRI quality, as this is critical in studies evaluating prostate MRI performance and reproducibility [3].

The PI-RADS guidelines are well established [25] and, as such, have already been evaluated in studies that have demonstrated their generalisability and reliability. The PI-QUAL score, on the other hand, had been more recently developed and as such it is important to determine the reproducibility of the system [26].

If we have a closer look at Table 5, we can see that the highest levels of disagreement in terms of visual assessment on T2-WI and DCE corresponded to the delineation of specific anatomic landmarks, such as the ejaculatory ducts, the neurovascular bundles and the urethral sphincter muscles on T2-WI along with the vessels on DCE, while higher agreement was observed as far as the prostatic capsule and seminal vesicles on T2-WI are concerned. This could be explained by the fact that some structures (e.g. the ejaculatory ducts and the urethral sphincter muscles) are more difficult to be identified and one of the differences that emerged during the post-scoring consensus meeting was that the senior consultant radiologist in prostate MRI reading used both the axial and coronal acquisitions to judge the visibility of these structures, while the other radiologist used only the axial T2-WI [27]. Also, the visual assessment of DCE images is more difficult compared to T2-WI, as multiple dynamic time-points need to be reviewed without a clear temporal definition for what constitutes early enhancement (e.g. a possible solution could be to use the time point when the adenoma in the transitional zone just starts to enhance, which should occur before any peripheral zone enhancement if the temporal resolution is adequate).

As far as DWI is concerned, the disagreement was substantial when assessing the adequacy of the apparent diffusion coefficient (ADC) map. This is an interesting point, as visual assessment is often used as the primary method to assess the quality of DWI [28] and this could be strengthened by the extrapolation of standardised ADC values from different institutions and MRI scanners or by using built-in phantoms on each scanner.

Interestingly, in the 16 cases reported in Table 5, the two readers had substantial agreement in the evaluation of the absence of artefacts on all sequences, in particular on T2-WI and DCE.

A key feature of our study is its multi-centre setting, which reflects the heterogeneity of prostate MRI conduct across different institutions and is representative of patients undergoing prostate MRI during routine clinical practice, thereby reinforcing the generalisability of our findings. Moreover, both readers interpreted the full examinations on imaging workstations rather than screen captures, ensuring the most accurate assessment for each MRI study.

Our results suggest that the PI-QUAL score can be applied with reasonable consistency by radiologists familiar with prostate MRI on the basis of a clear understanding of the lexicon provided in the scoring sheet [29]. However, it needs to be stressed that only genitourinary radiologists with appropriate training in prostate MRI should assess the quality of the scans [30]. The findings of our study may be also useful for guiding the future iteration of the PI-QUAL score, which should focus on the features that have the greatest objective reproducibility and that should receive more emphasis in the future versions.

Also, a more objective definition of some less reproducible features such as the ‘reader-estimated’ criteria (e.g. what constitutes ‘adequacy’ of the ADC map) will need to be addressed in the next version of PI-QUAL.

Our study has some limitations. Firstly, both readers assessing the PI-QUAL score worked at the same centre and the junior consultant radiologist had been previously trained by the senior consultant radiologist; thus, they may have had the tendency to approach cases similarly. It should be also noted that the PI-QUAL score has been only recently published [5] and there is currently no literature on its application in a clinical setting.

Another limitation of the present study is that we did not investigate the correlation of PI-QUAL with radical prostatectomy. Although we know that the use of PI-QUAL has clinical implications (i.e., ruling in and ruling out clinically significant prostate cancer), the aim of this specific study was to test the interobserver reproducibility of the PI-QUAL score and not to evaluate the correlation between image quality, lesion conspicuity on mpMRI and histology. Though we did not test the inter-reader variability in ruling in and ruling out clinically significant prostate cancer, we can anticipate that other work is already underway to investigate PI-QUAL score and diagnostic accuracy with pathology as reference standard. It should be also noted that the population included in this study consisted of patients who were candidates for radical prostatectomy, so there could have been a bias towards those studies with MR-visible lesions and higher PI-RADS scores.

As previously stated in the original publication [5], the PI-QUAL score will be refined in the future (e.g. an international group is currently working on the next version of PI-QUAL in order to see if this scoring system should be still based on a 5-point scale or simplified into a 3-point scale) and we believe that the findings of our study may be useful for guiding the future iteration of this scoring system. Thus, it is envisaged that the interobserver agreement will continue to improve over time with the new iterations of the PI-QUAL score.

In conclusion, the findings of our study support the reliability of the PI-QUAL score for the assessment of prostate MRI quality. However, it is anticipated that the scoring system will need to undergo further refinements.