Computed tomographic colonography: how many and how fast should radiologists report?

This retrospective study used routinely collected data and was approved as a service evaluation by the relevant departments. We collected data from the Radiology Information Systems for all CTC examinations reported by seven gastrointestinal radiologists at two centres, spanning the period January 2013 to December 2015 (centre 1) and January 2012 to December 2015 (centre 2). We only included radiologists who had interpreted more than 200 CTC examinations during this period, to ensure percentages could be calculated with sufficiently narrow 95% confidence intervals to be meaningful. All radiologists had pre-existing CTC expertise; each was a gastrointestinal radiologist, had undergone specific training and had interpreted > 500 examinations. Both centres employed a similar CTC protocol during this period, employing normal-dose post-contrast supine and low-dose prone scans after combined purgation and faecal tagging, intravenous spasmolytics (hyoscine butylbromide), and automated carbon dioxide insufflation. For each radiologist, we extracted (a) the date and time of report verification for all examinations they had reported during this period and (b) the full text of any CTC examinations reported during that period. Subsequently, we inspected the CTC reports to determine if the radiologist had, or had not, reported a 6-mm+ polyp or colorectal cancer. We used each hospital’s patient record system to ascertain whether or not patients with a positive CTC underwent confirmatory testing (i.e. endoscopy or surgery) and, if so, whether the CTC finding was a true positive or false positive. We regarded the presence of any endoscopically or surgically proven polyp or cancer as a true-positive CTC finding, regardless of location or final histology (i.e. a per-patient match). For each radiologist, we estimated their potential “referral rate” (defined as the proportion of CTC examinations in which they reported a 6-mm+ polyp or cancer, i.e. that might be expected to precipitate a referral for colonoscopy), their positive predictive value (PPV; defined as the percentage of cases in which a polyp or cancer was ultimately found if confirmatory testing was done) and their polyp detection rate (PDR; defined as the proportion of cases in which a polyp or cancer was ultimately confirmed, relative to the total number of cases interpreted). For all these proportions (expressed as percentages), 95% confidence intervals were estimated using the Wilson method [17].

Using the extracted dates and times of each radiologist’s complete reporting record for this period, for any given CTC examination, we recorded whether it was the first, second, third (and so on) CTC study reported by the radiologist on that particular day. We also estimated the length of time taken to interpret each CTC examination by deducting the time of report verification for a given CTC study from the time at which the immediately preceding report was verified. For example, if a radiologist verified a chest radiograph at 9:00 a.m., and their next report verification was a CTC examination at 9:30 a.m., we assumed that the radiologist had spent 30 min interpreting the CTC. If a CTC examination was the first report verified on a particular day, it was retained for the purposes of estimating each radiologist’s referral rate, PPV and PDR, but not included when estimating reporting time (since there was no immediately preceding examination to calculate interpretation time). Since CTC examinations that are positive for polyps or cancers take longer to interpret than those that are negative, for each radiologist we calculated their negative interpretation time (by analogy with the colonoscopic negative withdrawal time), by taking the mean of the estimated CTC interpretation time for cases in which no polyp was reported. This better reflects image scrutiny alone (i.e. a normal case) rather than combining both scrutiny and interpretation (e.g. detection followed by characterisation and measurement). To allow for interruptions and batch verification of multiple reports dictated at an earlier time, we set plausible limits on CTC interpretation times; any CTC that appeared to take less than 5 or more than 60 min were assumed to have been pre-reported (and therefore re-checked or verified), or reported after an interruption respectively, and were excluded. Both sites had both 2D and 3D interpretation software available, although at one site this was a thin client launched from the PACS, whereas at the other it was a stand-alone workstation. Computer-aided detection (CAD) was not used routinely at either site. Since the two institutions investigated had different CTC interpretation workflows (e.g. availability of CTC workstations), voice recognition systems and RIS software, we presented negative interpretation time as a proportion relative to their colleagues at the same centre, by dividing by the centre mean.

To assess the effect of interpreting multiple CTC examinations on a given day, we calculated the referral rate and polyp detection rate grouped by the sequence in which the CTC was reported (i.e. first CTC reported that day, second, etc.). To estimate effect size and statistical significance, we used multilevel logistic regression (three levels: CTC, radiologist, centre), with the presence of a polyp as the binary outcome variable and sequence in which the CTC had been performed as the main explanatory variable. To assess the effect of interpretation time on polyp detection, we compared the negative interpretation time for each radiologist with their referral rate, PDR and PPV. We assessed statistical significance using linear regression, with referral rate, PDR and PPV as the outcome variables and negative interpretation time as the explanatory variable. All analysis was performed using R version 3.5.1 for Mac [18].

Overall, 5191 CTCs were reported by the 7 radiologists. The individual radiologist referral rate, PDR and PPV are shown in Table 1. There was a moderate spread in the referral rate and PDR, ranging from 13.1 to 27.5% for the referral rate and 7.8 to 16.3% for the PDR. PPV was grouped more tightly, ranging from 83.3 to 96.1%. The radiologist with the highest PDR had the lowest PPV, and the radiologist with the highest PPV had the second lowest PDR, but overall there was no consistent relationship between radiologist-level PPV and PDR (weak negative correlation, Pearson r = − 0.51, p = 0.25). Overall, both the referral rate and PDR were higher at centre 1 than at centre 2 (referral rate, 19.7% vs 16.1%, p = 0.0019; PDR, 12.2% vs 9.5%, p = 0.0039), but PPV was lower (85.3% vs 93.5%, p = 0.0018).

The rate of positive CTC examinations declined with increasing numbers interpreted on a particular day (Fig. 1). For the first CTC study reported, 21.7% (95% CI, 19.9 to 23.6%) were believed positive for 6-mm+ polyps or cancer by the radiologists, with 12.3% (95% CI, 11.0 to 13.9%) ultimately having polyps or cancer confirmed. By the time of the fifth (or greater) CTC interpretation, only 13.7% (95% CI, 11.7 to 15.9%) were interpreted as abnormal, with a mean PDR of 7.6% (95% CI, 6.1 to 9.4%) for such studies. Therefore, an approximately 40% decline in polyp detection occurred during the day when multiple CTC studies were reported. This was highly statistically significant, with an odds ratio of 0.93 (95% CI, 0.88 to 0.97; p < 0.001) for referral rate (i.e. abnormality identified at CTC) and an odds ratio of 0.93 (95% CI, 0.90 to 0.97; p < 0.001) for polyp detection (i.e. confirmed at endoscopy or surgery). Therefore, for each successive CTC study reported on a given day, the odds of both identifying and confirming a polyp at CTC dropped by 7%. There was no consistent effect of reporting multiple CTC examinations on PPV, which remained consistent at around 90% regardless of examination sequence (p = 0.11).

Three hundred twenty-nine CTCs were reported as the first examination on a given day, and so the interpretation time for these could not be estimated. For the remaining 4862 studies, the mean time taken to interpret a negative CTC examination was 30.5 min (centre 1, 17.4 min; centre 2, 34.6 min). Overall, there was a weak positive association between negative reporting time and PDR; radiologists who spent longer interpreting cases that they ultimately called normal detected more polyps than those who reported more quickly (Fig. 2). This effect was small but statistically significant (p = 0.028), with the regression model suggesting that each 16% increase in interpretation time was associated with a 1% increase in detection rate. There was no clear relationship between negative interpretation time and PPV (p = 0.478).

As the number of CTC studies reported on a given day increased, the mean time spent interpreting each study reduced at centre 1 (reducing from a mean of 19.9 min for scan 1 to a mean of 16.4 min by the time five or more scans had been interpreted), a statistically significant reduction (p = 0.0012). Conversely, the negative interpretation time remained constant at centre 2, irrespective of how many scans had been reported that day (p = 0.59).