Observer training for computer-aided detection of pulmonary nodules in chest radiography

It is common clinical practice that small primary lung carcinomas are missed on two-view chest radiographs although they were frequently visible in retrospect. Miss rates of between 20 % and 90 % have been reported for primary lung carcinomas [14‐20]. Two recent papers reported a CAD stand-alone sensitivity of 35 % and 47 % for pulmonary tumours initially overlooked by the radiologist [1, 4], indicating the potential of CAD to improve the readers' detection performance. Both papers, however, did not include observer performances; thus, to which extent radiologists would have taken advantage of CAD and accepted these true-positive candidates remains unanswered. Equally the risk of accepting false-positive candidates with unnecessary follow-up diagnostics cannot be quantified. The importance of the interaction between the CAD and observer has been previously illustrated by a study that tested the impact of CAD on observer performance for the detection of T1 tumours on chest radiographs of patients who had been part of a CT screening trial: the number of malignancies initially not seen by the observers but correctly annotated by CAD varied between 5 and 16 per observer. However, 80 % of these correctly annotated lesions (46/59) were not accepted by the readers and subsequently dismissed [9]. One underlying reason for the readers' inability to differentiate true- from false-positive lesions might have been the lack of experience with the CAD algorithm and subsequently insufficient trust in its performance.

In the current study we therefore tested the effect of short-term feedback on the detection of pulmonary nodules on digital chest radiography. Our hypothesis was that individual feedback after the interpretation of each of the four subsets would help readers to build up more confidence in the performance of the CAD algorithm, eventually resulting in an increased ability to distinguish true- from false-positive candidates and in a higher acceptance of true-positive CAD candidates.

We found a slight but not significant increase in baseline performance between sessions 1 and 2 compared with sessions 3 and 4, meaning that there was a small overall training effect and that readers detected more nodules in the last two readings than in the first two readings. For none of the paired sessions, however, could we prove a significant influence of reader performance by CAD. Neither the acceptance of true-positive CAD candidates nor the ability to dismiss false-positive candidates was significantly influenced by the feedback information. There was an overall tendency towards increased performance with CAD but the differences were too small to reach significance.

Seventy-eight percent of the dismissed true-positive CAD candidates referred to lesions of low and very low conspicuity, indicating that readers had difficulties in assigning sufficient credibility to CAD candidates, indicating nodules with low conspicuity. Feedback did not help in this respect because the number of dismissed true-positive candidates was the same in sessions 1 and 2 compared with sessions 3 and 4.

There are very few studies in the literature evaluating the impact of training on the use of CAD and those referred for CT colonography and mammography. Results were quite different in the respect that a short, 1-day period of training already affected observer performance in CT colonography as opposed to mammography, which had a lower learning curve requiring at least 4 weeks [10, 11]. It seems that the effect of CAD follows different perception and learning rules in CT vs. radiography, and that the beneficial use of CAD to reduce perception errors of well-defined and more conspicuous lesions (e.g. colon polyps) can be learned faster than the use of CAD to detect lesions of low conspicuity that require differentiation from obscuring background noise (e.g. lesions on mammography). This is also supported by our result that most of the dismissed true-positive CAD candidates referred to lesions of low and very low conspicuity.

It is very likely that lesions of high or moderate conspicuity that have been missed by the readers owing to “inattentional blindness” meaning that the lesions missed during routine reporting take advantage of the availability of CAD more effectively and more easily . Unfortunately it is much more difficult to prove this effect under study conditions because readers tend to analyse the radiographs with an especially high degree of alertness compared to normal conditions. Even though the readers in our study generally had a low level of experience in reading chest radiographs—5 out of 6 were residents—the baseline sensitivity was relatively high with a mean of 65 % for the first two subsets and 70 % for the last two subsets. It is possible that this high baseline sensitivity impeded a further increase in sensitivity with the availability of CAD results.

Lesions of low or very low conspicuity, however, have different diagnostic requirements: correct diagnosis requires not only visual localisation but also correct differentiation from surrounding “anatomical” noise. For this type of lesion, our results suggest that CAD has no significant impact on reader behaviour and short-term feedback has no effect. Whether a longer learning period would be more efficient as postulated for mammography [12] remains to be proven for chest radiography.

Our results also underline the need to further decrease the number of false-positive CAD candidates. A lower number of false-positive candidates will not only decrease reading time but also increase readers' confidence in the reliability of CAD, and will help them to focus on the presence of underlying lesions in the circled areas of interest. The number of false-positive calls provoked by CAD was quite low in this study: 10 and 6, respectively, pooled over all readers for the first two and last two readings.

It might also be the case that the pure presentation of CAD candidates alone is not sufficient and more information is needed to help the reader to correctly differentiate true- from false-positive lesions. In this context the availability of likelihood calculations together with an active localisation procedure by the reader him/herself has been found to be very effective for mammography [21].

Our study suffers from the following limitations. Nodule incidence in the study population was higher than usually seen under clinical conditions. Although the readers did not know the exact distribution of positive and negative cases, they certainly were more alert to detecting focal lesions than in a usual clinical setting.

The number of CAD false-positive candidates was not equally distributed over the four subsets, although we consider it unlikely that this had an effect on our results as all readers interpreted the subsets in a different order and there was a generally low number of accepted CAD false-positive candidates.

This study used a specific type of CAD algorithm. It has to be noted that these results are not necessarily transferrable to other CAD algorithms: a different or updated CAD algorithm with a different performance may yield different results in a context in which perception, reader experience and confidence as well as lesion conspicuity form a complicated framework.

Our study represents the experience of one institution; whether a different reader group or involving readers from various institutions would have yielded different results remains speculative.

We conclude that short-term feedback does not significantly increase the ability of readers to differentiate true- from false-positive candidate lesions in chest radiography in order to use CAD more effectively for the detection of nodular lesions. Further research is needed to determine whether a longer training period or additional processing and display tools such as temporal subtraction, rib suppression or likelihood calculations can increase the benefits of CAD for reader performance in chest radiography.