Validation of deep learning-based computer-aided detection software use for interpretation of pulmonary abnormalities on chest radiographs and examination of factors that influence readers’ performance and final diagnosis

Anonymized CRs (posteroanterior view) of 453 patients were retrospectively selected from two institutions in Japan. Only patients who were over the age of 19 years were included. Institution A, a university hospital, supplied the data of 238 patients who presented for physical examination between January 2012 and December 2019. Institution B, a health screening center, supplied the data of 215 patients who had routine health screening between January 2016 and December 2018. One CR image was acquired from each patient; thus, 453 images were collected. The images were acquired using Aero DR system (Konica Minolta, Tokyo, Japan) or FUJIFILM DR PRELIO U(Fujifilm, Tokyo, Japan)with 120–130 kVp tube voltage, 1–8mAs tube current time product. The exclusion criterion was poor image quality, for which no CR was excluded. Pulmonary nodules/masses and consolidation on the CRs were considered abnormal findings, while extrapulmonary abnormal findings, such as cardiomegaly and rib fractures, were considered normal for the purpose of this study. Nodules and masses were defined as focal lung opacities with smooth border, measuring  ≤ 3 cm and  > 3 cm in diameter, respectively. Consolidation was defined as lung opacities apart from nodules and masses. A total of 194 images (43%) included abnormal findings, while 259 (57%) were normal. For the observer-performance test, 60 images with and 140 without abnormal findings were randomly selected. Two board-certified radiologists (with 6 and 14 years of experience) reviewed the images and recorded the area, lesion type (nodule, mass, or consolidation), and the degree of subtlety of each lesion by mutual agreement. The degree of subtlety was measured on a five-point scale as follows: level 1, extremely subtle (detection is extremely difficult); level 2, very subtle (detection is very difficult); level 3, subtle (detection is difficult); level 4, relatively obvious (detection is relatively easy); and level 5, obvious (detection is easy) [13].

All images were independently reviewed by three board-certified radiologists (with 14, 25, and 33 years of experience) to establish the ground truth. The radiologists confirmed the presence of abnormal findings on the images and marked the lesion locations. The common areas annotated by at least two of the three radiologists with an intersection over union (IoU) greater than specific threshold for each finding were adopted as abnormal lesions. The thresholds for nodule/mass and consolidation were determined as 0.5 and 0.0, respectively.

CAD-Chest X-ray (Konica Minolta, Inc. and Enlitic, Inc.) software was used for this study. This software is currently commercially available in Japan (Approval No. 30300BZX00271000). This software was designed as a second-reader type CAD. It automatically detects pulmonary nodules/masses and consolidation, and marks the areas of the lesions.

First, a performance test of the CAD software alone (standalone test) was conducted. All 453 images in the dataset were interpreted with the software alone. Then, an observer-performance test was performed to assess whether the software would improve doctors’ performance. The test had a sequential-only design and was done in accordance with the US Food and Drug Administration guideline [14]. The CRs of the selected 200 patients were interpreted by 12 doctors, including three radiologists, 3 pulmonologists, three non-pulmonology physicians, and three junior residents with various years of experience (2–12 years). The readers were blinded to the clinical information of the patients, and the radiologists who defined the reference standard did not participate in the performance test. The test consisted of two sessions. In the first session, the readers were asked to determine whether each CR showed any nodule, mass, or consolidation. If any of these was present, the readers then marked the center of the lesion. All procedures were performed without CAD software. The readers were also asked to input a confidence score with a continuous value between zero and one for each annotation. In the second session, the readers were asked to re-evaluate every CR with the assistance of the CAD software and to modify their original decisions and confidence scores.

The sensitivity and specificity of the CAD software for detecting pulmonary nodules, masses, and consolidation were analyzed in the standalone test. Per lesion sensitivity and patient specificity were calculated. In the observer-performance test, the detection performances of the readers with and without CAD were compared. Jackknife alternative free-response receiver operating characteristic (JAFROC) analyses were performed using R statistical software version 4.0.2 (R Project for Statistical Computing, Vienna, Austria) and RJafroc version 2.0.1. Both the readers and CRs were treated as random effects. The weighted alternative free-response receiver operating characteristic (wAFROC) figure of merit (FOM) score was used as the performance measure for the analyses. The weights were equally divided by the number of lesions. Statistical significance was evaluated using the Dorfman-Berbaum-Metz method [15]. The results were stratified according to the specialty, years of experience of the readers. The mean FOM scores with and without CAD in each group were compared. For the analysis of the sensitivity of the CAD software and the adoption rate of the lesions that readers initially missed but CAD detected, the lesions were grouped by the anatomic location and the degree of subtlety. Fisher’s exact test was used for the analyses. Statistical significance was set at P < 0.05.

In the standalone test using 453 images, the CAD software achieved sensitivities of 83% (85/103) and 80% (74/93) for nodules/masses and consolidation, respectively. Thus, its sensitivity for detecting abnormal lesions was 81% (159/196), and its specificity was 62% (160/259).

Comparison of detection performances with and without CAD software.

When the 200 images were used for the observer-performance test, the CAD software achieved sensitivities of 79% (26/33) and 81% (26/32) for nodules/masses and consolidation, respectively. Thus, its sensitivity for detecting abnormal lesions was 80% (52/65), and its specificity was 63% (88/140). The mean wAFROC FOM scores of all readers with and without CAD were 0.746 (95% confidence interval [CI] 0.668–0.823) and 0.810 (95% CI 0.746–0.873), respectively. Thus, the increase in the wAFROC FOM score by the CAD software was 0.064. The mean wAFROC FOM score with CAD was significantly higher than without CAD (P = 0.007). Fig. 1 shows ROC curves with and without CAD. When stratified by the specialties of the readers, the mean wAFROC FOM scores for radiologists, pulmonologists, non-pulmonology physicians, and junior residents without CAD were 0.806 (95% CI 0.699–0.913), 0.817 (95% CI 0.714–0.920), 0.746 (95% CI 0.677–0.815) and 0.613 (95% CI 0.535–0.692), respectively. The wAFROC FOM scores increased with use of the CAD software to 0.835 (95% CI 0.765–0.904), 0.839 (95% CI 0.763–0.915), 0.815 (95% CI 0.747–0.884) and 0.749 (95% CI 0.6338–0.861), respectively. Thus, the increments were 0.029, 0.022, 0.069, and 0.136, respectively. The mean wAFROC FOM score of non-expert doctors (with < 6 years of experience) significantly improved with CAD from 0.680 (95% CI 0.586–0.773) to 0.779 (95% CI 0.705–0.853) (P = 0.011). The wAFROC FOM scores of experts (with ≥ 6 years of experience) also improved from 0.811 (95% CI 0.745–0.877) to 0.841 (95% CI 0.782–0.899) with CAD, but the difference was not significant (P = 0.12). The increments for each group were 0.099 and 0.030, respectively. Table 2 summarizes the results.

Non-experts and experts detected 66% (256/390) and 78% (305/390), respectively, of all abnormal lesions without using CAD. Table 3 shows the sensitivity of the CAD software and human readers, stratified by anatomic location and degree of subtlety of the lesion. Experts had higher sensitivity than non-expert doctors in all groups. The sensitivity of the CAD software for detecting obscure lesions (subtlety level 1, 2 or 3) was significantly lower than those for distinct lesions (subtlety level 4 or 5) [50% (10 of 20) vs 93% (42/45); P < 0.001]. The human readers initially missed 117 lesions without CAD. For 78 (67%) of these lesions, detection by the CAD software was accepted by each reader. Figures 2 and 3 show true-positive examples of pulmonary nodules and consolidation, which some readers missed without using CAD but acknowledged after using the software. Table 4 describes the accepted lesions, stratified by their characteristics. Of the initially missed lesions, 67% (55/82) and 66% (23/35) were corrected with CAD in the non-expert and expert groups, respectively. When stratified by the degree of subtlety, the adoption rate of obscure lesions and 50% (54/73) was significantly lower than that of distinct lesions [55% (24 of 44) vs 74% (54/73); P = 0.04]. The CAD software did not detect 13 lesions, and 12 of these were detected by at least one human reader.