Inter-rater agreement in the assessment of abnormal chest X-ray findings for tuberculosis between two Asian countries

This study was approved by the ethics committees of the Ministry of Health, Viet Nam and the National Center for Global Health and Medicine (formerly, International Medical Center of Japan). Written informed consent was obtained from each participant prior to the investigations, including the prevalence survey and the follow-up study.

A population-based TB prevalence survey of 11,624 people aged 15 and over was conducted in Hanoi in 2003 as reported previously [3]. Briefly, subjects suspected of having active TB based on CXR or on symptoms underwent sputum smear microscopy and/or mycobacterial culture. Details of HIV status were not obtained from the study subjects. According to the report of World Health Organization during this period, estimated prevalence of HIV co-infection in new TB patients aged 15-49 was relatively low (2.8%) in Viet Nam [13].

Barring 317 individuals, active TB was radiographically excluded for the rest. Of these 317 individuals, 22 (6.9%) were diagnosed by bacteriological methods, including sputum culture [3]. In 2004, individuals who presented with radiographic findings during the initial survey were advised to undergo sputum smear and culture tests following the World Health Organization recommendation [14, 15]. In the 2006 follow-up, in which the same group of individuals was recalled for plain chest radiographic examination (AGFA X-ray film, Beijing, China; Shimadzu UD 150L-30V, Kyoto, Japan) and sputum test, including direct smear and culture. Using a questionnaire, we collected information regarding individual history, additional examinations performed, and treatment for TB undergone after the initial survey. Demographic information (including addresses) collected during the prevalence survey was used to trace the target group in the follow-up period.

The CXR films analyzed in this study were those in which active TB had not been radiographically excluded during the prevalence survey and were those taken during the follow-up in 2006. In total, 258 of the 317 films in the prevalence survey and 93 follow-up films were available at the time of analysis in this study. The rest of TB-suspected films in the prevalence survey were missing.

Two coding systems were used to classify the CXR findings. The chest radiograph reading and recording system (CRRS) was developed in 2005 to detect TB and other forms of lung disease [5]. Profusion score and details of abnormalities unrelated to TB were omitted. All the other coding items of this system were retained. A Japan-Vietnam CXR coding system (JVCS) (Figure 1) consisting of rather simple codes was also used: We newly developed this system, considering a registration form used in a public payment system for TB treatment expenses in Japan and reading practice in Viet Nam. CRRS classifies parenchymal abnormalities as primary or secondary lesions depending on the significance of the lesion. In contrast, JVCS does not consider the significance of the lesion, though it records pleural effusion and thickening separately. Additionally, CRRS classifies nodules based on their size and calcification, whereas JVCS separately records nodules and calcification.

Two Japanese pulmonary physicians (E.T. and N.K.) and two Vietnamese radiologists (L.D.H. and P.T.C.) read the CXR films. These readers were different from those who read the CXR films during the initial survey. All CXR films were first read using CRRS. After the completion of readings by CRRS, CXR films were read using JVCS without the results of CRRS being made known to the readers. Each reader was also blinded to the others' readings and clinical information. Instruction and training regarding the two coding systems were given prior to the actual reading. The four raters were asked to reach a consensus while assessing 10 standard films from Japan and another 10 films from Viet Nam.

We adopted a double entry system of data entry. JMP version 7.0.1 (SAS Institute Inc., Cary, NC, USA) and SAS version 9.1 (SAS Institute Inc.) were used for analysis. Kappa statistics were used to investigate inter-rater agreement on the presence or absence of lesions of interest. We adopted the following guidelines for interpretation of kappa coefficients: < 0, poor agreement; 0-0.20, slight; 0.21-0.40, fair; 0.41-0.60, moderate; 0.61-0.80, good; and 0.81-1.00, very good [16‐18]. Weighted kappa was used to assess inter-rater agreement on variables with more than two categories. McNemar's test or its extension, Bowker's test of symmetry, was used to investigate the symmetry of disagreement between two raters, which tests whether the frequency of an abnormality detected by one rater is significantly different from that by another rater. The generalized estimation equation (GEE) was also used to test the similarities in frequencies of positive findings between groups of raters (marginal homogeneity). No symmetry or non-marginal homogeneity was considered to be significant when P < 0.05.

In 2004, one year after the prevalence survey, 204 (64.4%) of the 317 individuals who presented with radiographic findings of suspected TB underwent a sputum smear test, one of whom tested positive. The initial CXR film of this case showed infiltrates, fibrosis/scarring, and calcification. The follow-up radiograph in 2006 showed improvement after treatment.

In the follow-up in 2006, 93 individuals were checked, one of whom was diagnosed by smear and culture as TB positive (Figure 2). Besides calcification, which was seen in the initial CXR film, infiltrates were present in the follow-up film. All raters evaluated this case as "worse" based on the radiographic findings.

In total, five individuals were reported to have active TB during the 3-year follow-up period. Two were diagnosed bacteriologically and three were diagnosed based on self-reported TB episodes. All the films were randomly mixed in the study set.

Using the two coding systems, four raters assessed the 258 films taken during the 2003 prevalence survey; two raters assessed the 93 films taken in the 2006 follow-up. A total of 2,436 readings were conducted (Figure 2).

Agreement levels regarding overall parenchymal abnormalities assessed by CRRS varied. Their kappa values were interpreted as fair to good, ranging from 0.24 to 0.63, from the following six comparisons: a comparison between the two Japanese raters (JP-JP); four comparisons between Japanese and Vietnamese raters (JP-VN (1) to (4)); and a comparison between the two Vietnamese raters (VN-VN) (Table 1). Agreement levels regarding calcification also varied. They were considered as fair to good with JVCS and slight to fair with CRRS. Kappa values for pleural effusion with JVCS were interpreted as moderate to good, ranging from 0.54 to 0.77, indicating high level of agreement irrespective of country or rater.

Major parenchymal findings, cavity, fibrosis/scarring, infiltrates, and nodules were assessed in a similar way, as shown in Table 2. Agreement levels regarding primary and secondary cavities in CRRS were rather low (kappa values ranged from -0.02 to 0.36) except for relatively high agreement levels regarding a primary cavity between the Japanese raters (kappa = 0.60), and a secondary cavity between the Vietnamese raters (kappa = 0.43). Cavitation was, thus, mainly classified as a primary lesion by the Japanese raters and as a secondary lesion by the Vietnamese raters.

Although agreement levels relating to fibrosis/scarring were also low, kappa values for secondary fibrosis/scarring with CRRS revealed fair levels of agreement between raters from the same country (kappa = 0.28 [JP-JP] and 0.22 [VN-VN]), but revealed only slight agreement between raters from different countries (kappa = 0.11 to 0.20 [JP-VN]). Among all Japanese-Vietnamese pairs, the Vietnamese raters specified secondary fibrosis/scars more frequently than the Japanese raters (P = 0.0001 or P < 0.0001 by McNemar test). The frequency of positive findings of secondary fibrosis with CRRS by both Vietnamese raters was 26/255 (10.2%), whereas that by both Japanese raters was only 7/245 (2.9%) (Table not shown). The frequency of positive findings of fibrosis/scarring with JVCS by both Vietnamese raters (56/255 = 22.0%) also tended to be higher than that by both Japanese raters (42/245 = 17.1%). GEE further confirmed the significant difference in frequencies of fibrosis/scarring between raters from different countries (P = 0.0082).

Agreement levels regarding infiltrates between the two raters from the same country were considered as moderate (kappa = 0.49 [JP-JP] and 0.57 [VN-VN]) and as fair between two raters from different countries (kappa = 0.21 to 0.30 [JP-VN]) according to JVCS (Table 2). The Japanese raters detected infiltrates more frequently than the Vietnamese raters (P < 0.0001 by McNemar test) in all comparisons. The frequency of positive findings of primary infiltrates with CRRS by both Japanese raters was 68/245 (27.8%), whereas that by both Vietnamese raters was only 22/255 (8.6%) (Table not shown). The frequency of positive findings of infiltrates with JVCS by both Japanese raters (119/245 = 48.6%) also tended to be higher than that by both Vietnamese raters (46/255 = 18.0%). The different frequencies of positive infiltrate readings between the raters from the two countries were also confirmed by using GEE (P < 0.0001).

The levels of inter-rater agreement were considered slight to fair for nodules, irrespective of the raters' home country or the coding system used.

An overall assessment of CXR changes after 3 years was conducted by one of the two raters from each country. Agreement was moderate for both coding systems (weighted kappa = 0.47 and 0.40). The Japanese rater indicated deterioration more frequently than the Vietnamese rater (Table 3); this difference was considered highly significant for both JVCS and CRRS by the symmetry test (P = 0.0002 and 0.0008, respectively, by Bowker's test). When assessing changes in specific findings after 3 years, the Japanese rater detected infiltrates more frequently than the Vietnamese rater (P < 0.0001; Figure 3). Among 55 cases of infiltrates, 12 (22%) were assessed as "further spread" by the Japanese rater while 2 (8%) out of 25 cases of infiltrates were assessed as "further spread" by the Vietnamese rater (data not shown).