The effect of a tuberculosis chest X-ray image reference set on non-expert reader performance

The study was conducted between April and June 2010, at the Queen Elizabeth Central Hospital (QECH), a tertiary referral hospital in Blantyre, Malawi. Blantyre had an estimated TB incidence of 304/100 000 in 2009 (WHO report 2009—global tuberculosis control). Malawi is among the ten poorest countries worldwide in terms of both GDP and per capita income, and has approximately one doctor per 100,000 of the population. Ethical approval was granted by the College of Medicine Research Ethics Committee, University of Malawi (P.04/05/353) and the Ethics committee of the Liverpool School of Tropical Medicine.

All COs and junior or middle-grade doctors (DRs) in the Department of Medicine were invited. A CO has completed a 3-year diploma in Clinical Medicine, Surgery and Community Health and a 1-year internship. A junior doctor has 1 year’s post-qualification experience and a middle-grade of at least 2. A questionnaire on training and experience was completed (online appendix A). Three radiologists, with at least 5 years’ experience in TB CXR participated S.K., E.J., K.I., S.K. being the sole radiologist in Malawi when this study was undertaken. Two COs were selected to attend the Chest Radiograph Reading and Recording System (CRRS) course in May 2010 in South Africa.

The test set was compiled from new, adult PTB suspects participating in a study investigating early mortality in PTB [9]. Unlike most of the TB cases managed at QECH, CXR and microbiological culture results were available for all. To minimise confounding factors in this pilot study, those with previous or extra-pulmonary disease were excluded, as well as HIV positive patients who had been established on anti-retroviral therapy. The principal investigator (C.W.) randomly selected 60 X-rays from smear-positive, culture-positive (23), smear-negative, culture-positive (17) and smear-negative, culture-negative (20) patients. All films were anonymised and digitised [10]. The culture-negative group included six normal X-rays, three films suggestive of PTB and ten X-rays with abnormalities not considered characteristic of PTB. Of these, six had confirmed diagnoses: cardiac failure, metastases, lymphoma, Pneumocystis jirovecii pneumonia, Kaposi’s sarcoma and pneumococcal pneumonia. The remaining films were suggestive of congestive cardiac failure, primary lung malignancy and chronic obstructive pulmonary disease. HIV status was not taken into account during film selection, but retrospectively documented as 68 %.

Anonymised images were selected from teaching material (E.J.), including both high-quality digital X-rays, as well as X-rays from settings similar to Malawi. A spectrum of PTB appearances in adults was presented in 17 black and white, A4 paper prints of JPEG files. This included two examples of atypical presentation in HIV co-infected patients, e.g. focal lower lobe pneumonia. In addition, examples of a normal CXR, an over-exposed and an under-exposed film were included, as well as seven examples of common diagnoses, not typical of PTB, e.g. cardiac failure and septic emboli. Text was deliberately kept to a minimum, describing only the intended use of the tool, the characteristics of a normal film and diagnosis and likelihood of PTB for each image. Figure 1 illustrates sample images.

Readers were aware that films were from PTB suspects, but blinded to other results. They answered two questions: “Would you treat this as PTB? Yes or no?” and: “How certain are you of your decision?” (online appendix B). This format, rather than a detailed radiological film description, was chosen to reflect the clinical setting and force a decision [11]. Reading time was documented. All participants reviewed the set at baseline. Subsequently, the set was re-read by COs and DRs, using the Tuberculosis CXR Image Reference Set (TIRS). Two COs then attended the CRRS course in South Africa and re-read the test set after a 3-week interval on return to Malawi. One week later these two COs re-read the set again using the standardised CRRS proforma (online appendix C). Before every reading the set was reshuffled.

The primary outcome was the number of correct decisions to treat for PTB, using TIRS, compared with the “gold standard” of mycobacterial culture. Secondary outcomes were the comparison of performance between non-experts and experts, effect of CRRS training, level of certainty of diagnosis and time taken to read.

Results for one culture-negative film were removed after expert readings identified miliary TB (categorised as extrapulmonary TB). Using the remaining 59 films and subsequently the subset of smear-negative, culture-positive films, the following were computed for the treatment decisions of each participant at baseline and for all non-experts using TIRS (taking culture outcome as the “gold standard”): sensitivity, specificity, absolute percentage agreement between treatment decision and culture result, as well as this agreement corrected for chance (kappa statistic). The kappa statistic was calculated for chance agreement with the gold-standard of culture, rather than for inter-observer agreement as is commonly used in imaging studies because our hypothesis related to increasing the number of correct decisions, rather than increasing observer agreement. Participants rated their level of certainty on a four-point scale: 1 = 0 % sure, 2 = up to 30 % sure, 3 = 30–60 % sure, 4 = 60–100 % sure. Averages of these statistics are reported for the CO and DR cohorts, both separately and combined, as means and standard deviations; the changes from baseline to TIRS are reported as mean differences with their 95 % confidence intervals (adjusted for clustering within participants). Mean changes in certainty scores were also computed for correct and incorrect diagnoses separately. Finally, the average scores at baseline, with TIRS, and then with CRRS, are reported for the two COs who went to CRRS; no formal statistical comparisons were possible with this small sample. Similarly, results were not considered separately by HIV status, given the low power generated by inclusion of only 19 films from HIV-negative subjects in the test set. Statistical significance was set at the conventional 5 % level for all analyses. Kappa values of > 0.207 were considered to be statistically significant.