Accurate Determination of Imaging Modality using an Ensemble of Text- and Image-Based Classifiers

Imaging modality can aid retrieval of medical images for clinical practice, research, and education. We evaluated whether an ensemble classifier could outperform its constituent individual classifiers in determining the modality of figures from radiology journals. Seventeen automated classifiers analyzed 77,495 images from two radiology journals. Each classifier assigned one of eight imaging modalities—computed tomography, graphic, magnetic resonance imaging, nuclear medicine, positron emission tomography, photograph, ultrasound, or radiograph—to each image based on visual and/or textual information. Three physicians determined the modality of 5,000 randomly selected images as a reference standard. A “Simple Vote” ensemble classifier assigned each image to the modality that received the greatest number of individual classifiers’ votes. A “Weighted Vote” classifier weighted each individual classifier’s vote based on performance over a training set. For each image, this classifier’s output was the imaging modality that received the greatest weighted vote score. We measured precision, recall, and F score (the harmonic mean of precision and recall) for each classifier. Individual classifiers’ F scores ranged from 0.184 to 0.892. The simple vote and weighted vote classifiers correctly assigned 4,565 images (F score, 0.913; 95% confidence interval, 0.905–0.921) and 4,672 images (F score, 0.934; 95% confidence interval, 0.927–0.941), respectively. The weighted vote classifier performed significantly better than all individual classifiers. An ensemble classifier correctly determined the imaging modality of 93% of figures in our sample. The imaging modality of figures published in radiology journals can be determined with high accuracy, which will improve systems for image retrieval.