An experimental study on breast lesion detection and classification from ultrasound images using deep learning architectures

In early machine learning, the mainstream machine learning methods were based on statistics, and they did not care about features. However, computer vision is the application of machine learning in the field of vision, which a good feature extraction method is crucial. Feature extraction is a process of dimension reduction, which reduce the number of resources needed for processing without losing important or relevant information, and facilitate the speed of learning and generalization steps in the machine learning process. There were a lot of manual feature extraction methods that can be divided into three categories [2, 3]: i) Interest point detection (such as Laplacian of Gaussian, Difference of Gaussian, Harris Corner Detection, Features From Accelerated Segment Test), ii) Dense features [4] (such as Scale Invariant Feature Transform [5], Histogram of Oriented Gradient [6], Local Binary Pattern [7, 8]), iii) Feature Combinations (such as Deformable Part-based Model [9, 10]).

Several previous methods discussed on how to automatically classify breast lesions. In [11], the authors built three M-dimensional feature sets and selected the features by principal component analysis and mutual information to classify 641 ultrasound images. In [12], the authors segmented the Breast Ultrasound images based on watershed transform and extracted 22 morphological features from segmented lesions, and selected the features based on mutual information and statistical tests to classify 641 ultrasound images. In [13], the authors proposed a computer-aided diagnosis method depending on the lesion’s shape type of ultrasound image. They used Zernike moments and invariant moment to extract feature, meanwhile, they used support vector machine and multilayer perceptron to classify 45 ultrasound images. In [14], The authors proposed a classified method by using texture analysis to extract features, and perceptron classification method was used to classify 57 ultrasound images. In [15], the authors classed the primary and secondary occurring of benign and malignant cases. they extracted Laws’ mask texture features from the ultrasound images and used support vector machine as a classifier to distinguish 172 ultrasound images of the breast lesions.

In the past, researchers usually studied hand-crafted features within the traditional detection framework. For example, Dalal et al. [6] used support vector machine with the Histogram of Oriented Gradients features for the pedestrian detection task. Felzenszwalb et al. [9, 10] proposed a Deformable Part-based Model using latent support vector machine, which achieved the best performance in the 2006 Pattern Analysis, Statistical Modelling and Computational Learning person detection challenge. In [16], the authors used the dictionary learning method to obtain a sparse expression of an image, which was called Histograms of Sparse Codes. Histograms of Sparse Codes was used to replace Histogram of Oriented Gradients for classifier training and target detection. Although the performance has been considerably improved, the detection speed is quite slow. In [17], the author proposed an object detector based on co-occurrence features, which was three kinds of local co-occurrence features constructed by the traditional Harris Corner Detection, Local Binary Pattern, and Histogram of Oriented Gradients respectively.

Several previous methods discussed on how to automatically locate ROI of breast lesions. In [18], A self-organizing map neural network was used for the detection of the breast lesion. The ROI can be extracted automatically by employing local textures and a local gray level co-occurrence matrix which is a joint probability density function of two positions. Compared with the basic texture feature, the gray level co-occurrence matrix can reflect the comprehensive information about the direction, the interval and the amplitude of the image. In [19], Shan et al. developed an automatic ROI generation method which consisted of two parts: automatic seed point selection and region growing. However, the method depends on textural features, and these features are not effective for breast ultrasound images when there exists a fat region close to the lesion area or contrast is low. In [20], a supervises learning method was proposed to categorize breast tissues into different classes by using a trained texture classifier, where background knowledge rules were used to select the final ROI for the tissues. However, due to the inflexibility of the introduced constraints in the proposed method, its robustness was reduced. In [21], the authors improved the method in [20] by proposing a fully automatic and adaptive ROI generation method with flexible constraints. In their work, the ROI seed can be generated with high accuracy, and can also well distinguish the datasets lesion regions from normal regions. However, as shown in the experiments, the recall is still unsatisfactory, that average recall rate was low that benign was 27.69%, malignant was 30.91%, the total was 29.29%.

Recently, deep learning techniques have attracted a lot of attention from researchers, because of the good data interpretability as well as the high discriminable power. Noticeably, deep convolutional neural network (CNN) have substantially improved the performance not only for general object detection [22‐26], but also for image classification [27‐32]. So far in the literature, people have employed CNN based methods to handle detection and classification tasks for medical images, such as mammograms [33]. To the best of our knowledge, there is little work that has comprehensively evaluated the performance of different CNN based detection and classification methods for lesions in breast ultrasound images.

Collecting a well-defined dataset is key to the research on breast lesions detection/classification. For that, we have been collaborating with Sichuan Provincial People’s Hospital to have experienced clinicians annotate breast ultrasound images obtained from breast lesions patients. Specifically, the patients were told to get scanned by LOGIQ E9 (GE) and IU-Elite (PHILIPS) to generate those ultrasound images. Each ultrasound image was later reviewed and diagnosed by two or three clinicians. Based on the ratings obtained from the Breast Imaging-Reporting and Data System (BI-RADS) [35], each diagnosed image was then grouped into 7 categories indexed from 0 to 6, where 0 means more information is needed, 1 negative, 2 benign finding, 3 probably benign (less than 2% likelihood of cancer), 4 suspicious abnormality, 5 highly suggestive of malignancy, and 6 proven malignancy. According to [35], some medical specialists proposed to further partition the fourth category (suspicious abnormality) into three sub-category, i.e., 4A (low suspicion for malignancy), 4B (intermediate suspicion of malignancy) and 4C (moderate concern, but not obvious for malignancy). For that, by following the professional instructions from our clinicians, we divide our datasets into two classes: benign and malignant. The benign class is constructed by the images grouped into categories 2, 3 and 4A, while the malignant class consists of the images from categories 4B, 4C, 5 and 6. By working with the clinicians, we have collected 577 benign and 464 malignant cases from patients. Moreover, the lesion in each image has also been marked out by those experienced clinicians. Figure 1 show cases four ultrasound images containing either benign or malignant lesions. To the best of our knowledge, there is no such a publicly available ultrasound image datasets as ours for breast lesions.

The remarkable progress of deep learning techniques, especially CNN, have largely promoted the research of visual object detection. Fast Region-based convolutional neural networks (R-CNN) [22], Faster R-CNN [23], You Only Look Once (YOLO) [24], YOLO version 3 (YOLOv3) [25], and Single Shot MultiBox Detector (SSD) [26] are existed state-of-the-art object detection methods. However, these CNN-based methods only focus on general object detection. In this paper, we apply them to detecting lesions in our newly collected breast ultrasound dataset. We also combine each CNN based detection method with different existing neural networks, e.g., Visual Geometry Group (VGG16) [29], ZFNet [28].

We use the result of method [21] as a baseline for the detection of breast lesions. All CNN are modified to evaluate these CNN architecture from ImageNet detection task to our dataset. Next, we will introduce the difference between these algorithms.

Fast R-CNN R-CNN [36] and Spatial Pyramid Pooling Net [37] using CNN to classify region proposals, and achieves excellent object detection accuracy. However, two major issues still exist: i) the training phase is a multi-stage pipeline; and ii) object detection is slow. To overcome these drawbacks, also inspired by Pyramid Pooling Net [37], Girshick et al. [22] improved R-CNN by proposing Fast R-CNN which adds an ROI pooling layer to the last convolution layer, the ROI pooling layer uses max pooling to convert the features inside any valid region of interest into a small feature map with a fixed spatial extent. Each feature is fed into a fully connected layers that finally branch into two output: one output produces softmax probability estimates and another output does bounding-box regression. In other words, performs classification and bounding box regression simultaneously.

Faster R-CNN Fast R-CNN, as selective search is used for region proposals, the detection time is not very fast. To avoid the standalone step to generate regions, Ren et al. [23] proposed to integrate a so-called Region Proposal Network (RPN) into Fast R-CNN, and RPN and fast R-CNN share large number of convolutional layers. In Faster R-CNN, an image as input fed into RPN and outputs a set of rectangular object proposals, each with an objectness score, which is fed into two sibling fully connected layers: an object category classification layer and a box regression layer, simultaneously regress objectness scores and region bounds at each location on a regular grid.

YOLO YOLO [24] employed a single convolutional neural network to predict the bounding boxes and class labels of detected regions. Since the YOLO limits the number of bounding boxes, it avoids repetitive detection of the same object and thus greatly improves the detection speed, making YOLO suitable for real-world applications. Due YOLO may fail to localize small objects, Redmon and Farhadi propose YOLO version 2 (YOLOv2) [38], an improved version of YOLO. YOLOv2 use a new classification model Darknet-19, and achieved state of the art on standard detection tasks. In [25], Redmon and Farhadi made a bunch of little design changes to YOLO, that present a faster and more accurate detecor than YOLOv2 which is called YOLOv3. YOLOv3 predicts bounding boxes with dimension priors and location. YOLOv3 use a much more powerful feature extractor network, which is a hybrid approach between the network used in YOLOv2, Darknet-19, and the newfangled residual network stuff. YOLOv3 is a fast and accurate detecor.

SSD In order to improve detection speed and accuracy, Liu et al. [26] proposed SSD, which only needs an input image and ground truth boxes for each object during training. For objects of different size, SSD adds several auxiliary convolutional feature layers which progressively decrease in size, and predicts detections at multi-scale. SSD uses shallower layers for detecting small objects. Furthermore, in a convolutional fashion, the SSD framework evaluate a small set of default boxes of different aspect ratios at each location in several feature maps with different scales. In order to efficiently to discretize the space of possible output box shapes allows different default box shapes in several feature maps. For each default box, SSD predicts both the shape offsets and the confidences for all object categories.

In this work, we mainly explore and evaluate different CNN architectures with different model training parameter values in classify breast lesions tasks. These CNN architectures learning from the labeled set, which has major advantages over more traditional approaches that use hand-crafted features. We also evaluate the transfer learning from no-medical datasets due to the lack of big data.