Breast histopathological image analysis using image processing techniques for diagnostic purposes: A methodological review

Due to inconsistency in staining and image acquisition, histopathological images suffer from color and illumination variations. The main factor that influences the color variation is the difference in staining procedures adopted in different laboratories [22, 23, 46]. The different scanners and types of equipment used to capture the image results in illumination variation. Ignoring the variance in color and illumination of histopathological images often leads to incorrect results [47]. To overcome these problems pre-processing step, known as color normalization is employed for the histopathological image analysis. An illustration of color and illumination variance of BCHI are shown in Fig. 6. In this section, a summary of various works on the color normalization techniques for histopathological images is presented.

Many attempts have been made to eliminate the color and illumination variations in histopathological images over the past years [48‐51] , [52‐54]. Systematic study on assigning of an image color to another image was carried out in 2001 by Reinhard et al. [55]. The color characteristics of a template image are brought to an input image based on mean and standard deviation in LAB color space representation. This method is popularly known as the Rehinard method. To improve the color consistency in histopathological images Basavanhally et al. [48] proposed an Expectation-Maximization (EM) based segmentation–driven Standardization (EMS) algorithm. In this approach, the input and the normalized image were qualitatively compared during the nuclei segmentation step. To obtain an accurate stain density map in histopathology images Vahadane et al. [51] proposed a Structure-Preserving Color Normalization (SPCN) algorithm. They converted the source and the target RGB images to optical density space using the Beer-Lambert law. The structure of the original image is preserved by employing the SPCN algorithm. But this method fails to preserve the color information of the source image and suffers from local minima. Roy et al. [50] developed a Fuzzy based Modified Reinhard (FMR) method to handle the color variations in H&E stained images. The conventional Reinhard method reduces the contrast of the source image. Hence to overcome this problem, fuzzy logic with the Reinhard method was incorporated. The authors of [49] experimented on different color normalization algorithms and proposed stain normalization using saturation-weighted statistics. Their method is capable of identifying the cause for color variation. An alternative reference space for color normalization in histopathological images was reported by [54]. Euclidean distance was utilized to measure the intra-cluster and inter-cluster ratios. Each eight-bit input image is reduced to ten colors using K-means clustering. To overcome inconsistency in the staining process, Macenko et al. [52] proposed a stain quantization algorithm that belongs to a class of unsupervised normalization approaches. The images were analyzed on the shape and stain-based features. Further, an algorithm to obtain the optical stain vector with various stain combinations was provided. Tosta et al. [53] developed an estimation method to handle the faded regions in the histopathological images. The downside of the proposed method is that fails to preserve the tissue color corrections resulting in low performance. A Quantile normalization approach was proposed by Cao et al. [56] to develop an automatic breast cancer grading system. Gadermayr et al. [57] investigated the various stain normalization techniques concerning tissue classification. The study included five different normalization techniques, followed by extraction of five features on two differently stained renal images on two experimental setups. Khan et al. [58] proposed a non-linear mapping technique for the normalization of an input image to the color distribution of the reference image based on color deconvolution. Bukenya [59] developed a hybrid technique for stain normalization which consists of two stages namely, the stain separation stage and color transfer stage. The input images and the reference image were converted from RGB to Optical Density space and performed stain separation and color transfer steps to normalize the image. A stain deconvolution approach by employing a multi-resolution wavelet representation of the image to evaluate the stain mixing matrix was presented by Alsubaie et al. [60]. They converted the input image to optical density image and extracted R, G, and B channels, and decomposed each color channel to its sub-band by employing wavelet decomposition. In the last few years, much study is carried out to address the problem of stain normalization in terms of processing time and additional system memory utilization. Anghel et al. [61] proposed an unsupervised approach for stain normalization in WSI. They adopted the method proposed in [52] to perform a high-performance stain normalization system and also proposed a method to detect low-quality images. The main objective of the study was on optimizing and enhancing the robustness of the stain normalization algorithm. An automated color segmentation approach was developed by Kothari et al. [62]. The experiment was performed on four different types of H&E stained images. The images were normalized by adopting two types of normalization techniques namely quantile normalization on all the pixels and normalization on the color maps of the images which are obtained by extracting the unique color. A color segmentation accuracy of 85% was reported. An experiment on extracting the color and texture information to evaluate the need for stain normalization was conducted by Gupta et al. [63]. The authors suggested a method for the selection of reference images. A comparative study on different normalization techniques for epithelium and stromal classification was performed by Sethi et al. [64]. A multi-resolution segmentation approach was performed for super pixel-based classification. The patch-based classification was performed by using Convolutional Neural Network (CNN). Bejnordi et al. [65] developed a fully automated algorithm called whole-slide image color standardizer for the standardization of whole-slide histopathological images. Color and spatial information were employed to classify the image pixel into distinctive stain components. In [66], a pipeline that utilizes an unsupervised method based on stain vector estimation was proposed to handle the memory and runtime bottlenecks in high magnification images. They claimed the method is computationally less expensive in terms of memory. Magliaro et al. [67] developed an open-source tool called Histology for the separation of dye colors in histology images. The graphical user interface was developed by utilizing the K-means clustering algorithm to isolate the dye colors in histological images. The performance of the tool was compared with the ImageJ color deconvolution plugin in terms of speed of color separation. An algorithm to reduce the stain variations in H&E stained histopathological images was proposed by Bejnordi et al. [46]. The standardization algorithm is based on the clustering of the images into two tissue components. Bautista et al. [68] implemented a color correction method by utilizing the color information of nine color patches of a color calibration slide. The proposed method does not work when there are color variations caused by the staining procedure. A flexible and robust image analysis algorithm was developed for the separation and quantification of immunohistochemical staining by Ruifrok et al. [69]. The proposed algorithm was designed to deconvolve the color information captured with RGB cameras and to estimate the contribution of each of the applied stains. A color transfer approach based on YCbCr color space for the enhancement of peripheral blood smear images was developed by Prasad et al. [70]. An approach for the selection of the template image and the effect of the good template image on color normalization was described. Clarke et al. [71] developed a color calibration assessment slide for digital pathology. The positive aspect of the method is the variations in the tissue thickness does not alter the shape of the spectrum. The authors also claimed that the proposed method will add to the reproducibility of the automation of image analysis systems. Bautista et al. [72] proposed a method for the detection and visualization of tissue folds in pre-scanned WSI. The proposed method incorporates the color enhancement technique which distinguishes between the folds and non-fold regions in an image. Color saturation and luminance of the image pixels with weighted differences are used as a shifting factor to the input RGB color of the image. In recent years color normalization was also carried out by using DL techniques [73, 74]. Generative Adversarial Networks (GANs) model for color normalization of H&E stained images was developed by Zanjani et al. [74]. The model has an end-to-end framework that was trained to learn the chromatin space of H&E images. Janowczyk et al. [73] proposed a stain normalization technique using Sparse Autoencoders (StaNoSA) to handle the color variations in H&E stained images. The StaNoSA was developed by incorporating sparse encoders as a core method to segregate the input images into distinct tissue classes. Hamidinekoo and Zwiggelaar [75] proposed a DL-based network to detect mitosis in BCHI. Color normalization was performed by utilizing the RGB histogram specification method. In this method, color values of the target image are normalized to the source image on a pixel-by-pixel basis.

Segmentation methods divide an image into a smaller group of pixels. Application of segmentation algorithms to BCHI aids in identifying the ROI . In this section, we provide details of the state-of-the-art methods for ROI detection and segmentation.

Feature is defined as an “interesting” part of an image and helps in describing different regions in an image. These features help in identifying the same objects in different images. Classification is a process of assigning labels to different groups based on the identified features. In this section, we provide details of the state-of-the-art methods used for feature extraction and classification.

The development of CAD systems for analysis of BCHI is greatly dependent on the availability of large-scale datasets with annotations. Moreover, different annotations need to be provided for semantic segmentation and classification. For example, segmentation of nuclei needs annotations where nuclei regions are marked, image-level annotations are necessary for the image classification problem, bounding box based annotations are needed for detection problems. However, providing these annotations requires domain knowledge and is a tedious task. Considerable care must be taken while providing annotations since it has a great impact on the learning algorithm. The lack of a standard dataset in literature for the analysis of BCHI limits the development of CAD systems. In our view, the establishment of a standard large-scale dataset with annotations is the need of the hour as it provides a common platform to compare various algorithms. Summary of the most frequently used datasets in the literature is shown in Fig. 10. From the literature, it can be observed many research groups used BreakHis and Bioimaging 2015 database images for their study.

One of the most popular publicly available database for the analysis of BCHI is BreakHis [24]. From Figures 11 and 12 we can see that the number of cases for the benign class is significantly less as compared to the malignant class. This imbalance in classes results in poor training of DL models. The class imbalance problem is one of the challenges which needs to be addressed. A possible solution is to consider data augmentation to increase the number of samples for the benign class.

To determine the condition of tumor, a pathologist analyses the images at varying magnifications such as 40x, 100x, 200x, and 400x. At 100x magnification, nuclei distribution and tissue structures are studied which helps in the determination of different types of cancer. At 400x magnification, various characteristics of nuclei are studied namely mitotic count, prominent nuclei and hyperchromatic nuclei. Hence, each magnification plays an important role in the determination of breast cancer. Despite this interest, most researchers have focused on developing CAD systems for 400x magnification images which may fail to capture the distribution of the tissue structure. It can be thus suggested that, in the future more attention could be given to the analysis of 100x and 200x magnification images for the determination of breast cancer.

The slide preparation process has a large impact on the results since variations in the slide preparation process result in different color distribution of the histopathological images. To handle variations in color distribution, the color normalization process is popularly used. It is observed in the literature that there are no appropriate evaluation metrics for evaluating color normalization processes. Hence, identifying the proper metrics for evaluation of the color normalization process is important. Also, GAN can be explored in the future as it can learn to generate images based on constraints and hence, can be used to transfer the color distribution of a reference image to an input image.

The most prominent task in the analysis of BCHI is studying the structure and characteristics of nuclei since it contributes greatly to the determination of malignancy of the tumor. This involves the segmentation of nuclei which is challenging due to clustered and overlapping nuclei, heterogeneous structure of nuclei and poor staining process. Moreover, at 100x magnification, nuclei are small in size. Several algorithms are developed in literature based on traditional and DL based methods to segment nuclei. Watershed-based algorithm is popularly used to separate connected nuclei. However, the separation of overlapping nuclei is seldom addressed in the literature. Also, the segmentation of nuclei from 100x magnification is very challenging as it needs annotations at 100x magnification for the validation process which is a tedious process.

Earlier traditional methods which include the pipelined approach had much popularity in processing BCHI. These methods include various steps such as pre-processing, segmentation, feature extraction, and classification. However, the accuracy of these systems depends on the segmentation process and feature extraction from the identified regions. Handcrafted features were used to define the ROI. However, the handcrafted features may fail to capture all the variations in the pattern of the data and thus reducing the accuracy of the system. Also, developing these systems required extensive domain knowledge in image processing and histopathological images.

DL has gained much popularity in the last few years for processing of histopathological images due to its ability to model complex patterns and increases the computational power [14, 186]. CNN is a popular choice for feature extraction as it learns to extract the most relevant features based on the backpropagation algorithm. However, researchers have to analyze the results of all layers to justify the performance of the developed model for the analysis of BCHI. Also, the development of CNN architecture for the analysis of histopathological images requires expertise in DL. But DL-based algorithms require a large-scale dataset with annotation for training the models. The lack of standard datasets with annotations makes it challenging to develop DL-based models for processing histopathological images. Moreover, feature visualization and analysis is necessary to understand the behaviour of the model. Figure 13 shows the usage of various popular CNN architectures used for multiclass and binary classification problems. It is observed that VGG-16, ResNet-50, and InceptionNet are popular choices for binary classification problems. VGG-16 and ResNet-50 are also used for multiclass classification of BCHI. Also, it is observed that deeper architectures such as VGG-19 and ResNet101 are less explored for classification as compared to other methods.