Automated segmentation and diagnosis of pneumothorax on chest X-rays with fully convolutional multi-scale ScSE-DenseNet: a retrospective study

A deep learning-based typical segmentation architecture is composed of two parts: a down-sampling path (contraction) and an up-sampling path (expansion), where the down-sampling path is responsible for feature learning and the up-sampling path aims to restore the spatial information and image resolution. Alternatively, skip connections can be used to help the up-sampling path to recover spatial detail information from the down-sampling path by reusing feature maps. In this study, we employ FC-DenseNet [24, 25] as the network backbone for its advantages of parameters reduction, computational efficiency and better withstand of over-fitting problem.

The down-sampling path of FC-densenet consists multiple blocks, each containing a dense block followed by a transition-down block. For each dense block, it iteratively concatenates all feature maps in a feedforward paradigm. A dense block contains multiple layers, each consisting of a batch normalization, a non-linearity activation function, a convolution operation, and a dropout connection (see Fig. 1c). Each layer in the dense block, l, takes all feature maps of the preceding layers that match the spatial resolution as input, outputs k feature maps and passes them to the subsequent layers (see Fig. 1b), where k is known as growth rate. Hence, the number of feature maps in the dense block grows linearly with the depth of the down-sampling path of FC-DenseNet and the output of the \(l{\mathrm{th}}\) layer can be defined as:where \(x_l\) denotes the feature maps at the \(l{\mathrm{th}}\) layer, the notation \(\oplus\) denotes the channel-wised concatenation for the feature maps from the layer \(l-1\) to the layer 0. H is a composition of batch normalization, exponential linear unit and convolutional layer with dropout rate of 0.2 (see Fig. 1c), and \(H_l\) represents a composite function of the \(l{\mathrm{th}}\) layer.

In order to reduce the spatial dimensionality of the feature maps, a transition down block following the dense block is introduced (see Fig. 1d). The transition down block consists of batch normalization, exponential linear unit and \(1\times 1\) convolution for depth preserving with dropout rate of 0.2, and followed by a \(2\times 2\) max pooling operation. In particular, the end block of the down-sampling path is called bottleneck and is connected to the up-sampling path.

Through the up-sampling path, the spatial resolution of the input can be recovered by transition up blocks, dense blocks, and skip connections from the corresponding blocks of the down-sampling path. The transition up block is a transposed \(3\times 3\) convolution (see Fig. 1e), which implements the up-sampling of the previous feature maps. Then, the up-sampled feature maps are channel-wisely concatenated with the feature maps from the corresponding skip connections in the down-sampling path as the input of the dense block in the up-sampling path. At the end of up-sampling path, the feature maps of the output are convolved with a \(1\times 1\) convolution layer, and followed by a softmax layer and average max-pooling operation to generate the final segmentation map. This connection pattern strongly encourages the reuse of features and allows all layers of the architecture to receive direct supervision signals.

To learn the relations across lesion features on multiple scales, multiple convolution kernels with different receptive fields were parallelly incorporated into the first convolution layer of FC-DenseNet to capture variability of viewpoint-related object. The module for processing chest X-ray images with varying size of convolution kernels is called the multi-scale convolution module. GoogLeNet [26] has introduced the multi-scale convolution kernels into a parallel sub-network as a inception module, allowing the abstract convolution features with different scales to be transported to the subsequent layer simultaneously. The inception module of GoogLeNet contains different size of convolution filters such as \(1\times 1\), \(3\times 3\) and \(5\times 5\) convolutional kernels, and \(3\times 3\) max-pooling operation.

In the semantic segmentation task, a small convolution kernel can help the detection of small target regions, and a larger convolution kernel can not only detect the larger target regions, but also eliminate the false positive regions. Therefore, we add a larger convolution kernel (\(7\times 7\)) to expand the receptive field for the segmentation of PTX. To avoid the reduction of segmentation accuracy caused by dimension reduction, we also removed the \(1\times 1\) convolution kernel and \(3\times 3\) max-pooling, making the multi-scale convolution kernel module more efficiently in the PTX segmentation architecture. After these different convolution operations, all feature maps are channel-wisely concatenated for the subsequent dense block (see Fig. 2)

Most of fully convolutional networks (FCNs)-based segmentation methods mainly focus on the joint space and channel encoding. For example, FC-DenseNet can simultaneously transmit the spatial and channel information of the current filters to the subsequent convolution layers to improve the utilization of features. However, spatial- and channel-wise independent coding are less utilized. Recently, Hu et al. [27] proposed a framework embedded with squeeze and excitation (SE) blocks to model the interdependencies between feature channels, and achieved state-of-the-art results in image classification. Roy et al. [28] introduced three variants of the SE blocks, including the channel SE (cSE) module, the spatial SE (sSE) module, and the concurrent spatial and channel squeeze and excitation (scSE) module, to migrated the SE blocks from image classification to image segmentation with promising performance. The purposes of the SE and cSE module are to adaptively recalibrate feature maps along the channels and to elucidate useful channels while suppressing the less useful channels. The cSE module can only reweight channels and the sSE module can only reweight spaces, while the scSE module can recalibrate the feature maps of channels and spaces respectively, and then merge these feature maps into output layer.

In this study, we embedded the scSE module into each dense block and proposed the application of scSE dense block in pneumothorax segmentation (see Fig. 1a, b). We denote the input feature maps of a dense block as \({\mathrm {U}}\), \({\mathrm {U}}\in {\mathbb {R}}^{H\times W\times C}\), where H, W, and C denote the spatial height, width, and the number of channels, respectively. As illustrated in Fig. 3, the input feature maps \({\mathrm {U}}\) can be recalibrated to the output feature maps \({{\mathrm {U}}}_{{\mathrm{scSE}}}\), \({{\mathrm {U}}}_{{\mathrm{scSE}}}\in {\mathbb {R}}^{H\times W\times C}\), through the two branches of \({{\mathrm {U}}}_{{\mathrm{sSE}}}\) and \({{\mathrm {U}}}_{{\mathrm{cSE}}}\). The \({{\mathrm {U}}}_{{\mathrm{scSE}}}\) can be formulated as:where \({{\mathrm {U}}}_{{\mathrm{sSE}}}\) and \({{\mathrm {U}}}_{{\mathrm{cSE}}}\) are recalibrated from \({\mathrm {U}}\) in spatial space and on the channels, respectively. \({{\mathrm {U}}}_{{\mathrm{sSE}}}\) can provide more relevant spatial locations by ignoring irrelevant spatial locations and \({{\mathrm {U}}}_{{\mathrm{cSE}}}\) can be adaptively tuned to ignore less important channels and to emphasize more important channels.

Specifically, \({{\mathrm {U}}}_{{\mathrm{sSE}}}\) can be obtained from \({\mathrm {U}}\) through a \(1\times 1\times 1\) convolution kernel and a sigmoid function. The computing weight of the convolution kerner, denoted as \(W_s\), \(W_s\in {\mathbb {R}}^{1\times 1\times C\times 1}\), can be used to learn a projection tensor Q, where \(Q\in {\mathbb {R}}^{H\times W}\). Then the sigmoid function \(\sigma (\cdot )\) is applied to rescale the activations of Q into [0, 1]. Hence, \({{\mathrm {U}}}_{{\mathrm{sSE}}}\) can be defined as:For the cSE module, a global average pooling operation \(g(\cdot )\) is first performed on the input feature maps \({\mathrm {U}}\) to generate a vector z embedded with globally spatial information, where \(z=g({\mathrm {U}})\), \(z\in {\mathbb {R}}^{1\times 1\times C}\). Then two consecutive fully connected layers are used to convert the vector z into a new vector \(\hat{z}\), \(\hat{z}=W_1(\delta (W_2 z))\), where \(W_1\) (\(W_1\in {\mathbb {R}}^{C\times \frac{C}{2}}\)) and \(W_2\) (\(W_2\in {\mathbb {R}}^{\frac{C}{2}\times C}\)) denote the weights of the two consecutive fully connected layers, respectively, and \(\delta (\cdot )\) denotes the operation of ReLU. Afterwards, we apply a sigmoid function \(\sigma (\cdot )\) to normalize the activations into [0, 1]. Therefore, the formulation of cSE module can be defined as:In summary, the scSE module combines the advantages of sSE module and cSE module, enabling better adaptive recalibration of feature maps, so that the scSE dense block can elucidate more useful information while suppressing less useful features in the application of pneumothorax segmentation.

The serious pixel class imbalance issue between the region of interests (ROIs) and the surrounding background generally exists in medical image segmentation. The number of pixels with pathology is much less than that without pathology. This tends to cause the learning model to fall into a local minimum. The typical cross entropy loss (CEL), which measures the quantization error of all pixels by calculating the pixel-level probabilistic error between the predicted output class and the target class, is susceptible to the class imbalance problem. Then weighted cross-entropy loss (W-CEL) is introduced to mitigate the effect of class imbalance by giving different weights to target classes and background pixels. Meanwhile, dice loss [29] is also proposed to optimize the dice overlap coefficient between the predictive segmentation map and the ground truth map. However, due to the narrow boundary of the pneumothorax class, it is still difficult to distinguish the target classes from the background pixels through W-CEL and dice loss. Therefore, the boundary class is also required to be considered along with the target and background classes.

Pneumothorax segmentation is generally formulated as a binary classification task with respect to object (pneumothorax) versus background, where ‘0’ is used to represent the background pixels and ‘1’ is used to represent the pneumothorax pixels. In this study, if the eight neighborhoods of the pixel value ‘1’ have a pixel value of ‘0’, we define this pixel value ‘1’ as boundary contour pixels. To formulate the boundary contour pixels of pneumothrax, an edge detector is used to determine whether a pixel is a boundary pixel or not, and then the boundary range is cross-expanded by morphological dilation. Therefore, a spatial weighted cross-entropy loss (SW-CEL) is proposed by considering the different weights of target, background and boundary [30]. As shown in Fig. 4, spatial weight maps generated from the ground-truth images are used to calculate the weight loss of each pixel in the cross-entropy loss. The spatially weighted cross-entropy (SW-CEL) loss can be formulated as:where X denotes the training samples, W denotes the set of learnable weights, \(W=(w_1, w_2, \ldots , w_l)\), and \(w_l\) denotes the weight matrix of the \(l{\mathrm{th}}\) layer. \(p(t_i|x_i;W)\) represents the probability prediction for a pixel \(x_i\), and \(t_i\) is the target label of the pixel \(x_i\), \((x_i\in X)\). \(w_{map}(x_i)\) is the estimated weight for each pixel \(x_i\), which can be defined as:where \(F_T(x_i)=\left\{ \begin{array}{ll} 0, &{}x_i\notin T_c \\ 1,&{} x_i\in T_c \end{array} \right.\) and \(F_B(x_i)=\left\{ \begin{array}{ll} 0, &{}x_i\notin B_c \\ 1, &{}x_i\in B_c \end{array} \right.\). C denotes the set of all ground truth classes, i.e., pneumothorax class and background class. For each chest X-ray image, N denotes the set of total pixels and \(T_c\) denotes the set of pixels corresponding to each class c, \(c\in C\), and \(B_c\) denotes the boundary contour pixel set, \(B_c \subset T_c \subset N\). \(F_T(x_i)\) and \(F_B(x_i)\) denote the indicator functions defined on the subsets \(T_c\) and \(B_c\), respectively.

Most of previous studies of pneumothorax on chest X-ray mainly focous on PTX or not PTX diagnosis with image-level annotation. The learning of pneumothorax diagnosis with image-level annotation is a typical weakly supervised learning method, which often leads to inaccurate locations of pneumothorax lesions because the locations of pneumothorax lesions are not marked. Pneumothorax segmentation can accurately provide pixel-level lesion locations and better assist radiologists in pneumothorax diagnosis. In this study, we propose a pixel-wise level supervised network for the automatic segmentation and diagnosis of PTX (see Fig. 1). Since the predicted segmentation maps are the result of binary pixel-wise classification network, a simple classifier is added and a threshold is set to classify the predicted segmentation maps. We specify that if the predicted segmentation map is greater than a threshold, it is pneumothorax; otherwise, it is non-pneumothorax. If the threshold is too small, it may be segmentation noise; If the threshold is too high, small pneumothorax may be missed. Therefore, the threshold is empirically set to 50 pixels for pneumothorax diagnosis according to the predicted segmentation maps.

The study data was conducted with three-stage procedures. The first stage searched a keyword “pneumothorax” in picture archiving and communications system (PACS) of our institution to obtain all relevant chest radiographs and radiology reports. In second stage, the key word “pneumothorax” was identified in the radiological report, and those without pneumothorax were classified as non-pneumothorax (Non-PTX) group, while those with pneumothorax were classified as pneumothorax (PTX) group. Third, all image data in PTX group was pixel-wisely annotated by three medical students and then revised by an experienced radiologist.

Our eligible sample included a total of 11,051 front-view chest X-ray images (5566 cases of PTX and 5485 cases of Non-PTX). We named this dataset as “PX-ray”. As shown in Table 1, the PX-ray dataset was randomly divided into the training, validation and test sets by stratified sampling strategy, so as to ensure that the ratio of PTX group and Non-PTX group in each set was the same.

MPA is the average ratio of the accuractely classified pixels on the classes of PTX and non-PTX, defined as:where N denotes the number of samples, C denotes the number of classes, \(p_c\) denotes the number of the accuractely classified pixels of class c, and \(P_c\) denotes all pixels of class c in the ground truths. More importantly, we defined the pixel-wise accuracy (PA) of the PTX group class as \({\mathrm {PA}}_1\).DSC is a standard measure for segmentation evaluation by calculating the overlap rate between the ground-truth map and the predicted segmentation map.where \(A_c\) denotes all pixels of class c in the predicted segmentation map and \(B_c\) denotes all pixels of class c in the ground-truth map. More importantly, we define the DSC of the PTX group class as \({\mathrm {DSC}}_1\).Hausdorff (HD) metric is also used to measure the contour distance between the ground-truth map and the predicted segmentation map, which can be defined as:where P and G are the pixel sets of the predicted segmentation map and the ground-truth contours, respectively. The smaller the Hausdorff value, the higher the matching degree of the two contours.

All experiments in this study were conducted on Nvidia Tesla V100 GPU server. The weights of the PTX segmentation network were initialized with HeUniform [31]. We used Adam optimizer (\(\beta _1=0.9\), \(\beta _2=0.999\)) with learning rate of 1e\({-}\)4 and weight decay of 1e\({-}\)4 to train the segmentaion network model for 200 epochs. During the training process of all models, data augmentation was performed by random horizontal flips and the validation set was used to early stop the training process. We monitored the dice similarity coefficient (DSC) score in the pneumothorax group with patience value of 20 epochs.

Table 2 shows that our PTX segmentation network, i.e., MS_scSE_FC-DenseNet, outperforms U-Net, SegNet, DeepLab v3+, DenseASPP and original FC-DenseNet in terms of MPA with \(0.93\pm 0.13\), \({\mathrm {PA}}_1\) with \(0.86\pm 0.27\), DSC with \(0.92\pm 0.14\) and \({\mathrm {DSC}}_1\) with \(0.84\pm 0.27\). Meanwhile, our network MS_scSE_FC-DenseNet performs better than the original FC-DenseNet, and MS_scSE_U-Net performs better than the original U-Net, which shows that the performance of the network embedded with the multi-scale module and scSE modules is better than that without them. This indicates that the proposed multi-scale module and scSE module play an important role in improving the performance of the segmentation networks for PTX. In addition, compared with the original FC-DenseNet, the parameter number of the proposed network increased by 10.59%, but is still much less than that of other segmentation networks. Our method has a low time cost in terms of giga floating-point operations per second (GFLOPS).

Figure 5 shows some result cases of large, moderate and small pneumothorax with different segmentation algorithms. For each case, we present the orginal chest X-ray image, the ground-truth image, and the segmentation results of the comparison methods and our proposed method MS_scSE_FC-DenseNet, as well as the corresponding \({\mathrm {DSC}}_1\) and HD scores. It can be found that our method performs better with a larger \({\mathrm {DSC}}_1\) and a smaller HD scores, which can more accurately help radiologists find the pneumothorax area. In addition, as shown in the bottom line of Fig. 5, our algorithm can segement the small bilateral thoracic regions that are very difficult for radiologists to manually label, indicating the potential of our method for clinical computer-assisted diagnosis.

Figure 6 shows qualitative evaluation of our proposed PTX segmentation network and the five comparison frameworks on the PTX segmentation task. The X-axis represents the intervals of DSC score, and the Y-axis represents the number of samples falling into the DSC intervals from the X-axis. Compared with other frameworks, our segmentation network has the largest number of sample size in the range of [0.9, 1.0] and the smallest sample size in the range of [0, 0.6].

The quantitative performances of pneumothorax diagnosis with different models are shown in Table 3. Our network shows the best results in terms of accuracy, sensitivity, negative predictive value (NPV) and \(F_1\)-score. The original FC-DenseNet shows the best performance on specificity and positive predictive value (PPV). In addition, all the segmentation networks used for PTX diagnosis achieve good performance. This indicates great potential for the pixel-wise level supervised networks. The pixel-level supervised network not only provides image-level information but also provides pneumothorax location and size information, which is of great help to network learning.

Table 4 discusses that our pneumothorax segmentation network performance with different growth rate (k) parameters. Note that according to students’ t-test of the two independent samples, the number with \(*\) in the table represents that there is a significant difference (\(p<0.05\)) between the model with \(k=12\) and other models. We can see that under the same framework, the results grow steadily as the value of k increases. The segmentation network with \(k=12\) shows the best performance. Therefore, we use \(k=12\) model as our final network for pneumothorax segmentation.

Table 5 discusses the segmentation performance of three different loss functions, including CEL, W-CEL and SW-CEL. In order to further evaluate the performance of the loss function, we carry out experiments on our proposed network and the previous state-of-the-art networks including U-Net [18], SegNet [32], DeepLab v3+ [33], DenseASPP [34] and FC-DenseNet [25]. The statistical T-tests on the test set indicates that models trained on SW-CEL had no statistical significance in terms of DSC scores, while most models trained with SW-CEL showed the best performance in terms of Hausdorff distance scores. This indicates that the weight penalty for contour pixels could help to learn boundary contour accurately.