Elsevier

Medical Image Analysis

Volume 45, April 2018, Pages 108-120
Medical Image Analysis

A novel multi-atlas strategy with dense deformation field reconstruction for abdominal and thoracic multi-organ segmentation from computed tomography

https://doi.org/10.1016/j.media.2018.02.001Get rights and content

Highlights

  • A multi-atlas approach to abdominal/thoracic organs segmentation is presented.

  • The proposed method explores the spatial relationships between nearby organs.

  • It was validated for the segmentation of twelve organs in computed tomography.

  • Its accuracy was assessed in an online segmentation benchmark – VISCERAL anatomy.

  • A competitive performance was obtained against state-of-the-art methods.

Abstract

Anatomical evaluation of multiple abdominal and thoracic organs is generally performed with computed tomography images. Owing to the large field-of-view of these images, automatic segmentation strategies are typically required, facilitating the clinical evaluation. Multi-atlas segmentation (MAS) strategies have been widely used with this process, requiring multiple alignments between the target image and the set of known datasets, and subsequently fusing the alignment results to obtain the final segmentation. Nonetheless, current MAS strategies apply a global alignment of a deformable object, per organ, subdividing the segmentation process into multiple ones and losing the spatial information among nearby organs. This paper presents a novel MAS approach. First, a coarse-to-fine method with multiple global alignments (one per organ) is used. To make the method spatially coherent, these individual organs’ global transformations are then fused in one using a dense deformation field reconstruction strategy. Second, from the candidate segmentations obtained, the final segmentation is estimated through an organ-based label fusion approach. The proposed method is evaluated and compared against a conventional MAS strategy through the segmentation of twelve abdominal and thoracic organs from the VISCERAL Anatomy benchmark. Average Dice coefficients for the liver, spleen, lungs and kidneys are all higher than 90%, are around 85% for the aorta, trachea and sternum and 70% for the pancreas, urinary bladder and gallbladder. The novel MAS strategy, with dense deformation field reconstruction, shows competitive results against other state-of-the-art methods, proving its added value for the segmentation of abdominal and thoracic organs, mainly for highly variable organs.

Introduction

Non-invasive clinical assessment of the inner body is usually performed through medical imaging. Imaging modalities, such as magnetic resonance imaging (MRI) or computed tomography (CT), are used to evaluate multiple organs through a full or partial body acquisition. However, because of the huge amount of data acquired, a correct 3D assessment of the target structures is difficult and time-consuming to obtain. Thus, a multitude of (semi-)automatic segmentation techniques were presented and validated in research (Heimann and Meinzer, 2009, Iglesias and Sabuncu, 2015, Jimenez-Del-Toro et al., 2016). Automated inter- and intra-observer variability are usually reduced, whereas any time-consuming manual interaction is largely minimised. Nonetheless, manual corrections are still typically required to obtain accurate results.

Several methods have been proposed for the segmentation of individual organs in CT images (Cuingnet et al., 2012, Okada et al., 2007, Roth et al., 2015, Ruskó et al., 2009, Xu et al., 2014). However, these techniques do not take full advantage of the large field-of-view (FOV) acquired, disregarding valuable information, such as the co-localization and interrelationship among organs (Cerrolaza et al., 2015, Gass et al., 2014, Linguraru et al., 2012, Okada et al., 2015, Wang and Smedby, 2015). Recently, simultaneous segmentation of multiple organs has been investigated, using deformable models (Uzunbas et al., 2013), pixel classification (Criminisi et al., 2013, Selver, 2014), active shape/appearance models (Cerrolaza et al., 2015, Heimann and Meinzer, 2009, Okada et al., 2015), deformable models with a shape prior (Kohlberger et al., 2011), or atlas-based methods (Iglesias and Sabuncu, 2015, Wolz et al., 2013, Xu et al., 2015). Contrarily to others, atlas-based methods use the entire content of labelled images (i.e. atlas), instead of a model-based average representation, making these methods more flexible when dealing with expected anatomical variations between subjects (Iglesias and Sabuncu, 2015).

Multi-atlas segmentation approaches (MAS) are atlas-based methods that spatially align each atlas with the unlabelled image via an image registration strategy (Klein et al., 2010). They fuse all the information, only at the end, through a label fusion approach. Thus, MASs maintain the flexibility and variability available in the atlases, obtaining more robust results than other atlas-based approaches. Nonetheless, several limitations are still found in the state-of-the-art strategies when performing the registration of multiple objects. Please refer to the extensive survey on MAS strategies found in (Iglesias and Sabuncu, 2015).

Within the MAS methods for multiple organ segmentation, recent strategies apply a global and/or deformable alignment per each organ (Jimenez-Del-Toro and Muller, 2014, Wolz et al., 2013). Unfortunately, this methodology subdivides the segmentation process into multiple problems (i.e. one segmentation per individual organ), failing to correctly explore the spatial information between nearby organs. Moreover, without an additional post-processing, these strategies do not guarantee that no overlaps exist between different labels, meaning that the same anatomical region can simultaneously belong to two or more labels. Consequently, with the previously described MAS pipeline, label fusion strategies can only be applied per organ. Furthermore, because the registration was centred in specific structures, the alignment of neighbour anatomical structures was neglected, leading to the presence of spatial incoherencies in the warped atlases of nearby organs. Indeed, such incoherencies can prevent the application of sophisticated label fusion methods, mainly patched-based approaches. Particularly, aside from voxel intensity evaluations, patched-based approaches explore the anatomical information around each voxel (e.g. spatial relationships between nearby structures), increasing the accuracy of MAS strategies (Iglesias and Sabuncu, 2015). Recently, patches-based methods, which interpret and correlate the information from local patches to further improve MAS accuracy, have been presented (Asman and Landman, 2013, Wang et al., 2013, Wolz et al., 2013).

Inspired by the previous papers, we propose a novel MAS strategy that uses a coarse-to-fine method with multiple global alignments (i.e. one per organ) (Jimenez-Del-Toro and Muller, 2014, Wolz et al., 2013). Nevertheless, to make the method spatially coherent, these individual organs’ global transformations are fused into one before applying a deformable alignment stage. From the candidate segmentations obtained, the final segmentation is then estimated via a two-stage label fusion approach based on statistical selection and local weight voting. By using spatially coherent warped atlases, an improvement of the accuracy during estimation of the anatomical correlations between atlases through the label fusion stage can be expected (Wang et al., 2013). This new MAS formulation has the potential to improve the segmentation of highly variable organs by using inter-organ spatial relationships while preventing overlaps between structures and spatial inconsistencies.

Overall, the current work introduces two novelties. The first is an extension of conventional organ-based MAS strategies through the addition of a novel dense deformation field reconstruction module. In detail, the proposed module fuses all single organ alignment results, generating full-body and spatial coherent patient templates, allowing us to explore the spatial relationships between nearby organs for a refined registration, improve the segmentation accuracy from label fusion strategies that explore anatomical information and use multi-label fusion methods. The second novelty entails validation of the proposed MAS pipeline in an online segmentation benchmark with twelve organs from the ventral cavity.

This paper is structured as follows. Section 2 describes the methodology of the proposed framework, with implementation details presented in Section 3. Section 4 introduces the validation experiments, followed by results in Section 5. In Section 6, the performance of each module of the proposed framework is discussed and compared with state-of-the-art results. The conclusions are given in Section 7.

Section snippets

General overview

The proposed approach aims to develop a robust method to segment multiple structures from the ventral cavity in high-resolution datasets. Such a tool can be used to improve the diagnosis and surgical planning for multiple procedures (e.g. surgical navigation, cancer detection). We divide the framework into two conceptual blocks (Fig. 1).

The first block corresponds to the atlas alignment and is divided into three modules. The first estimates an initial coarse alignment between the atlases and

Implementation details

Because of the large FOV of the CT images used (i.e. trunk FOV), multiple structures (e.g. table) are acquired beyond the relevant organs. Thus, to improve the method's performance, we automatically pre-process the datasets by removing the surrounding regions through a fixed thresholding (HU = −200) followed by a fill-holes technique.

Subsequently, in both non-deformable alignments, affine transformations are used (Klein et al., 2010). Moreover, the ROIs in the local alignments are defined by

Data

To evaluate the performance of the proposed framework, the Visual Concept Extraction Challenge in Radiology (VISCERAL) Anatomy3 benchmark was used (Langs et al., 2012). Contrast-enhanced CT volumes were acquired from patients with malignant lymphoma at the same hospital with the same imaging protocols. The complete CT dataset (training and testing dataset) contains 30 volumes with anisotropic pixel spacing, ranging from 0.604 to 0.793 mm having a spacing between slices of at least 3 mm. The

Training dataset validation

The performance of the proposed algorithm is addressed throughout this experiment using the entire training dataset. Regarding the computational time, the proposed strategy required approximately 20 h per patient.

Average DSC of the entire training dataset after the global non-deformable module (Section 2.2.1), local non-deformable (Section 2.2.2), deformation field reconstruction (Section 2.3), deformable (Section 2.4) and label fusion (Section 2.5) are presented in Fig. 4. Considering the

Discussion

The current work presents a fully automated multi-organ MAS approach for CT images. The proposed method uses a novel dense deformation field reconstruction module within a coarse-to-fine registration strategy, exploring the spatial relationship between organs during the entire process, ultimately improving the segmentation of highly variable and small organs. The proposed method was tested on 12 organs from the thoracic and abdominal cavities, showing high accuracy against state-of-the-art

Conclusion

In summary, a novel fully automatic MAS method to segment structures from the ventral body cavity in CT datasets was presented. The novel MAS strategy used a novel dense deformation field reconstruction block that explored the spatial relationships between organs during the alignment stage, proving its advantage for the segmentation of highly variable and small organs. The proposed method showed statistically significantly better results than a conventional MAS strategy. Moreover, it showed

Conflict of interest

The authors declare that they have no conflict of interest

Acknowledgements

The authors acknowledge Fundação para a Ciência e a Tecnologia (FCT), Portugal and the European Social Found, European Union, for funding support through the “Programa Operacional Capital Humano” (POCH) in the scope of the PhD grants SFRH/BD/93443/2013 (S. Queirós), SFRH/BD/95438/2013 (P. Morais), and PD/BDE/113597/2015 (J. Gomes-Fonseca).

Moreover, authors gratefully acknowledge the funding of the projects NORTE-01-0145-FEDER-000013 and NORTE-01-0145-FEDER-024300, supported by Northern Portugal

References (42)

  • T. Tong et al.

    Discriminative dictionary learning for abdominal multi-organ segmentation

    Med. Image Anal.

    (2015)
  • Z. Xu et al.

    Efficient multi-atlas abdominal segmentation on clinically acquired CT with SIMPLE context learning

    Med. Image Anal.

    (2015)
  • V. Arsigny et al.

    A fast and log-Euclidean polyaffine framework for locally linear registration

    J. Math. Imaging Vis.

    (2009)
  • X. Artaechevarria et al.

    Combination strategies in multi-atlas image segmentation: application to brain MR data

    IEEE Trans. Med. Imaging

    (2009)
  • Cha, K.H., Hadjiiski, L., Samala, R.K., Chan, H., Caoili, E.M., Cohan, R.H., 2016. Urinary bladder segmentation in CT...
  • R. Cuingnet et al.

    Automatic detection and segmentation of kidneys in 3D CT images using random forests

    Medical Image Computing and Computer-Assisted Intervention–MICCAI 2012

    (2012)
  • T. Gass et al.

    Multi-atlas segmentation and landmark localization in images with large field of view

    International MICCAI Workshop on Medical Computer Vision

    (2014)
  • Hadjiiski, L., Chan, H., Cohan, R.H., Caoili, E.M., Law, Y., Cha, K., Zhou, C., Hadjiiski, L., Chan, H., Cohan, R.H.,...
  • B. He et al.

    Fully automatic multi-organ segmentation based on multi-boost learning and statistical shape model search

    VISCERAL Challenge@ ISBI

    (2015)
  • M.P. Heinrich et al.

    Multi-modal multi-atlas segmentation using discrete optimisation and self-similarities

    VISCERAL Challenge@ ISBI

    (2015)
  • O. Jimenez-Del-Toro et al.

    Hierarchic anatomical structure segmentation guided by spatial correlations (AnatSeg-Gspac): VISCERAL anatomy3

    VISCERAL Challenge@ ISBI

    (2015)
  • Cited by (33)

    • Multi-organ segmentation network for abdominal CT images based on spatial attention and deformable convolution

      2023, Expert Systems with Applications
      Citation Excerpt :

      To address this problem, knowledge-based methods obtain anatomical knowledge of different organs from labelled datasets, which improve the robustness and accuracy of multi-organ segmentation. Common knowledge-based methods include multi-atlas (Wolz et al., 2013; Tong et al., 2015; Oliveira et al., 2018), statistical shape models (Cerrolaza et al., 2015; Okada et al., 2015) and deep learning models (Gibson et al., 2018; Heinrich et al., 2019; Zhang et al., 2020). The comparisons of different methods in terms of dataset size, segmentation accuracy and time cost are listed in Table 1.

    • WORD: A large scale dataset, benchmark and clinical applicable study for abdominal organ segmentation from CT image

      2022, Medical Image Analysis
      Citation Excerpt :

      It is worth pointing out that the comparisons are unfair, as the dataset and experimental settings for each method are different. It can be found that the results in WORD dataset are more comprehensive and competitive than those in recent works (Oliveira et al., 2018; Wang et al., 2019; Tang et al., 2021; Liang et al., 2021; Chen et al., 2021b), but it is still not good enough for clinical application. So, we think the abdominal multi-organ segmentation task is not a well-solved problem.

    • Adaptable volumetric liver segmentation model for CT images using region-based features and convolutional neural network

      2022, Neurocomputing
      Citation Excerpt :

      Beside, some state-of-the-art evaluation results are available directly trained and tested on the VISCERAL database itself. Such results are also added in the lower part of Table 5 for comparison, including various techniques: graph-cut based [55], shape model based [56], multi-atlas based [57,58], self-supervised 3D feature learning [59]. The obtained results show that our proposed method successfully generalizes and performs well on previously unseen data too.

    View all citing articles on Scopus
    1

    Both authors contributed equally.

    View full text