QuickNAT: A fully convolutional network for quick and accurate segmentation of neuroanatomy
Graphical abstract
Introduction
Magnetic Resonance Imaging (MRI) provides detailed in-vivo insights about the morphology of the human brain, which is essential for studying development, aging, and disease (Giedd et al., 1999; Draganski et al., 2004; Shaw et al., 2006; Raznahan et al., 2012; Alexander-Bloch and Giedd, 2013; Wachinger et al., 2016; Lerch et al., 2017). In order to access measurements like volume, thickness, or shape of a structure, the neuroanatomy needs to be segmented, which is a time-consuming process when performed manually (Fischl et al., 2002). Computational tools have been developed that can fully automatically segment brain MRI scans by warping a manually segmented atlas to the target scan (Fischl et al., 2002; Ashburner and Friston, 2005; Rohlfing et al., 2005; Svarer et al., 2005). Such approaches have two potential shortcomings: (i) the estimation of the 3D deformation field for warping is computationally intense, and (ii) lack of homologies may result in erroneous segmentations of the cortex (Lerch et al., 2017). Due to these drawbacks, existing atlas-based methods require hours of processing time for each scan and may result in sub-optimal solutions.
We propose a method for the Quick segmentation of NeuroAnaTomy (QuickNAT) in MRI T1 scans based on a deep fully convolutional neural network (F-CNN) that runs in seconds on GPUs, compared to hours for existing atlas-based methods. We believe that this speed up by several orders of magnitude can have a wide impact on neuroimaging: processing of large datasets can be performed on a single GPU workstation, instead of a computing cluster; quantitative morphological measurements can be derived from a scan within seconds, boosting its translation. Furthermore, the fast processing speed allows for sampling multiple segmentations in a reasonable amount of time to estimate segmentation uncertainty for automated quality control (Roy et al., 2018). Beside its speed, QuickNAT produces state-of-the-art segmentation accuracy as demonstrated on multiple datasets covering a wide age range, different field strengths, and pathologies. Moreover, it yields effect sizes that are closer to those of manual segmentations and therefore offers advantages for group analyses. Finally, QuickNAT exhibits high test-retest accuracy making it useful for longitudinal studies.
Deep learning models have had ample success over the last years, but require vast amounts of annotated data for effective training (LeCun et al., 2015). The task of semantic image segmentation is dominated by F-CNN models in computer vision (Long et al., 2015). The limited availability of training data with manual annotations presents the main challenge in extending F-CNN models to brain segmentation. To address this challenge, we introduce a new training strategy (Fig. 2) that exploits large brain repositories without manual labels and small repositories with manual labels. First, we apply existing software tools (e.g., FreeSurfer (Fischl et al., 2002)) to segment scans without annotations. We refer to these automatically generated segmentations as auxiliary labels, which we use to pre-train the network. Auxiliary labels may not be as accurate as expert annotations; however, they allow us to efficiently leverage the vast amount of initially unlabeled data for supervised training of the network. It also makes the network familiar with a wide range of morphological variations of different brain structures that may exist in a wide population. In the second step, we fine-tune (i.e., continue training) the previous network with smaller manually annotated data. Pre-training provides a good prior initialization of the network, such that scarce manual annotations are optimally utilized to achieve high segmentation accuracy. As a side note, we observed that a network trained only on FreeSurfer segmentations can produce more accurate results than FreeSurfer itself.
QuickNAT consists of three 2D F-CNNs operating on coronal, axial and sagittal views followed by a view aggregation step to infer the final segmentation (Fig. 3). Each F-CNN has the same architecture and is inspired by the traditional encoder/decoder based U-Net architecture with skip connections (Ronneberger et al., 2015), enhanced with unpooling layers (Noh et al., 2015) (Fig. 1). We also introduce dense connections (Huang et al., 2016) within each encoder/decoder block to aid gradient flow and to promote feature re-usability, which is essential given the limited amount of training data. The network is optimized using a joint loss function of multi-class Dice loss and weighted logistic loss, where weights compensate for high class imbalance in the data and encourage proper estimation of anatomical boundaries.
The two main methodological innovations of QuickNAT are the training strategy with auxiliary labels and the F-CNN architecture. To the best of our knowledge, this is the first work to conduct such a large number of experiments on highly heterogeneous datasets to evaluate the robustness of an F-CNN for brain segmentation. The code and trained model are available as extensions of MatConvNet (Vedaldi and Lenc, 2015) at https://github.com/abhi4ssj/QuickNATv2. This is an extension of our early work (Roy et al., 2017), where we introduced the concept of pre-training with auxiliary labels. In this work, we improved upon the architecture, segment more brain structures and show exhaustive experiments for a wide range of possibilities to substantiate the effectiveness of the framework.
Section snippets
Methods
Given an input MRI brain scan I, we want to infer its segmentation map S, which indicates 27 cortical and subcortical structures. Given a set of scans and its corresponding segmentations , we want to learn a function . We express this function as an F-CNN model, termed QuickNAT, which is detailed below.
Experimental datasets
We use nine brain MRI datasets in our experiments. We use five datasets with manual annotations to evaluate segmentation accuracy. Three datasets were used for testing reliability of the segmentation framework. Table 1 summarizes the number of subjects per dataset, the age range, the diagnosis, and the annotated structures. Present diagnoses are Alzheimer's disease (AD), mild cognitive impairment (MCI), and psychiatric disorders. Details about acquisition protocol used in each of the datasets
Experiments and results
We evaluate QuickNAT in a comprehensive series of eight experiments to assess accuracy, reproducibility, and sensitivity on a large variety of neuroimaging datasets, summarized in Table 2. In all experiments, we pre-train QuickNAT on 581 MRI volumes from the IXI dataset to get auxiliary segmentations from FreeSurfer (Fischl et al., 2002). We conducted 5 experiments to evaluate the segmentation accuracy (experiments 1 to 5; Sec. 4.1 and Sec. 4.2), and another 3 experiments (experiments 6 to 8;
Comparison with deep learning approaches
Recently, convolutional neural networks have been proposed for brain segmentation (Chen et al., 2018; Dolz et al., 2018; Fedorov et al., 2017; Wachinger et al., 2018; Moeskops et al., 2016). DeepNAT (Wachinger et al., 2018) reported competitive results on the MALC data, but as shown in Table 3, QuickNAT yields significantly higher accuracy, while requiring only seconds (Fig. 6). Dolz et al. (2018) proposed a network for segmenting 8 structures based on skull-stripped and intensity normalized
Conclusion
We have introduced QuickNAT, a deep learning based method for brain segmentation that runs in seconds, achieving superior performance with respect to existing methods and being orders of magnitudes faster in comparison to patch-based CNNs and atlas-based approaches. We have demonstrated that QuickNAT generalizes well to other, unseen datasets (training data different to testing) and yields high segmentation accuracy across diagnostic groups, scanner field strengths, and age, while producing
Acknowledgment
Support for this research was provided in part by the Bavarian State Ministry of Education, Science and the Arts in the framework of the Center Digitisation.Bavaria (ZD.B). We thank Neuromorphometrics Inc. for providing manual annotations, neuroimaging initiatives for sharing data, and NVIDIA corporation for GPU donation. We would also like to thank Dr. Sebastian Pölsterl for proofreading the manuscript and providing feedback. Data collection and sharing was funded by the Alzheimer's Disease
References (51)
- et al.
Unified segmentation
Neuroimage
(2005) - et al.
A reproducible evaluation of ants similarity metric performance in brain image registration
Neuroimage
(2011) - et al.
Training labels for hippocampal segmentation based on the EADC-ADNI harmonized hippocampal protocol
Alzheimer's Dementia
(2015) - et al.
VoxResNet: deep voxelwise residual networks for brain segmentation from 3d MR images
Neuroimage
(2018) - et al.
Cortical surface-based analysis: I. Segmentation and surface reconstruction
Neuroimage
(1999) - et al.
3d fully convolutional networks for subcortical segmentation in MRI: a large-scale study
Neuroimage
(2018) - et al.
Whole brain segmentation: automated labeling of neuroanatomical structures in the human brain
Neuron
(2002) - et al.
Cortical surface-based analysis: II: inflation, flattening, and a surface-based coordinate system
Neuroimage
(1999) - et al.
Brain tumor segmentation with deep neural networks
Med. Image Anal.
(2017) - et al.
Tracking pathophysiological processes in Alzheimer's disease: an updated hypothetical model of dynamic biomarkers
Lancet Neurol.
(2013)
Fsl. Neuroimage
Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation
Med. Image Anal.
A comprehensive testing protocol for MRI neuroanatomical segmentation techniques: evaluation of a novel lateral ventricle segmentation method
Neuroimage
A Bayesian model of shape and appearance for subcortical brain segmentation
Neuroimage
MR-based automatic delineation of volumes of interest in human brain PET images using probability maps
Neuroimage
Segmentation of the hippocampus by transferring algorithmic knowledge for large cohort processing
Med. Image Anal.
Improving automated multiple sclerosis lesion segmentation with a cascaded 3d convolutional neural network approach
Neuroimage
DeepNAT: deep convolutional neural network for segmenting neuroanatomy
Neuroimage
Imaging structural co-variance between human brain regions
Nat. Rev. Neurosci.
Formulating spatially varying performance in the statistical fusion framework
IEEE Trans. Med. Imag.
Segnet: a Deep Convolutional Encoder-decoder Architecture for Image Segmentation
The hippocampus in Aging and Disease: from Plasticity to Vulnerability
Deep 3d convolutional encoder networks with shortcuts for multiscale feature integration applied to multiple sclerosis lesion segmentation
IEEE Trans. Med. Imag.
Neuroplasticity: changes in grey matter induced by training
Nature
Almost Instant Brain Atlas Segmentation for Large-scale Studies
Cited by (223)
Neural deformation fields for template-based reconstruction of cortical surfaces from MRI
2024, Medical Image AnalysisAutomated Segmentation of Sacral Chordoma and Surrounding Muscles Using Deep Learning Ensemble
2023, International Journal of Radiation Oncology Biology PhysicsSelf-supervised-RCNN for medical image segmentation with limited data annotation
2023, Computerized Medical Imaging and Graphics
- 1
Data used in preparation of this article were obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database (adni.loni.usc.edu). As such, the investigators within the ADNI contributed to the design and implementation of ADNI and/or provided data but did not participate in analysis or writing of this report. A complete listing of ADNI investigators can be found at: http://adni.loni.usc.edu/wp-content/uploads/how_to_apply/ADNI_Acknowledgement_List.pdf.