QuickNAT: A fully convolutional network for quick and accurate segmentation of neuroanatomy

Highlights

• Introduces a deep learning based whole brain segmentation tool called QuickNAT, processing each 3D MRI T1 brain scans in 20 secs.

• The high segmentation accuracy of QuickNAT was evaluated on 5 different benchmark datasets, containing a wide age range, subjects with different pathologies (AD, MCI and CN), and different scanners (1.5T and 3.0T).

• QuickNAT can be effectively used for longitudinal studies as it performs well in test-retest and multi-center experiments.

Abstract

Whole brain segmentation from structural magnetic resonance imaging (MRI) is a prerequisite for most morphological analyses, but is computationally intense and can therefore delay the availability of image markers after scan acquisition. We introduce QuickNAT, a fully convolutional, densely connected neural network that segments a MRI brain scan in 20 s. To enable training of the complex network with millions of learnable parameters using limited annotated data, we propose to first pre-train on auxiliary labels created from existing segmentation software. Subsequently, the pre-trained model is fine-tuned on manual labels to rectify errors in auxiliary labels. With this learning strategy, we are able to use large neuroimaging repositories without manual annotations for training. In an extensive set of evaluations on eight datasets that cover a wide age range, pathology, and different scanners, we demonstrate that QuickNAT achieves superior segmentation accuracy and reliability in comparison to state-of-the-art methods, while being orders of magnitude faster. The speed up facilitates processing of large data repositories and supports translation of imaging biomarkers by making them available within seconds for fast clinical decision making.

Introduction

Magnetic Resonance Imaging (MRI) provides detailed in-vivo insights about the morphology of the human brain, which is essential for studying development, aging, and disease (Giedd et al., 1999; Draganski et al., 2004; Shaw et al., 2006; Raznahan et al., 2012; Alexander-Bloch and Giedd, 2013; Wachinger et al., 2016; Lerch et al., 2017). In order to access measurements like volume, thickness, or shape of a structure, the neuroanatomy needs to be segmented, which is a time-consuming process when performed manually (Fischl et al., 2002). Computational tools have been developed that can fully automatically segment brain MRI scans by warping a manually segmented atlas to the target scan (Fischl et al., 2002; Ashburner and Friston, 2005; Rohlfing et al., 2005; Svarer et al., 2005). Such approaches have two potential shortcomings: (i) the estimation of the 3D deformation field for warping is computationally intense, and (ii) lack of homologies may result in erroneous segmentations of the cortex (Lerch et al., 2017). Due to these drawbacks, existing atlas-based methods require hours of processing time for each scan and may result in sub-optimal solutions.

We propose a method for the Quick segmentation of NeuroAnaTomy (QuickNAT) in MRI T1 scans based on a deep fully convolutional neural network (F-CNN) that runs in seconds on GPUs, compared to hours for existing atlas-based methods. We believe that this speed up by several orders of magnitude can have a wide impact on neuroimaging: processing of large datasets can be performed on a single GPU workstation, instead of a computing cluster; quantitative morphological measurements can be derived from a scan within seconds, boosting its translation. Furthermore, the fast processing speed allows for sampling multiple segmentations in a reasonable amount of time to estimate segmentation uncertainty for automated quality control (Roy et al., 2018). Beside its speed, QuickNAT produces state-of-the-art segmentation accuracy as demonstrated on multiple datasets covering a wide age range, different field strengths, and pathologies. Moreover, it yields effect sizes that are closer to those of manual segmentations and therefore offers advantages for group analyses. Finally, QuickNAT exhibits high test-retest accuracy making it useful for longitudinal studies.

Deep learning models have had ample success over the last years, but require vast amounts of annotated data for effective training (LeCun et al., 2015). The task of semantic image segmentation is dominated by F-CNN models in computer vision (Long et al., 2015). The limited availability of training data with manual annotations presents the main challenge in extending F-CNN models to brain segmentation. To address this challenge, we introduce a new training strategy (Fig. 2) that exploits large brain repositories without manual labels and small repositories with manual labels. First, we apply existing software tools (e.g., FreeSurfer (Fischl et al., 2002)) to segment scans without annotations. We refer to these automatically generated segmentations as auxiliary labels, which we use to pre-train the network. Auxiliary labels may not be as accurate as expert annotations; however, they allow us to efficiently leverage the vast amount of initially unlabeled data for supervised training of the network. It also makes the network familiar with a wide range of morphological variations of different brain structures that may exist in a wide population. In the second step, we fine-tune (i.e., continue training) the previous network with smaller manually annotated data. Pre-training provides a good prior initialization of the network, such that scarce manual annotations are optimally utilized to achieve high segmentation accuracy. As a side note, we observed that a network trained only on FreeSurfer segmentations can produce more accurate results than FreeSurfer itself.

QuickNAT consists of three 2D F-CNNs operating on coronal, axial and sagittal views followed by a view aggregation step to infer the final segmentation (Fig. 3). Each F-CNN has the same architecture and is inspired by the traditional encoder/decoder based U-Net architecture with skip connections (Ronneberger et al., 2015), enhanced with unpooling layers (Noh et al., 2015) (Fig. 1). We also introduce dense connections (Huang et al., 2016) within each encoder/decoder block to aid gradient flow and to promote feature re-usability, which is essential given the limited amount of training data. The network is optimized using a joint loss function of multi-class Dice loss and weighted logistic loss, where weights compensate for high class imbalance in the data and encourage proper estimation of anatomical boundaries.

The two main methodological innovations of QuickNAT are the training strategy with auxiliary labels and the F-CNN architecture. To the best of our knowledge, this is the first work to conduct such a large number of experiments on highly heterogeneous datasets to evaluate the robustness of an F-CNN for brain segmentation. The code and trained model are available as extensions of MatConvNet (Vedaldi and Lenc, 2015) at https://github.com/abhi4ssj/QuickNATv2. This is an extension of our early work (Roy et al., 2017), where we introduced the concept of pre-training with auxiliary labels. In this work, we improved upon the architecture, segment more brain structures and show exhaustive experiments for a wide range of possibilities to substantiate the effectiveness of the framework.