Efficient and generalizable statistical models of shape and appearance for analysis of cardiac MRI

https://doi.org/10.1016/j.media.2007.12.003Get rights and content

Abstract

We present a framework for the analysis of short axis cardiac MRI, using statistical models of shape and appearance. The framework integrates temporal and structural constraints and avoids common optimization problems inherent in such high dimensional models. The first contribution is the introduction of an algorithm for fitting 3D active appearance models (AAMs) on short axis cardiac MRI. We observe a 44-fold increase in fitting speed and a segmentation accuracy that is on par with Gauss–Newton optimization, one of the most widely used optimization algorithms for such problems. The second contribution involves an investigation on hierarchical 2D + time active shape models (ASMs), that integrate temporal constraints and simultaneously improve the 3D AAM based segmentation. We obtain encouraging results (endocardial/epicardial error 1.43 ± 0.49 mm/1.51 ± 0.48 mm) on 7980 short axis cardiac MR images acquired from 33 subjects. We have placed our dataset online, for the community to use and build upon.

Introduction

In 2004, cardiovascular disease (CVD) contributed to almost one third of global deaths (American Heart Association, 2004). CVD is the leading cause of death in the developed world and by 2010, CVD is estimated to be the main cause of death in developing countries. According to 2001 estimates, if all forms of CVD in the United States were eliminated, the average life expectancy would increase by around 10 years (Frangi et al., 2001). An elimination of all forms of cancer, on the other hand, would cause the average life expectancy to increase by 3 years.

Three-dimensional imaging of the heart using imaging modalities such as ultrasound, magnetic resonance imaging (MRI) and X-ray computed tomography is a rapidly developing area of research in medical imaging. Screenings that detect problems at an early stage, when treatment is most effective, can help prevent heart disease. The manual segmentation of short axis cardiac MRI provides clinically useful indicators of heart function, such as the ejection fraction (EF) ratio (Frangi et al., 2001). However, manual segmentation is a slow and error prone procedure and fully automated methods are highly desirable.

There are various reasons why the algorithms described in the literature for the segmentation and functional analysis of cardiac images exhibit lower success rates than that of human experts (Bosch et al., 2002, Gering, 2003, Mitchell et al., 2001, Mitchell et al., 2002):

  • 1.

    Existing methods do not incorporate a sufficient amount of a priori knowledge about the segmentation problem.

  • 2.

    Most existing methods do not consider the three-dimensional context as an integral part of their functionality.

  • 3.

    Most existing methods do not consider the temporal context as an integral part of their functionality.

  • 4.

    The algorithm needs to be applicable for a wide variety of image situations. The variability in the images is due to patient health, patient movement and image noise and artifacts.

The following general observations can be made about model-based approaches for the functional analysis of cardiac images. Four-dimensional (4D) cardiac models – which incorporate knowledge about the 3D structure of the heart and its temporal deformation – that rely heavily on training data to build the model, suffer from the curse of dimensionality problem: a 4D cardiac model that is capable of generalizing and reliably segmenting the full spectrum of normal and abnormal cardiac deformations requires a considerable number of training samples. Furthermore, as the dimensionality of the model increases, so does the number of parameters which affect the model’s deformation. The speed, accuracy and reliability of optimization algorithms is negatively affected when dealing with models that use hundreds of parameters.

At the same time, models which incorporate prior knowledge about the 3D cardiac structure and its temporal deformation are necessary if we are to achieve reliable segmentation with minimal user input. It is not uncommon to observe cardiac sequences where imaging induced artifacts make it impossible to segment certain frames of the sequence without prior knowledge of both the temporal deformation and 3D cardiac structure. Models which incorporate such prior knowledge can make reasonable assumptions regarding the segmentation and cause the algorithm’s performance to degrade gracefully in cases where non-model-based segmentation algorithms would fail catastrophically.

Models which rely on object appearance – such as active appearance models (AAMs) – are quite robust when dealing with images that are degraded due to noise and other artifacts. With short axis cardiac MRI, the slices closest to the apex tend to have blurred myocardial borders making edge-based segmentation methodologies inappropriate. Prior knowledge about the 3D cardiac structure and appearance can further improve the reliability of the segmentation.

In contrast, basal slices tend to have sharper myocardial borders, making edge-based approaches for model fitting more appropriate. Furthermore, various image structures are often better delineated using edge-based methods. However, edge-based approaches tend to be quite unreliable when not initialized close to the desired boundary.

As the above brief discussion suggests, there is currently no approach to cardiac segmentation that can deal with the full spectrum of image variability. This makes it worthwhile to investigate the integration of various approaches for dealing with the cardiac segmentation problem.

In this paper, we tackle these issues by combining and extending various approaches based on statistical models of shape and appearance (Cootes and Taylor, 2004). We incorporate knowledge about typical cardiac structure by using a 3D active appearance model (AAM) and a 2D + time active shape model (ASM) of the heart to segment the left ventricle (Frangi et al., 2001). The first model we use is a 3D AAM which encodes the typical deformation of the left ventricle along the x-, y- and z-axis. This model incorporates knowledge about cardiac structure and, thus, provides a robust methodology for performing the segmentation (Mitchell et al., 2001, Mitchell et al., 2002). We use this model to obtain an initial segmentation from each frame of a data set of cardiac MRI.

Once we have this initial segmentation, we use the 2D + time ASM to improve the segmentation. This second model encodes knowledge about the typical temporal deformation of the heart. More specifically, the second model encodes knowledge about the typical deformation along the x, y, and temporal axis of the cardiac cycle. The relative accuracy and robustness provided by the 3D AAM is used to provide an accurate initialization for the 2D + time ASM. This allows us to use edge-based approaches to further refine the segmentation. We apply this model at various heights of our MRI sequence (along the long axis/z-axis of the heart) in order to improve the segmentation accuracy.

The contributions presented in the paper are the following. Firstly, we introduce a method for optimizing the 3D AAM used in the first stage of the model fitting. Brute force non-linear optimization using techniques such as Gauss–Newton optimization is accurate and reliable but extremely slow. Faster methods described in the literature are theoretically unsound and often unreliable. To this end, we developed an efficient, robust and theoretically sound algorithm for optimizing 3D active appearance models, the accuracy of which is on par with Gauss–Newton optimization and which runs around 44 times faster. We use various extensions of the inverse compositional algorithm (Baker and Matthews, 2001, Matthews and Baker, 2004) to accomplish this.

The second contribution involves an approach for handling the problem of training 3D ASMs in the presence of small training sets. We accomplish this through the use of wavelets. More specifically, ASMs are often limited by the inability of a small training set to capture the full range of biological shape variability. Using an approach that was inspired by Davatzikos et al. (2003), we use the statistical properties of the wavelet transform of our shape, to make the shape deformation more generalizable and, thus, get better results in the presence of small training sets. This enables us to have robust ASMs which are not as dependent on the training set size and which can improve the segmentation provided by the 3D AAM.

The dataset of short axis cardiac MR images that we use was provided by the Department of Diagnostic Imaging of the Hospital for Sick Children in Toronto, Canada. The dataset and the ground truth of manual segmentations is provided online, for the medical imaging community to use and build upon.

In the following subsection, we present an overview of previous work on cardiac models that are applicable to the segmentation of cardiac MRI. In Section 2, we present our inverse compositional-based approach for rapid and robust fitting of 3D AAMs and we also present the wavelet based hierarchical 2D + time ASM that we use to further improve our results. In Section 3, we present a detailed description of our experimental setup. In Section 4, we provide results demonstrating the validity of our approach. In Section 5, we present a critical assessment of the method and provide suggestions for future research. In Section 6, we conclude the paper.

We are interested in cardiac models because they incorporate expert knowledge about cardiac structure and its temporal deformation, thus offering a robust method for estimating various indices of cardiac performance. In this section, we provide a brief overview of some of the more distinctive work that has been done on cardiac image analysis using 3D models. This will provide an overview of the main approaches for cardiac modeling.

We limit our exposition to cardiac models built by the medical imaging and computational vision communities which satisfy two criteria. The first criterion is that the models and any shape reconstructed by such models is in 3D or 4D (incorporating temporal properties of the heart). The second criterion is that such models can be applied to cardiac MR images as a means of extracting parameters to interpret cardiac performance. This can include the segmentation of parts of the heart or the motion analysis of the heart. For an introduction to cardiac models built by the cardiology community for simulating cardiac electrophysiology, the reader is referred to Panfilov and Holden (1996). With the exception of the work by Sermesant et al. (2003), and the Yale group (Papademetris et al., 2002, Papademetris, 2000, Yan et al., 2005) which we briefly describe below, most models that have been built by the medical imaging/computer vision community incorporate only limited aspects of the physiology of the heart. We believe that greater interaction between these two communities can bring significant advances to the interpretation of medical images.

The models we are dealing with below are meant to contribute to the extraction of information about cardiac performance from cardiac images. Most of the models we present are meant to facilitate segmentation, but this is not always the case. The model we propose in this paper is meant to improve the segmentation stage. We focus on methods that are applicable to short axis cardiac MRI. The most thorough survey of such methods to date was done by Frangi et al. (2001). The authors classified all models as surface models, volume models or deformation models and we follow this classification in this brief exposition.

Surface models focus on modeling the epicardial and/or endocardial wall. Such models usually deform by positioning the boundaries of the model at the locations of the strongest local features such as edges. In (Pentland et al., 1991), the authors recover a set of global descriptors of motion, which they use to model an approximately elastic object. They use the so called governing equationMU¨+CU˙+KU=Rwhere U is a vector of the model’s surface nodes’ displacements over time and M, C, K are matrices that model the object’s mass, damping and material stiffness. R is a vector describing the forces acting on the nodes. Similar work based on modeling the LV as an elastic object was presented by other researchers (McInerney et al., 1995, Cohen et al., 1991, Cohen and Cohen, 1992). These represent some of the first efforts at 3D modeling of the heart through so called deformable models. Park et al. (1996) used a deformable superquadric to create a volumetric model of the LV and RV of the heart from tagged MRI data. Park and Metaxas introduce parameters for modeling local shape variation. These parameters provide a few intuitive indicators of the heart function, such as the degree of contraction/expansion of the heart. The main advantage of the model is that it is a compact model, as it provides a very small number of parameters that accurately describe the cardiac function. Bardinet et al. (1996) simultaneously segment and track a dynamic sequence of 3D images during a complete cardiac cycle. They accomplish this using a combination of superquadrics-based deformations and free form deformations. There has also been a significant amount of research on ASM-based models for cardiac segmentation (van Assen et al., 2006, Davatzikos et al., 2003, Cootes and Taylor, 1995, Shen and Davatzikos, 2000, Shen and Davatzikos, 2001, de Bruijne et al., 2003, Nain et al., 2006, Tölli et al., 2006). Rousson et al. (2004) have presented a methodology for incorporating more knowledge about cardiac structure through the use of level set methods and ASMs, and could potentially offer a robust approach to cardiac MRI segmentation. In (Delhay et al., 2005), the authors present a 3D surface model which also considers a temporal constraint to increase the accuracy of the segmentation.

In contrast to the vast number of surface models which admittedly constitute a sizeable part of the literature on 3D cardiac models, there have been fewer volume models. Volume models focus on modeling the entire cardiac volume, and not just the surface as surface models do. One such model was introduced in (Mitchell et al., 2002). The authors apply the work of Cootes and Taylor (2004) on statistical point distribution models (PDMs), to model cardiac shape variation. The main advantage of this work is the ease with which shape deformations can be learned from a given training set, and the model’s ability to also incorporate the appearance variation of the model. Such models have been extremely successful in the medical imaging field, leading to the creation of many commercially successful products (Cootes and Taylor, 2004). Numerous similar approaches have appeared since then. For example (Kaus et al., 2004), demonstrate the usefulness of integrating prior sources of knowledge into a model for robust segmentation. Sermesant et al. (2003) present a model which simulates the electromechanical properties of the heart in a computationally efficient way. By incorporating a priori knowledge about cardiac properties, the fitting/segmentation of cardiac images should become more robust. In (Lorenzo-Valdés et al., 2004), the authors use the EM algorithm with a probabilistic 4D atlas of the heart for automatic atlas-based segmentation. The EM algorithm is used to estimate initial model parameters and integrate a priori information in the model. Promising results are presented.

Deformation models rely on explicit correspondences between tissue landmarks across time frames. They model the tissue deformation over time, incorporating expert knowledge of the temporal deformation of the heart. Hamarneh and Gustavsson (2004), present a 3D ASM of the heart, where the third dimension corresponds to time, and models the temporal deformation of an object across time. The authors present some examples of the method’s applicability for cardiac segmentation from ultrasound images. Amini and Duncan (1992) introduce a model in which they estimate the cardiac wall motion with the help of a metric measuring the curvature properties of the surface. Benayoun et al. (1998) use an adaptive mesh to estimate the non-rigid motion of sequences of 3D images. Meshes at two time frames are adjusted in such a way that image differential properties, such as curvature and gradient, are matched. Another recently introduced model is described in Papademetris et al., 2002, Papademetris, 2000, in which the authors propose a deformation model based on principles from continuum mechanics. They employ these properties to calculate the strain throughout the heart wall, which in turn, is used in combination with surface curvature properties to calculate the displacement of the cardiac wall during the cardiac cycle. Work on these ideas continued and was further expanded by researchers at Yale. See for example Yan et al. (2005). We continue with presenting a methodology for fitting 3D AAMs, providing an initial segmentation of our MRI data.

Section snippets

Optimization of 3D AAMs for short axis cardiac MRI segmentation

Active appearance models (AAMs) provide a robust approach for the analysis of medical images (Cootes and Taylor, 1998, Cootes and Taylor, 2004). The ability of AAMs to learn the 3D structure of the heart and not lead to unlikely segmentations has stirred up interest in the medical imaging community regarding their use for the segmentation of the left ventricle from short axis cardiac MRI (Frangi et al., 2001, Mitchell et al., 2002).

The algorithms described in the literature for fitting AAMs,

Test data

The dataset on which we trained and fitted the models was comprised of short axis cardiac MR image sequences acquired from 33 subjects, for a total of 7980 2D images which were provided by the Department of Diagnostic Imaging of the Hospital for Sick Children in Toronto, Canada. The images were scanned with a GE Genesis Signa MR scanner using the FIESTA scan protocol. Most of the subjects displayed a variety of heart abnormalities such as cardiomyopathy, aortic regurgitation, enlarged

Results

Table 1 presents our results using the four different optimizations approaches for fitting a 3D AAM (IC9, IC3, Gauss–Newton and the Classical approach). The mean time it took to segment each 3D volume is presented in seconds in the table’s first column. The error (means ± standard deviation), maximum error, minimum error and tests of statistical significance of the differences in the errors achieved by different optimization algorithms are presented. Similarly, Table 2 presents the improvements

Discussion

From inspecting Table 1, we reach the following general conclusions. Overall, based on the ENDO + EPI metrics, IC9 performs the best. It is followed by IC3, then by Gauss–Newton optimization and finally, by the Classical approach. However, in terms of the endocardial (ENDO) volumetric, landmark and volume errors, Gauss–Newton optimization performs the best. However, the difference is not statistically significant compared to IC9 for the ENDO volume error (p = 0.2649), and the difference is

Conclusions

We have proposed a reliable model for the analysis of short axis cardiac MRI. We first presented an algorithm for fitting 3D active appearance models on short axis cardiac MR images, and observed an almost 44-fold improvement in the segmentation speed and a segmentation accuracy that is on par (and often better) with Gauss–Newton optimization, the most widely used algorithm for such optimization problems. We then showed how a 2D + time hierarchical shape model could significantly improve our

Acknowledgements

We thank Dr. Paul Babyn and Dr. Shi-Joon Yoo of the Department of Diagnostic Imaging at the Hospital for Sick Children in Toronto for providing us the MRI data. J.K.T. holds the Canada Research Chair in Computational Vision and gratefully acknowledges its financial support. A.A. would also like to thank the Natural Sciences and Engineering Research Council of Canada (NSERC) for its financial support through the PGS-M/PGS-D scholarship program. We also thank the reviewers for their insightful

References (46)

  • H.C. van Assen et al.

    SPASM: a 3D-ASM for segmentation of sparse and arbitrarily oriented cardiac MRI data

    Med. Image Anal.

    (2006)
  • B. Zitova et al.

    Image registration methods: a survey

    Image Vis. Comput.

    (2003)
  • American Heart Association, 2004. International Cardiovascular Disease Statistics. <http://www.americanheart.org>...
  • Baker, S., Matthews, I., 2001. Equivalence and efficiency of image alignment algorithms. In: Proceeding of the IEEE...
  • S. Baker et al.

    Lucas-Kanade 20 years on: a unifying framework

    Int. J. Comput. Vis.

    (2004)
  • E. Bardinet et al.

    Tracking and motion analysis of the left ventricle with deformable superquadrics

    Med. Image Anal.

    (1996)
  • S. Benayoun et al.

    Dense and nonrigid motion estimation in sequences of medical images using differential constraints

    Int. J. Comput. Vis.

    (1998)
  • J.G. Bosch et al.

    Automatic segmentation of echocardiographic sequences by active appearance motion models

    IEEE Trans. Med. Imaging

    (2002)
  • Cohen, L.D., Cohen, I., 1992. Deformable models for 3D medical images using finite elements and balloons. In:...
  • Cohen, I., Cohen, L., Ayache, N., 1991. Introducing new deformable surfaces to segment 3D images. In: Proceedings of...
  • Cootes, T.F., Taylor, C., 1998. Active appearance models. In: Proceedings of the European Conference on Computer...
  • Cootes, T.F., Taylor, C.J., 2004. Statistical models of appearance for computer vision. Tech. rep., Imaging Science and...
  • C. Davatzikos et al.

    Hierarchical active shape models, using the wavelet transform

    IEEE Trans. Med. Imaging

    (2003)
  • Cited by (0)

    View full text