A survey of MRI-based medical image analysis for brain tumor studies

With a prevalence of less than 1‰ in the western population, brain tumors are not very common; however, they are among the most fatal cancers (DeAngelis 2001). A recent study estimated the US incidence rate for primary tumors of the brain or nervous system to be around 25 per 100 000 adults with approximately one-third of the tumors being malignant and the rest either benign or borderline-malignant (Kohler et al 2011). The word 'tumor' is of latin origin and means swelling. Today, it is frequently associated with a neoplasm, which is caused by uncontrolled cell proliferation.

Brain tumors can be classified according to their origin or degree of aggressiveness. Primary brain tumors arise in the brain, while metastatic brain tumors frequently originate from other parts of the body. The most widely used grading scheme today has been introduced by the World Health Organization (WHO) (Kleihues et al 1993). It classifies brain tumors into grades I–IV with increasing aggressiveness.

Gliomas are the most frequent primary brain tumors in adults and account for 70% of adult malignant primary brain tumors. They arise from glial cells and can be classified into four WHO grades, where high-grade gliomas are of grade III or IV. Grades I and II tumors may be considered as semi-malignant tumors that carry a better prognosis, while grades III and IV tumors are malignant tumors that almost certainly lead to a subject's death. The WHO grade IV gliomas are also called glioblastoma multiforme (GBM). They are the most common malignant primary brain tumors in humans, exhibiting very rapid growth (Deimling 2009). These tumors are infiltrative and spread mostly along the white matter fiber tracts (Giese et al 2003). They form abnormal vessels and exhibit a necrotic core. This and the surrounding edema leads to a mass effect, which they exert on the healthy tissues of the brain. Regarding tumor angiogenesis, current understanding is based on the fact that low-grade gliomas are moderately vascularized tumors, whereas high-grade gliomas show prominent microvascular proliferations and areas of high vascular density (Plate and Risau 1995). Average survival time for GBM is one year (Krex et al 2007). Treatment options for gliomas include surgery, radiation therapy or chemotherapy, including combinations of all these (DeAngelis 2001).

Meningiomas are the most common extra-axial intracranial neoplasms. They account for 15–20% of intracranial neoplasms. They are non-glial neoplasms that originate from the arachnoid cap cells of the meninges. Treatment is usually performed by surgery and/or radiation therapy (Greenberg et al 1999).

Primary brain tumors must be differentiated from metastatic brain tumors that originate most frequently from lung, breast, melanoma, renal and colon cancers and account for approximately 40% of intracranial neoplasms.

Due to clinical relevance and the amount of prior work in MRI-based medical image processing, the focus of this review will be on gliomas.

The standard technique for the brain tumor diagnosis is MRI (DeAngelis 2001, Wen et al 2010). MRI is a non-invasive technique, which provides good soft tissue contrast (Liang and Lauterbur 2000) and is widely available in clinics. It is used in combination with other imaging modalities, such as computed tomography (CT), positron emission tomography (PET) and magnetic resonance spectroscopy (MRS) to provide the most exact information about tumor morphology and metabolism. In particular, PET imaging can provide additional information (Chen 2007); however, so far MRI remains the accepted standard and therefore we will focus on MRI-based methods.

MRI makes it possible to produce markedly different types of tissue contrast by varying excitation and repetition times, which makes it a very versatile tool for imaging different structures of interest. Due to the nature and appearance of brain tumors, one MRI sequence is not sufficient to fully segment the tumor including all its subregions. In current clinical routine, different MRI sequences are employed for the diagnosis and delineation of tumor compartments (Drevelegas and Papanikolaou 2011). These sequences include T₁-weighted MRI (T₁), T₁-weighted MRI with contrast enhancement (T_1c), T₂-weighted MRI (T₂) and T₂-weighted MRI with fluid-attenuated inversion recovery (T_2FLAIR); however, acquisition parameters of these modalities are not standardized. Patients with gliomas are usually examined by the previously described MR imaging protocols according to the response assessment in neuro-oncology (RANO) guidelines with a slice thickness of ⩽5 mm without a gap between the slices (Pope and Hessel 2011). For volumetry, high-resolution 3D volume images are performed, including at least contrast-enhanced T₁-weighted images with isotropic resolution. Figure 1 shows an axial slice of the four standard sequences for a glioma patient including manually drawn tumor regions.

Zoom In Zoom Out Reset image size

T₁-weighting is the most commonly used sequence for the structural analysis; it also allows for an easy annotation of the healthy tissues. In T₁-weighted contrast-enhanced images (gadolinium-DTPA), the tumor borders appear brighter because the contrast agent accumulates there due to the disruption of the blood–brain barrier in the proliferative tumor region. In this sequence, the necrotic and the active tumor region can be distinguished easily. In T₂-weighted MRI, the edema region, which surrounds the tumor, appears bright. T_2FLAIR (FLAIR) is a special sequence, which helps in separating the edema region from the cerebrospinal fluid (CSF) because the free water signal is suppressed. The radiological definition of the tumor margins in the clinical context are often manually determined by the radiologist on the T₂ and post-gadolinium T₁ images by thresholding boundaries between T₂ hyperintense/T₁ contrast-enhanced lesions and the surrounding healthy tissue to define the outer margins of a tumor. Clinical measurements of the tumor size traditionally incorporate either the product of the major and minor axis (2D measures) or of the three main axes of a tumor (3D measures).

Despite all the advantages that non-invasive imaging offers, it should be noted that a final diagnosis can only be made after biopsy and histology. It must also be emphasized that in gliomas, in contrast to metastatic brain tumors, the imageable component of the tumor is only part of the complete extent of the tumor. The imageable component and physiological information of the tumor depend on the tumor cell distribution and the image modality (Kelly et al 1987, Tovi 1993).

This review is focused on the image analysis for brain tumor studies. The image analysis deals with automatic or semi-automatic methods to help interpret the acquired images. Due to the large amount of data, which are currently being generated in the clinics, it is not possible to manually annotate and segment the data in a reasonable time. Usually, the domain of medical image analysis is divided into segmentation and registration, as well as into several further areas such as enhancement, visualization, quantification and modeling (Bankman 2008). For brain tumor studies, segmentation, registration and modeling appear to be the most important and the most challenging, so the remainder of this review focuses on these aspects.

Image segmentation (Pham et al 2000) aims at partitioning an image into several segments. These segments can be chosen according to structures of interest, tissue types, functional areas, etc. Balafar et al (2010) presented a recent review targeted at brain segmentation specifically. Image registration (Maintz and Viergever 1998) aims at aligning two different images in a common reference space. This is especially important for aligning images taken at different points in time during longitudinal studies or for aligning images from different modalities taken from the same patient, but also for brain morphometry. Klein et al (2009) compared nonlinear algorithms for the brain registration specifically. Finally, image-based modeling and simulation (Neal and Kerckhoffs 2010) can help in disease understanding, treatment planning and decision making.

The primary use of MRI-based medical image analysis for brain tumor studies is in diagnosis, patient monitoring and treatment planning, but it could also be useful in clinical trials. The segmentation is crucial for monitoring tumor growth or shrinkage in patients during therapy, for tumor volume measurements and it also plays an important role in surgical planning or radiotherapy planning, where not only the tumor has to be outlined, but also surrounding healthy structures are of interest. In current clinical practice, the segmentation is usually still done manually, which is time consuming and tedious for the radiologists and is also of limited use for an objective quantitative analysis. Concrete application areas for the registration include multi-modal and longitudinal alignment of intrapatient images, but also aligning pre-, intra- and post-operative images as well as deformable registration for atlas-based segmentation or position mapping of gliomas for statistical analysis.

Although high-grade gliomas are not the most common cancers, they are among the most deadly. Novel therapies have improved patients' prognosis and novel clinical trial designs have gained in importance to further investigate the response of gliomas to different treatment regimens. A most accurate description of changes in tumor size and physiology is therefore mandatory. For many years now, treatment response of gliomas has been evaluated by MR and CT imaging; a set of guidelines, usually referred to as the Macdonald criteria (Macdonald et al 1990), is available to monitor the assessment of tumor response after treatment. These criteria include changes in tumor size and new or increasing enhancement. However, this approach carries some important limitations (Clarke and Chang 2009). Although the Macdonald criteria represented an important step in neuro-oncology research, it is now clear that the evaluation of gadolinium enhancement alone is not adequate to characterize tumor growth or response. The RANO working group has defined novel criteria for tumor progression dependent on measurable or non-measurable lesions in at least two directions (Wen et al 2010). Automatic 3D volumetric assessment of gliomas would offer the next important step forward to better understand tumor dynamics and response to treatment.

A previous review of the field was done by Angelini et al (2007). In the meantime however, there has been significant progress, with the most important methods having been developed after 2007, mostly due to the rapid advancements in machine learning (Wang and Summers 2012). Therefore, this review aims at providing an update on the state of the art in image analysis for brain tumor studies.

The methods presented here have been collected mostly from Pubmed and Google scholar. We included more than 150 journal papers and conference papers from the most important conferences in the field (MedIA, IEEE TMI, NeuroImage, MRI, MRM, JMRI, PMB; MICCAI, IPMI, IEEE ISBI and others). For the time before 2007, we included only the historically most relevant contributions; for the rest, we refer to Angelini et al (2007).

The remainder of this review is structured as follows: in the Methods and Results section (section 2), we present the different approaches and briefly summarize their results. The section is subdivided into methods for the evaluation of accuracy and validation (section 2.1), segmentation (section 2.2), registration (section 2.3) and integrated approaches (section 2.4). Within each subsection, we tried to group related methods together. Finally, in the Discussion and Outlook section (section 3), we review the state of the art in general, compare it to the clinical requirements and conclude about the applicability of the current methods, before we give an outlook on possible future directions and trends.

Validation and comparison against the state of the art is crucial for any newly developed method. Therefore, before diving into the presentation of the different approaches, we would like to briefly cover the possibilities and challenges for evaluating and validating methods in the brain tumor image analysis.

It would be optimal to compare any method against the real case. However, this is a big challenge in this field, if not impossible. In the lack of a well-accepted ground truth, the current gold standard for the evaluation is to compare with manual segmentations by an expert. However, this is an extremely time-consuming and tedious task; additionally, it is not objective. Mazzara et al (2004) reported intra-rater volume variabilities of 20±15% and inter-rater volume variabilities of 28±12% for manual segmentations of brain tumor images. Weltens et al (2001) found even larger values for inter-observer variability. One way to overcome this problem would be to use an algorithm that combines several expert segmentations in an optimal way like STAPLE (Warfield et al 2004). This method was applied for the evaluation of brain tumor segmentations by Archip et al (2007a); however in most cases, there are not enough expert segmentations available for using that method. Recently, Xu et al (2012) suggested to use web-based collaborative manual tumor labeling by a large number of briefly trained non-experts to address this problem and they reported encouraging results. A more subjective but occasionally used way is to employ a semi-automatic segmentation with a different well-accepted method as an intermediate result (e.g. Kikinis and Pieper 2011, Gao et al 2012, Egger et al 2013), which is manually corrected by a human expert where necessary. This approach still lacks in objectivity, but it alleviates the burden from the clinician to spend a large amount of time for the segmentation.

Another possibility for a first sanity check is to assess results on a synthetic dataset including ground truth. Although synthetic data lack important characteristics of real images, it has been used by many groups for initially assessing both segmentation and registration methods on healthy datasets with the BrainWeb phantoms (Cocosco et al 1997). For brain tumor studies, Prastawa et al (2009) made a similar attempt to provide simulated multi-sequence tumor image data including an objective ground truth, which was also based on the BrainWeb phantom and combined with a well-defined and accepted tumor-growth model.

The most common way to quantitatively evaluate segmentation or registration results is to calculate the overlap with the ground truth. Usually, the Dice similarity coefficient (DSC) or the Jaccard coefficient are used (Crum et al 2006). They can range from 0 to 1 with 0 indicating no overlap and 1 indicating perfect overlap. Other possibilities for the registration evaluation include manual landmark definition and calculation of landmark errors or surface distances. For a more qualitative assessment, checkerboard images can be shown or the outline of a structure can be analyzed visually. Zou et al (2004) compared the three different validation metrics: area under the receiver operating characteristic (ROC) curve, mutual information (MI) and DSC for the probabilistic brain tumor segmentation. They concluded that for overall classification accuracy, the area under the ROC curve should be used; when interested in sensitivity to changes in tumor size, MI is the metric of choice and for the spatial alignment evaluation, the Dice coefficient is best.

In the lack of a brain tumor database with ground-truth segmentations, that is available to a broad community of clinicians and researchers, so far most authors validated their algorithms on a limited number of cases from their own data. This makes it difficult to compare the performance of different methods against each other in an unbiased way. Therefore, and due to the different metrics used, the accuracy and speed of the individual methods, which have been collected from the respective publications, cannot be directly compared with each other. Until recently, the only data available, which could serve such a purpose of general comparison, were the synthetic data of Prastawa et al (2009), but so far only few groups tested their methods on these images. For the related topic of MS lesion segmentation from brain MRI (García-Lorenzo et al 2013), an open database has been available for some time (Styner et al 2008). Such an open image database has long been missing for an objective comparison of brain tumor segmentation algorithms; however, an effort in this direction has finally been undertaken by the BraTS challenge at MICCAI 2012⁴.

The segmentation of tumor-bearing brain images is a challenging task for several reasons. Firstly, high-grade gliomas usually exhibit unclear and irregular boundaries with discontinuities. Contrast uptake and image acquisition time after contrast injection can vary, which changes tumor appearance significantly and it is debatable if and how the non-imageable component of the tumor should be handled by segmentation algorithms. Additionally, tumor subregions can only be separated when several modalities are combined, which requires an accurate registration as a pre-processing step. Finally, clinical datasets are usually acquired in a highly anisotropic way, leading to a much higher intra-slice resolution than inter-slice resolution. This causes problems with partial-volume effects for the segmentation, but also registration and resampling of the different modalities in a common space of reference. A diagram illustrating the major steps during the segmentation pipeline of most algorithms is shown in figure 2. In the following, the different algorithms are discussed according to the major blocks of this pipeline.

Zoom In Zoom Out Reset image size

2.2.1. Image pre-processing

2.2.2. Image features for segmentation algorithms

2.2.3. Segmentation algorithms

Based on the previous section, we categorize segmentation algorithms according to the features they use. Therefore, we distinguish region- or edge-based methods, which mostly rely on deformable models, and classification or clustering methods, which make use of voxel-wise intensity and texture features. Many methods employ additional constraints for the regularization; this is discussed in a separate section. Another group of methods is based on the atlas-based segmentation. Atlas-based methods will be described later in sections 2.3 and 2.4 because the segmentation relies on registration methods.

Region- or edge-based methods.

Deformable models make use of regional characteristics or edge detection in the images (McInerney and Terzopoulos 1996). In most cases, a level- set is evolved toward the tumor boundary by searching for the largest gradient in the image or by employing region properties. Wang et al (2009) employed a fluid vector flow model to evolve a contour toward the tumor boundary edge in T₁-weighted images. Sachdeva et al (2012) made use of content-based intensity and texture patterns to evolve an active contour toward the tumor boundary in different MRI modalities.

Ho et al (2002) derived a tumor probability map from the difference image between T₁ and T_1c, which formed the basis for the evolution of a region-competition level-set algorithm applied. Rexilius et al (2007) initialized a region-growing algorithm with a tumor map, which was obtained from a multi-spectral histogram model adaptation. Harati et al (2011) suggested a fuzzy-connectedness algorithm, which made use of region properties but had no spatial constraints. Seeding was automatically performed on T₁-weighted images with a so-called tumor detector matrix. Characteristics of each method are summarized in table 1.

Table 1. Segmentation methods for brain tumor images using deformable methods or region-based methods. Evaluation was performed on different datasets; empty cells indicate no reported information. Dim stands for dimensionality, S for supervision. SA means semi-automatic, FA means fully automatic. Type describes the tumor type, which can be as follows: G—glioma, HGG—high-grade glioma, LGG—low-grade glioma, M—meningioma, Met—metastasis.

Authors	Modalities	Method	Accuracy	Speed	Dim	S	Type
Edge-based methods
Wang et al	T1	Fluid vector flow	0.6 (Tanimoto)	5 s	2D	SA
(2009)
Sachdeva et al	T1, T1c, T2	Content-based	0.72–0.98		2.5D	SA	G, M
(2012)		active contour model
Region-based methods
Ho et al	T1, T1c	Level-set evolution	0.85–0.93 (Jaccard)		3D	FA	G, M
(2002)		with region
		competition
Rexilius et al	T1c, T2, T2flair	Multi-spectral	0.73 (Jaccard)	10 min	3D	SA	G
(2007)		histogram model
		adaptation for
		region growing
Harati et al	T1c	Fuzzy connectedness	>0.9 (similarity index)	2.5 min		FA	G, M
(2011)

Classification and clustering.

In fact, most of the segmentation algorithms proposed so far are based on classification or clustering approaches. This is mostly owed to the fact that with these methods, multi-modal datasets can be handled easily because they can operate on any chosen feature vector. In most cases, the features on which these algorithms operate include voxel-wise intensities and frequently also local textures. The general idea is to decide for every single voxel individually to which class it belongs based on its feature vector. Classification requires training data to learn a classification model, based on which new instances can be labeled. Clustering, on the other hand, works in an unsupervised way and groups data based on certain similarity criteria (Wang and Summers 2012).

Clustering was introduced into the brain tumor segmentation community by Schad et al (1993) who analyzed texture patterns of different tissues. Phillips et al (1995) employed fuzzy c-means clustering and Vaidyanathan (1995) compared this to kNN clustering for the tumor volume determination during therapy on multi-spectral 2D image slices. Clark et al (1998), from the same group, further developed this approach to incorporate knowledge-based techniques. Later, Fletcher-Heath et al (2001) also combined fuzzy clustering with knowledge-based techniques for the brain tumor segmentation from multi-sequence MRI.

Later, there have been tremendous advances in developing more powerful discriminative classification methods and these new methods have also found their way into the field of medical image analysis. Cai et al (2007) and later Verma et al (2008) from the same group used a high number of MRI modalities (diffusion tensor imaging (DTI) channels in addition to the conventional ones) to create voxel-wise intensity-based feature vectors, which they classified with support vector machines (SVMs) (Schoelkopf and Smola 2002). They were able to not only segment the healthy tissues, but also segment sub-compartments of healthy and tumor regions. Ruan et al (2007) used a very similar approach based on SVMs, but with a lower number of modalities and they only segmented one tumor region. Later, they claimed that the feature selection using kernel class separability could slightly improve the results compared to their previous approach (Ruan et al 2011). Jensen and Schmainda (2009) explored different neural networks to detect brain tumor invasion from multi-parametric MRI (structural, diffusion and perfusion images). Kanaly et al (2011) chose a simpler approach by thresholding the voxels of the difference image of pre- and post-contrast T₁-weighted MRI after intensity normalization. Zikic et al (2012) later proposed a computationally efficient approach based on the decision forest classification (Criminisi et al 2011) with context-aware features and an additional generative model as an input, which is able to identify tumor sub-compartments from multi-modal images (see figure 3). They claimed that context-aware features eliminate the need for a post-processing step imposing smoothness constraints by spatial regularization. Geremia et al (2012) had the idea to generate synthetic tumor images, which can be used to train a discriminative regression forest algorithm using different groups of features. They argued that this approach allowed them not only to segment patient images, but also to estimate latent tumor cell densities. Characteristics of each method are summarized in table 2.

Zoom In Zoom Out Reset image size

Table 2. Segmentation methods for brain tumor images using classification or clustering, grouped by different approaches. Evaluation was performed on different datasets; empty cells indicate no reported information. Dim stands for dimensionality, S for supervision. SA means semi-automatic, FA means fully automatic. Type describes the tumor type, which can be as follows: G—glioma, HGG—high-grade glioma, LGG—low-grade glioma, M—meningioma, Met—metastasis.

Authors	Modalities	Method	Accuracy	Speed	Dim	S	Type
Classification or clustering methods
Schad et al (1993)	T1, T2	Texture analysis and clustering			2D		HGG, Met
Phillips et al (1995)	T1, T2, PD	Clustering					HGG
Vaidyanathan (1995)	T1, T2, PD	Clustering		2–5 min	2D		HGG, M
Clark et al (1998)	T1, T2, PD	Knowledge-based techniques	0.69–0.99 (% match)		2D	FA	HGG
Fletcher-Heath	T1, T2, PD	Fuzzy clustering + knowledge-	0.53–0.91 (% match)		2.5D	FA
et al (2001)		based techniques
Cai et al (2007)	T1, T1c, T2, T2flair, DTI	Probabilistic segmentation	0.73–0.98 (classification rate)		3D	FA	HGG
		based on multi-modality MRI
Ruan et al (2007)	T1, T2, T2flair, PD	SVM classification	0.99 (TP)		3D	FA
Verma et al (2008)	T1, T1c, T2, T2flair, DTI	Multi-parametric	0.34–0.93 (classification rate)		3D	FA	HGG
		tissue classification
Jensen and	T1, T1c, T2, T2flair,	Different classifiers on	0.78–0.86 (overlap)		3D	SA	HGG, M
Schmainda (2009)	DWI, DTI, DSC	multi-parametric MRI
Ruan et al (2011)	T2, T2flair, PD	Image fusion for the		5 min	3D	FA
		brain tumor follow-up
Kanaly et al (2011)	T1, T1c	Difference image for			3D	SA
		volumetric tumor assessment
Zikic et al (2012)	T1, T1c, T2, T2flair, DTI	Decision forests for tissue-	0.7–0.9 (Dice)	2–3 min	3D	FA	HGG
		specific segmentation
Geremia et al (2012)	T1, T1c, T2, T2flair	Tumor cell density estimation	0.55–0.83 (Dice)		3D	FA	G
		for the discriminative segmentation

Classification and clustering with additional constraints.

It has been remarked that simple voxel-wise classification or clustering methods do not make use of the complete information contained in an image. Therefore, there have been numerous attempts to use additional information to further improve the segmentation result by adding additional constraints, which can be based on some form of neighborhood regularization or on imposing shape or localization constraints for the tumor. Neighborhood constraints are often imposed using a random field regularization method, while shape constraints are mostly handled by deformable models. Atlases can be used to restrict the tumor location and also for generative classification models.

Atlases have been used by a number of methods to incorporate spatial prior knowledge into the classification task. Kaus et al (2001) suggested to use a brain atlas for guiding a kNN classifier in their adaptive template-moderated classification algorithm. Other researchers used atlases not only to impose spatial constraints, but also to provide probabilistic information about the tissue model. Moon et al (2002) and also Prastawa et al (2003) employed a probabilistic tissue model and used an expectation–maximization (EM) method (Dempster 1977), which segmented tumor images by modifying an atlas with patient-specific information about tumor location from different MRI modalities. Later, Prastawa et al (2004) extended this to work on mono-modal images. Menze et al (2010) combined a healthy atlas with a latent tumor atlas to segment brain tumors from multi-sequence images using a generative probabilistic model and spatial regularization. Weizman et al (2012) used localization based on an atlas for their multi-modal segmentation of optic pathway gliomas, which performed classification with a probabilistic tissue model.

Deformable models (McInerney and Terzopoulos 1996) can be used to impose shape constraints after the previous tissue classification. Cobzas et al (2007) and Cobzas and Schmidt (2009) combined the tissue classification using a high-dimensional feature set with the level-set evolution. Popuri et al (2012) added a Dirichlet prior to the previous approach to better disambiguate tumor from healthy tissues. Khotanlou et al (2009) combined fuzzy clustering and brain symmetry features with the level-set evolution. Hamamci et al (2011) introduced the tumor-cut algorithm, which combines the tumor segmentation using cellular automata with a level set evolving on the tumor probability map to impose spatial smoothness.

Neighborhood relationships are often used for spatial regularization after the initial voxel-wise classification. In most case, this is performed with a random field method, either Markov random fields (MRFs) (Kindermann and Snell 1980) or Conditional Random Fields (CRFs) (Lafferty et al 2001, Kumar and Hebert 2006). Over the last few years, algorithms based on some form of graph cuts for the segmentation (Boykov and Funka-Lea 2006) have become increasingly popular. These algorithms make use of region-based properties by formulating neighborhood relationships as an energy minimization problem, which is solved by a graph-cut optimization method and the result is interpreted as a segmentation. In many cases, these methods require user-provided seeds; this is why the approaches are often semi-automatic. Lee et al (2008) used CRFs for spatial regularization after the previous voxel-wise SVM classification from multiple modalities. Wels et al (2008) integrated a probabilistic boosting tree classifier with a graph-cut approach for considering the spatial relationships in pediatric brain tumor images. Birkbeck et al (2009) proposed an interactive semi-automatic approach, which was able to include online user corrections and they argued that this method could not only reduce operator time, but also increased repeatability compared to manual segmentation. Hamamci et al (2010) proposed a variation of the original graph-cut method using cellular automata for solving the shortest-path problem iteratively. Nie et al (2009) used an EM algorithm, which was coupled with a spatial accuracy-weighted hidden MRF. This allowed them to consider anisotropic resolutions of different modalities. Zhu et al (2012) also combined the EM segmentation with the MRF regularization, but added a post-processing pipeline including thresholding and morphological operations. Bauer et al (2011a) claimed to be able to segment tumor and healthy tissues including sub-compartments based on the SVM classification with the integrated hierarchical CRF regularization. They also made use of prior knowledge about tissue adjacency probabilities. Hsieh et al (2011) chose a simple region-growing and knowledge-based post-processing after the fuzzy tissue classification to impose spatial coherence. Corso et al (2007) took a different approach and suggested an extended graph-shift algorithm, which performed energy minimization on a dynamic hierarchical representation of the image. In a later publication, they employed a multi-level approach for fusing the Bayesian tissue classification with affinity assignments to perform the segmentation by weighted aggregation (Corso et al 2008). Characteristics of each method are summarized in table 3.

Table 3. Segmentation methods for brain tumor images using classification or clustering with additional constraints, grouped by different approaches. Evaluation was performed on different datasets; empty cells indicate no reported information. Dim stands for dimensionality, S for supervision. SA means semi-automatic, FA means fully automatic. Type describes the tumor type, which can be as follows: G—glioma, HGG—high-grade glioma, LGG—low-grade glioma, M—meningioma, Met—metastasis.

Authors	Modalities	Method	Accuracy	Speed	Dim	S	Type
Classification or clustering with global constraints based on atlases
Kaus et al (2001)	T1	Adaptive template-moderated classification	>95% (accuracy)	5–10 min	3D	FA	LGG, M
Moon et al (2002)	T1, T1c, T2	Statistical classification with geometric prior	>0.9 (overlap ratio)		3D	FA	G, M
Prastawa et al (2003)	T1, T1c, T2	Subject-specific modification of atlas priors	0.49–0.71 (Jaccard)		3D	FA
Prastawa et al (2004)	T2	Outlier detection	0.59–0.89 (Jaccard)	1.5 h	3D	FA	G, M
Menze et al (2010)	T1, T1c, T2, T2flair	Generative model for multi-	0.4–0.7 (Dice)		3D	FA	G
		modal segmentation
Weizman et al (2012)	T1, T2, T2flair	Classification and follow-up	0.69 (Dice)	20 min	3D	FA	G
		for optic pathway gliomas
Classification or clustering with constraints based on deformable models
Cobzas et al (2007)	T1, T1c, T2	Variational segmentation with	0.16–0.76 (Jaccard)		3D	FA	G
		high-dimensional feature set
Khotanlou et al (2009)	T1	Fuzzy classification and spatially	0.92 (similarity index)	4.5 min	3D	FA	HGG, M
		constrained deformable models
Cobzas and Schmidt (2009)	T1c, T2flair	Level-set with embedded CRF	0.5–0.75 (Jaccard)		3D	FA	G
Popuri et al (2012)	T1, T1c, T2	Variational segmentation on	0.58 (Jaccard)		3D	FA	G
		a clustered feature set
Hamamci et al (2011)	T1c	Tumor-cut	0.8–0.89 (Dice)	1 s–16 min	3D	SA	G, M, Met
Classification or clustering with local constraints based on neighborhood regularization
Corso et al (2007)	T1, T1c, T2, T2flair	Extended graph-shift algorithm	0.87 (Jaccard)	1 min	3D	FA	HGG
Corso et al (2008)	T1, T1c, T2, T2flair	Segmentation by weighted aggregation	0.62–0.69 (Jaccard)	<1 min	3D	FA	HGG
Lee et al (2008)	T1, T1c, T2	Pseudo CRFs	0.84–0.9 (Jaccard)		2D	FA	G
Wels et al (2008)	T1, T1c, T2	Discriminative model-constrained graph cuts	0.78 (Jaccard)	1–2 min	3D	FA	G
Birkbeck et al (2009)	T1c, T2flair	Interactive graph cuts		2 min	2.5D	SA
Nie et al (2009)	T1, T2, T2flair	Spatial accuracy-weighted hierarchical MRF	0.72–0.76 (Jaccard)	20–25 min	3D	FA	G
Hamamci et al (2010)	T1contrast	Cellular automata segmentation	0.74–0.87 (Dice)		3D	SA	G, M, Met
Hsieh et al (2011)	T1, T2	Fuzzy clustering plus region-growing	0.73 (% match)		3D	FA	M
Bauer et al (2011a)	T1, T1c, T2, T2flair	Hierarchical SVM+CRF	0.77–0.84 (Dice)	<2 min	3D	FA	G
Zhu et al (2012)	T1c, T2	EM + MRF + post-processing	0.25–0.81 (Jaccard)	4 min	3D	SA	HGG

2.2.4. Tumor detection algorithms

2.2.5. Summary and conclusion—segmentation

Image registration of tumor-bearing brain images mainly focuses on two different objectives: intrapatient multi-modal or longitudinal registration (alignment) of images on the one hand and interpatient spatial normalization of brain tumor images to a brain atlas on the other hand. The latter is commonly used for atlas-based segmentation purposes (Cabezas et al 2011) or for the statistical analysis and functional/structural mapping of brain tumor images. It can also be applied for constructing statistical tumor atlases from multiple patients (Davatzikos et al 2011). A diagram illustrating the major blocks of most registration pipelines is shown in figure 4. The pre-processing steps are very similar to those discussed in section 2.2.1 and are therefore not repeated here. We decided to separate registration methods and methods integrating tumor modeling with the registration. Again, both can be sub-grouped into intra- or interpatient methods, which can use either standard or tumor-specific registration algorithms. Usually, an initial rigid alignment is followed by a non-rigid transformation step (Sotiras et al 2012). The feature metric, which is used by most of the discussed approaches, is intensity-based registration thanks to its general applicability.

Zoom In Zoom Out Reset image size

2.3.1. Standard intrapatient registration

2.3.2. Advanced longitudinal intrapatient registration to handle resections

2.3.3. Advanced longitudinal intrapatient registration for tumor-growth monitoring

2.3.4. Standard interpatient registration

2.3.5. Interpatient registration with lesion masking approaches

2.3.6. Interpatient registration with two distinct deformation models

Another idea for the interpatient registration is to incorporate a lesion model directly into the registration method. This allows for a decoupling of the deformations due to tumor-growth and inter-subject variations, without having to model the complex tumor-growth behavior explicitly. Bach Cuadra et al (2004) suggested a model of lesion growth for the atlas-based segmentation of tumor-bearing brain images. They incorporated a simplistic radial tumor-growth model into a Demons-based non-rigid registration method. The lesion-growth model modified a healthy atlas and adapted it to the tumor-bearing patient image; however, the method was completely based on registration methods and did not include any kind of bio-mechanical tumor-growth simulation. Later, Pollo et al (2005) from the same group conducted a more qualitative evaluation of the method. Finally, they further improved it by replacing the SSD-based Demons algorithm with an optical flow method based on MI (Bach Cuadra et al 2006). They reported better results for this method because the assumption of linear intensity correspondence between both images could be dropped (see figure 5 for results). Characteristics of each method are summarized in table 5.

Zoom In Zoom Out Reset image size

2.3.7. Summary and conclusion—registration

The major problem when registering a brain tumor image with a healthy atlas is the missing correspondence between pathologic patient image and healthy atlas. One idea to circumvent this problem is to seed the atlas with a tumor prior and simulate tumor growth in the atlas according to the patient image, followed by a final non-rigid registration step.

2.4.1. Types of tumor-growth models

2.4.2. Purely bio-mechanical growth models combined with the registration

2.4.3. Diffusion-based bio-mechanical growth models combined with the registration

Gooya et al (2011a) employed a more sophisticated diffusion–reaction model for tumor growth, which was coupled with a Eulerian method to simulate the mass effect. They operated on multi-sequence images to obtain a probability map of the different tissue classes using SVM classification techniques. The patient-specific tumor was grown in the atlas and a Demons-like algorithm was employed for the localized atlas-to-patient transformation for which the tissue probability maps from the multi-sequence classifier were considered for the cost function. The whole process was formulated in an EM framework to jointly estimate tumor growth and the spatial transformations for adapting the atlas to the patient image. In an extension, Gooya et al (2011b) dropped the requirement for a pre-classification and proposed a joint segmentation and registration model including tumor growth. To this end, they used again an EM algorithm to jointly estimate tumor-growth parameters and registration deformation field to obtain the tissue posterior probabilities in the patient image derived from the atlas. The approach was recently further improved and extended (Gooya et al 2012; see figure 6), where the authors also reported on employing the method for the construction of statistical atlases for gliomas. Characteristics of each method are summarized in table 6.

Zoom In Zoom Out Reset image size

2.4.4. Bio-mechanical growth method based on a cellular model combined with the registration

2.4.5. Summary and conclusion—integrated approaches

In this review, we gave an overview of the state of the art in the MRI-based medical image analysis for brain tumor studies. The focus was on segmentation and registration, whereas some of the registration methods also included tumor-growth modeling. The first attempts in this field were made almost two decades ago, but it can be observed that in recent years, the methods are becoming mature and an increase of their use in clinical practice is expected.

The majority of segmentation approaches operate on multi-sequence MRI data, employing classification methods using different features and taking spatial information in a local neighborhood into account. The trend is not to segment the tumor only, but also to delineate tumor sub-compartments and different healthy regions on images from standard clinical acquisition protocols. This provides the physician with a more comprehensive information on which diagnosis, tumor monitoring and therapy planning can be based. Apart from the evaluation of accuracy and robustness, an important criterion is the computation time. Processing times of a few minutes are the current standard. Performing a better problem-oriented selection of the segmentation technique instead of choosing the technique first and then try to make it work on the current problem could be accomplished in the future by paying more attention to feature selection than the segmentation algorithm.

The registration methods can be separated into intra- and interpatient registration. In intrapatient registration, the main challenge is to handle effects of tumor growth or resection, which is mostly done by tumor image-specific extensions to standard registration algorithms. The majority of interpatient registration approaches focus on the registration of tumor-bearing brain images with a normal atlas. This can be used for the atlas-based segmentation or for constructing statistical brain tumor atlases. One attempt to handle the missing correspondence between healthy atlas and pathologic patient image is solely based on registration methods; a different idea is to combine tumor-growth modeling with registration methods. While the registration approaches are more general, the integrated approaches tend to be more accurate; however, the tumor-growth model adds additional complexity, which also introduces additional risks. A major problem, especially for the integrated approaches, is their tremendous computation time. Therefore, these methods should be considered as pure research methods at the moment and will not reach clinical use until improvements in computational speed are attained.

Comparing the traditional segmentation techniques and the atlas-based segmentation methods, which rely on the registration, it is clear that the traditional segmentation techniques are more flexible and can be more easily adapted to handle multiple modalities simultaneously. Additionally, it is possible to treat individual tumor sub-compartments more easily. Atlas-based methods have advantages when segmenting tissues and structures surrounding the tumor, especially subcortical structures or functional areas. These structures can be segmented more easily if there is atlas-encoded information about spatial localization available. This can have important implications for surgery or radiation therapy. However, many approaches incorporating tumor-growth modeling have difficulties in handling multifocal lesions.

While preparing this review, it became apparent that some papers in the literature lack in precision, describing what exactly is being done, what type and grade of tumor are being considered, what image data are being used or how the algorithm performs in terms of robustness, accuracy and speed. This can be easily seen from the many empty fields in the presented tables. Particularly in the case of medical image analysis for brain tumor studies, better attention should be paid not only to differences according to the tumor type and grade, but also to robustness of the algorithm. Eventually, it would be useful to test any new method on a standard database of brain tumor images to allow for a fair comparison against the state of the art. The MICCAI BraTS dataset⁵ would be one candidate for such a database (see also section 2.1).

Although a lot of research has been done in this field over the last few years, application in the clinics is still limited. Despite the huge amount of work involved and the limited objectiveness, in many cases, clinicians still rely on manual tumor delineations. This is probably mostly due to a lack of communication between researchers and clinicians. Many tools developed so far are pure research tools, which are not easy to handle for clinicians. Therefore, a major effort should be spent on embedding the developed tools into more user-friendly environments in the future. So far, in most commercial workstations, only very simple methods are implemented (e.g. thresholding). One reason for this might be that the technology transfer from the bench to the bedside is still a major challenge because the performance achieved in a well-controlled research environment can often not be reproduced in clinical routine. Recently, more researchers have tried to consider standard clinical acquisition protocols when developing their methods, instead of focusing on feasibility studies that employ pure research data as image material. This will hopefully aid in spreading the application more quickly. Another important aspect is the computation time: the real-time segmentation will be hard to achieve, but computation times which are beyond a few minutes are unacceptable in clinical routine. For everyday use, robustness is crucial. This includes robustness against slight changes in the acquisition protocol, but also the flexibility to handle new acquisition protocols. If a method tends to fail completely in some cases, doctors will lose their trust in the automatic approach and will not be willing to change their habits. Therefore, a thorough demonstration of the robustness should be one of the major steps for each new method.

Currently, image acquisition protocols are not completely standardized. In order to be able to make the best possible use of automatic methods for the medical image analysis, it is essential to have image data, which have been acquired according to a well-defined protocol across different clinical sites. We expect that such a standardization would aid significantly in improving the applicability and spread the use of automatic methods because the need for fine-tuning according to specific imaging characteristics will be eliminated.

Standard therapeutic procedures for brain tumors include resection, followed by fractionated radiation therapy combined with temozolomide chemotherapy (Stupp et al 2009). Due to recent changes in glioma treatment, the traditional Macdonald criteria (Macdonald et al 1990) for the tumor response evaluation are increasingly recognized to be not adequate any more, as effects of pseudo-progression and pseudo-response are not covered by these criteria. Since the formulation of the Macdonald criteria, many novel MR techniques have become available, now widely used in clinical routine, that allow for the non-invasive retrieval of morphological, functional and metabolic properties of brain tumors (e.g. diffusion weighted imaging, dynamic susceptibility contrast-enhanced perfusion imaging, MRS). These techniques may be able to fill the gap in the proper evaluation of tumor progression in the future, together with improvements in the tumor volume estimation from the medical image analysis by forming an integrated platform for advanced tumor assessment.

Eventually, a dialogue should be initiated to convince practitioners that although a software tool might lack in accuracy compared to the manual method in certain cases, it offers more objectiveness, which is very important in longitudinal studies. However, while being objective, the automatic segmentation obviously has to be sufficiently close to what a radiologist perceives as the correct tumor segmentation. If this cannot be achieved by fully automatic methods, it might be worthwhile to consider an interactive method that is initialized or corrected by the user. In practice, the variability introduced by interactive user input is usually much smaller than the variability of completely manual segmentation results. In this case, ease of use for the clinician is crucial for any method to be accepted. It should also not be neglected that for automatic tumor image analysis methods to be used as a routine diagnosis tool, the certification is mandatory. This probably means that the application will not reach a breakthrough in the clinics before commercial software packages are being offered. Current clinical viewing systems are often not even well equipped yet for visualizing changes in multiple image classes over time.

Only very few implementations of the presented methods are publicly available. This impedes a comparison of new methods to existing approaches and also hinders a large-scale evaluation with clinical data from different sites. To the best of our knowledge, at the moment, there are only three methods publicly available for download, which are dedicated to the analysis of brain tumor images:

Prastawa et al (2009) provide their 'TumorSim' software for the simulation of synthetic brain tumor images⁶. Bauer et al (2011a) offer the 'DoctorNo' suite of tools (Bauer and Reyes 2011) as a plug-in for the GUI-based 'DoctorEye' software platform (Skounakis et al 2010) for download⁷. Gooya et al (2011a) provide the 'GLISTR' toolkit on their website⁸.

It can be expected that the current efforts for registration and segmentation of tumor-bearing brain images on standard MRI protocols will enter routine clinical procedures in the next years (Levy and Rubin 2011). In order for this to occur, the proof of robustness and sufficient speed of the methods is mandatory, as outlined before. This development is likely to be pushed with the anticipated shift from the diameter-based MacDonald (Macdonald et al 1990) and RECIST (Therasse et al 2000) criteria to volume-based criteria for the assessment of tumor response to treatment as suggested by the RANO group (Wen et al 2010). However, it remains to be investigated how well automatic methods can handle the impact of treatment effects and related pseudo-progression or pseudo-response on the segmentation outcome, which will also be interesting for clinical trials (Henson et al 2008). Nevertheless, the objectiveness of automatic methods in the tumor segmentation is undisputed and Angelini et al (2012) have made use of this in a first comprehensive study on longitudinal brain tumor data. It is likely that this line of research will be exploited more in the future.

Apart from the standard imaging methods, recently more advanced MRI modalities such as perfusion imaging, DTI and MRS are getting more attention (Cha 2006, Law 2009) in the brain tumor analysis. These modalities are mostly used for tumor grading and for the localization of the most active areas of the tumor. Researchers have started to apply machine learning approaches for tumor grading from perfusion images (Slotboom et al 2008, Zacharaki et al 2009b, Emblem et al 2009, Zöllner et al 2010, Zacharaki et al 2011) and also for the analysis of combined MRI/MRS data (Luts et al 2007, Fayed et al 2008, Georgiadis et al 2011). PET imaging has the potential to provide better prognostic information and distinguish recurrent tumor from radiation necrosis (Chen 2007, Dhermain et al 2010).

Another area, which is likely to get increased attention, is patient-specific image-based modeling of brain tumor growth (Konukoglu et al 2010, Menze et al 2011). This research could help to predict tumor progression and optimize treatment options on an individual basis.