Deformable appearance pyramids for anatomy representation, landmark detection and pathology classification

We begin by describing the parts at multiple scales. The part at a landmark is typically described by an image patch with a certain size. Choice of the patch size can significantly affect the performance of the model. For sharper local structures, a smaller patch can give more precise pixel location. At blurry structures, however, the patch size should be large enough to cover distinguishable textural information. A good feature descriptor is expected to have a high spatial specificity (pixel location) while maintaining good distinctive ability (textural properties). Due to the uncertainty principle in signal processing [15], a single scale patch cannot achieve both. We therefore represent the part with a multi-scale local feature pyramid (LFP), with the smaller scales containing local high frequency features, and the larger scales low frequency components.

A L-level LFP at a landmark, denoted by \(\{A_l\}_{l=1}^L\), is an assembly of patches extract from a L-level image pyramid. The patches \(A_l\) describe the local features with increasing scales and decreasing resolutions in octave intervals. The first-level patch is the smallest one with the finest resolution. A patch in the lth level has l octaves larger scale and lower resolution, which keeps the same size in pixel across all levels, see an example extracted from Gaussian image pyramid in Fig. 2.

A DAP is a part-based deformable model with each part delineated by a LFP. The DAP consists of two components:\(\{\mathcal {A}, \mathbf {s}\}\), with \(\mathcal {A}=\{\{A_{n,l}\}_{l=1}^L\}_{n=1}^N\), called an appearance pyramid, being the assembly of the LFPs, and \(\mathbf {s}\) the geometrical configuration accounting for the deformations. N is the total number of landmarks.

As the patches cover larger anatomical regions at lower-resolution pyramidal layers, fewer number of patches are required to describe the appearance of the anatomy at a coarser level. We trim the patches at these levels preserving only those denoting key features. In practice, a simple trimming algorithm can be designed to iteratively delete the patches which have least distance from their neighbourhood patches until a distance criterion is satisfied. Denoting \(\mathcal {K}_n\) as the subset of scale indices preserved at the nth landmark, the AP becomes \(\mathcal {A} = \{\{A_{n,l}\}_{l\in \mathcal {K}_n}\}_{n=1}^N\). At each level of \(\mathcal {A}\), the patches describe the anatomy with a certain degree of detail, and together, they give a multi-scale description, see Fig. 1b.

Various types of image pyramids can be used to build an AP for appearance delineation. To be able to preserve the full information of the anatomy and reconstruct the appearance, they are chosen to be either pyramids with redundant channels such as Gaussian pyramids or with complementary channels such as wavelet pyramids: we refer to the appearance delineations as Gaussian appearance pyramids and wavelet appearance pyramids, respectively. We briefly illustrate a recent method of wavelet pyramid decomposition in Fig. 3. A detailed introduction can be found in [18].¹

In the explicit method, the geometry is configured with the point distribution shape model. The shape is represented by \(\mathbf {s} = [\mathbf {x}_1, \mathbf {x}_2,\ldots ,\mathbf {x}_N]\), in which \(\mathbf {x}_n\) is the coordinate of the nth landmark. We follow the two-step fitting strategy commonly used in part-based models [11, 12], i.e. local feature searching followed by a geometrical regularisation. The local feature searching gives predictions of the landmark locations, while the shape prior regularises the geometry within plausible variations. The likelihood of a shape instance with respect to the shape prior and local landmark predictions can be calculated by,We show how the prior of the patch appearance \(\mathcal {A}\) is learnt and used for the local feature searching, and the prior of the geometry \(\mathbf {s}\) is learnt for the shape regularisation.

In the implicit model, we deduce the true shape \(\mathbf {s}^*\) from the observation at an initial shape \(\mathcal {A}(\mathbf {s}^{(0)})\), which is solving the regression problem, \(\mathcal {A}(\mathbf {s}^{(0)})\mapsto \mathbf {s}^*\). With SDM algorithm, it can be decomposed into a set of regressors and fitted recursively,Each regressor is modelled linearly by,The parameters \(\{R^{(i)},\mathbf {b}^{(i)}\}\) can be learnt from the training images. Specifically, at the ith iteration, the parameters can be learnt by minimising the residual error of regression in the training set,in which M is the number of training samples. \({\Delta }\mathbf {s}_k^{(i)}\) is the difference between the current shape \(\mathbf {s}^{(i)}\) and the true shape \(\mathbf {s}^*_k\) of the kth training data. In all cases, the initial shape \(\mathbf {s}^{(0)}\) for the first regressor is set as the average shape at the average location in the training dataset. The shape samples for training the subsequent regressors are generated by applying the previous regressor,In practice, to suppress the over-fitting problem in these situations with high-dimensional features and inadequate training data, a L2 regularisation is applied and the objective function (16) becomeswhere \(\lambda \) controls the extent of regularisation. Note that in the implicit model the shape prior is in a nonparametric form and is integrated in the training of the regressors. More details of SDM can be found at Xiong and Torre [16].

To reduce the dimensionality of the descriptors and enhance the fitting performance, instead of using intensity features, a more robust feature descriptor such as histogram of oriented gradients (HOG) [6] can be readily applied on the patches. Denoting \(h{(\cdot )}\) as the feature extraction function, the fitting process can be expressed bywith the parameters \(\{R^{(i)},\mathbf {b}^{(i)}\}\) learnt in the training data by

In the testing stage, the shape of an new object is fitted using the methods presented above. As the pyramidal channels are either redundant or complementary, we can recover the appearance of the object from the DAP. In other words, the objects can be represented compactly by the DAP parameters. Specifically, the shape parameters \(\mathbf {b}_\mathrm{s}\) can be calculated by (9) and the appearance parameters \(\mathbf {b}_\mathcal {A}\) by (3). For the classification tasks, the correspondence of anatomical features should be built such that the differences among the descriptors account for the true variations rather than the misalignment. In a DAP, the appearance correspondence is built by extracting local features at corresponding landmarks. A classifier predicts the label \(\ell \) given an anatomical observation \(\varPhi =[\mathbf {b}_\mathrm{s},\mathbf {b}_\mathcal {A}]\), i.e. \(\ell = \arg \,\text {max} p(\ell | \varPhi )\). The most significant variations in the training data \(\{\varPhi \}\) can be learned by a further PCA and the dimensionality reduced by preserving the significant components, which span a feature space \(P_\varPhi \). A DAP therefore can be represented in the feature space by a compact set of parameters \(\mathbf {b}_\varPhi \), i.e. \(\mathbf {b}_\varPhi = P_\varPhi ^\mathrm{T} (\varPhi -\bar{\varPhi })\), in which \(\bar{\varPhi }\) is the mean of \(\{\varPhi \}\). Using \(\mathbf {b}_\varPhi \) as inputs the classifier now predicts \(\ell = \arg \,\text {max} p(\ell | \mathbf {b}_\varPhi )\). We train the classifier using AdaBoost with 100 learning cycles, with decision trees as the weak learners.

For landmark detection, we evaluate the performance of DAPs with three configurations: Gaussian appearance pyramid with subspace LK as the fitting algorithm, Gaussian appearance pyramid with SDM and wavelet appearance pyramid with SDM. To cover richer pathological variations, we perform the landmark detection on the mixed dataset containing all 600 images. We randomly choose 300 images for training and detect the landmarks on the remaining 300. Two metrics are used for the evaluation: the point-to-boundary distance (PtoBD) and the dice similarity coefficients (DSC) of the canal and disc contours. PtoBD calculates the distance of the fitted landmarks to the ground truth contour, which is more accurate over point-to-point distance. DSC is defined as the amount of the intersection between a fitted shape and the ground truth, \(\text {DSC}=2\cdot tp/(2\cdot tp+fp+fn)\), with tp, fp and fn denoting the true positive, false positive and false negative values, respectively. It considers both the sensitivity and specificity. The mean results of the methods compared are shown in Table 1. We can see that the DAPs with all three configurations outperform the other methods by a favourable margin. In addition, the comparison of the three DAP instances shows that the implicit model with SDM as the fitting algorithm gives better results than the explicit model with subspace LK as the fitting algorithm. Delineating the objects with wavelet appearance pyramids shows further improvement giving the best performance. Several qualitative results by the DAP with wavelet pyramids and SDM fitting algorithm are shown in Fig. 5.

After the landmarks are detected, the DAPs are extracted from the image and used as input for classification. As the SDM algorithm detects the landmarks with higher precision compared with a subspace LK method, we use the landmark locations by SDM in the classification tasks and evaluate the accuracy by Gaussian appearance pyramids and wavelet appearance pyramids.

For central stenosis, in each of the three subsets, the morphology of the central canal is inspected and labelled with three grades: normal, moderate and severe. For illustration, the average appearances of these classes delineated by the wavelet DAP are shown in Fig. 6a. We randomly pick 100 samples to train the classifier and test on the remaining 100 and repeat for 100 times for an unbiased result. The DAP extracted from the detected landmarks are projected onto the feature space and represented by a compact set of parameters (Fig. 5, bottom), which are used as inputs of the classifier. The performance of normal/abnormal classification is measured by accuracy, which is calculated as \((tp+tn)/(tp+tn+fp+fn)\). The grading errors are measured with mean absolute errors (MAE) and root mean squared errors (RMSE). We compare the performance of DAPs against approaches using other models as inputs to the same classifier. The agreements of the results with manual inspection are reported in Table 2. We can see that the Gaussian DAP gives better or competitive performance in the classification and grading of the central stenosis, while the wavelet DAP outperforms the methods compared by a large margin. Similarly, we perform another normal/abnormal classification on the morphology of the neural foremen. The average appearances delineated by the wavelet DAP are given in Fig. 6b. The classification accuracy of methods compared is reported in Table 3. The result shows that the Gaussian DAP gives better performance compared with the popular shape and appearance models. The wavelet version of the DAP enables a further improvement. We believe that the DAP models benefit from its better local feature description and appearance delineation. The further improvement is brought on by the superior properties of wavelets, namely that they are complementary which preserves the full information of discriminating local appearance, and they decompose complex textures into simpler feature components.