3D facial landmarks: Inter-operator variability of manual annotation

The data used in this work consists of 36 facial scans of healthy unrelated subjects, recruited among volunteers in the Danish Blood Donor Study (DBDS) [13]. The 36 subjects were chosen by simple random sampling from our database consisting of facial scans from 641 subjects, having 50% males. The facial scans were captured using a Canfield Vectra M3 Imaging System, at the DBDS facility at Glostrup University Hospital. Each 3D facial scan contains about 70,000 to 100,000 3D points and has shape information (x-, y-, z-point positions) and texture information (red, green, blue intensities) for every 3D point. The study was approved ethically by the Danish Scientific Committee and was reported to the Danish Data Protection Agency. All the patients have given written informed consent prior to inclusion in the project. The facial image used in figures, is a statistically average face and does picture any participant.

The annotation framework initially developed by Fagertun et al. consists of 73 landmarks [14]. Here, 24 anatomical landmarks define distinct facial features, and 49 pseudo-landmarks define the curves and width of the jaw, lips, eyebrows etc. A description is presented in Figure 1.

All scans followed a three-step annotation scheme:

A 2D image was created by orthographic projection of the 3D scan. The face and eyes are automatically detected by a Viola-Jones Rapid Object Detection [17, 18], and serve as a starting point for an AAM search. When the AAM converges, the 73 2D annotation points (Figure 1) can be extracted. These annotation points are then transformed from the 2D image to the 3D scan. The 2D to 3D transformation is likely to fail in high curvature areas like the jaw as points from 2D images are wrongly projected onto the neck. To compensate for this limitation, an ASM search, initialized by an Iterative Closest Point search [19], is performed to locate the jaw in 3D. The annotation points are then manually corrected by an operator see section “Manual annotation tools and standard”. In summary, the low curvature points are found by a 2D AAM and transformed to 3D image, while high curvature points are found by a 3D ASM.

The 2D AAM and 3D ASM were constructed based on 605 individuals recorded by a Nikon D90 in 2D and a Canfield Vectra M3 Imaging System in 3D, respectively. Both the 2D and 3D data were annotated to create correspondence between individuals, in the same fashion as described in the following section.

The object of the manual annotation was to reach a consistent and stable standard for annotation. Prior to the study the annotation scheme was explained and discussed during a three-hour workshop (common training program), to ensure a common frame of reference. Further, all operators had annotated more than 100 scans prior to this training program. The manual annotation is a two-step process. First, the annotation is performed in a fixed frontal view by a custom-made annotation tool. The fixed view was chosen over free flying mode to allow faster annotation time. Second, points in high curvature areas are adjusted in fixed frontal, profile, and top/down views. The high curvature points are the jaw and nose points (35-45 and 63-73).

To analyze facial shape variation at positions not annotated by landmarks, a dense point correspondence is created. A variety of methods exist for establishing dense correspondence. In this work we employ a method that has previously produced excellent results when a sparse set of landmarks exist [6].

This method is based on propagating a well-formed template mesh to all shapes in the training set. For each shape the template mesh is initially deformed using a volumetric thin-plate spline warp [20] and using the sparse set of corresponding landmarks. In the next step the mesh vertices of the deformed template mesh are propagated to the target shape. This approach is very similar to the method used to create the dense surface models described by Hutton et al.[1, 4, 5]. While propagating each vertex to the Euclidian closest point on the target surface works for simple anatomy, it fails in regions with moderate curvature. A proven solution is to regularize the correspondence field and add curvature information in the propagation step. In Paulsen [6] and Hilger [21] this regularization is cast into a Markov Random Field (MRF) framework [22], where a prior and an observation term are defined. The prior model imposes a Gaussian prior on the deformation field that favors smooth deformation fields. The curvature of the deformed template mesh and the target shape is used in the observation term to guide the correspondence to areas with similar curvature. The mean curvature is estimated as the radius of a locally fitted sphere [23]. Finally, the regularization is bounded so the projected points are on the surface of the target shape. The optimal correspondence field is found using stochastic optimization. The involved weighting between the prior and observation terms is found as the weight that creates the most compact shape model as described by Hilger [21]. The result is a regularized dense correspondence between the template and all the shapes in the training set. In our experiments, the dense correspondence consists of 39,653 points and the associated mesh connectivity from the template mesh.

All results were produced with SAS version 9.4 and Matlab version R2010b.

Six operators(one female) annotated 36 scans (50% male, aged 18 to 65) twice, one week apart. All six operators went through a common training program and were unblinded to the study aim. The mean error and standard deviation of the combined variance of each annotation point are shown in Figure 2. We observed a association between the variability and the the specific annotation point. The center of the pupil was associated with minimal variance (SD = 0.09 mm), followed by landmarks of the eye (SD = 0.30-0.95 mm). The most error-prone annotation points are the landmarks of the jaw (SD = 1.55-3.34 mm), although the lateral points of the eyebrows are also error prone (SD = 2.24-2.37 mm). The variance of each annotation point is illustrated in Figure 3.

In order to test the PDM stability for different operator annotations we adapted a coupled leave-one-out cross-validation scheme. We built a PDM for a single operator and a PDM from a random sampling (using the same number of scans as the former) for the remaining five operators. The PDM’s are built on 35 individuals and the reconstruction error is measured on the 36th individual. We then loop over all individuals in the inner leave-one-out cross-validation loop and over all operators in the outer leave-one-out cross-validation loop. The mean reconstruction error to the mean annotation points for all six operators is presented in Table 1. The table shows that the PDM constructed by random operators is consistently better at reconstructing the annotations. Interestingly no single operator yields a PDM that perform better than the randomly selected PDM.

The 73 annotated points were associated with different variability. We tested four different sub selections of these annotation points in a effort to minimize the variance of the resulting dense point correspondence. Two landmark selections simply excluded annotation points with the highest variance in mean error (>1 mm) and operator error (>0.5 mm), respectively. Two landmark selections which aim at selecting landmarks from the main facial features and with low mean error and variance are also tested, see Figure 5.

The quality of the derived dense point correspondence was evaluated by the size of variance between the correspondence points. If the points had good correspondence, the resulting variation of the correspondence points will manly describe the difference in the samples (population variance). In the case of poor correspondence, the variation will now account for both the population variance and the inconsistency of correspondence points leading to a higher variation. We measured the dense mean point variance for each annotation point and the four suggested landmark selections (Tables 2 and 3).

The lowest variance is seen for landmark selection 2, having a mean variance = 0.54 mm. Figure 6 illustrates the variation of the PDM from landmark selection 2 compared to the original PDM (based on all landmarks). Annotated points with relatively small inter-operator variation, was not estimated better automatically. However, points with large inter-operator variation was better estimated. Based on these results we conclude that a reduced set of the original 73 landmarks provides optimal annotation. It is also noted that landmark selection 2, which consists of 14 landmarks, results in a more compact model and improves estimation of 16 of the 73 landmarks compared to the manual operator annotations, see (Table 3).