Evaluation of the channelized Hotelling observer with an internal-noise model in a train-test paradigm for cardiac SPECT defect detection

The first operator, U, called the channeling operator, measures numerical features of the image by applying filters that are intended to model the human visual system (Myers and Barrett 1987). The experiments reported in this paper tested the four most commonly used types of channeling operators.

• (1)  
  CHO-BP: rotationally symmetric, square profile, non-overlapping, bandpass filters, having cutoff frequencies of [1/32, 1/16, 1/8, 1/4, 1/2] cycles/pixel, yielding a total of four channels (see figure 1(a)) (Myers and Barrett 1987, Abbey and Barrett 2001, Barrett and Myers 2004).
• (2)  
  CHO-GB: Gabor filters with three spatial frequency bands of [1/8, 1/4, 1/2] cycles/pixel and eight orientations from 0 to π. This results in 24 channels (see figure 1(b)) (Barrett and Myers 2004, Zhang et al 2004, 2006). Like the bandpass filters, the Gabor filters are intended to reflect the response of neurons in the primary visual cortex.
• (3)  
  CHO-LG: Laguerre–Gauss polynomials. This paper considers polynomials of order 0–6 with three orientations: vertical, horizontal and rotationally invariant, yielding 18 channels (see figure 1(c)) (Barrett and Myers 2004, Zhang et al 2004).
• (4)  
  CHO-DOG: rotationally symmetric, overlapping, difference-of-Gaussians (DOG) profiles as defined in (Narayan and Herman 1999, Abbey and Barrett 2001) yielding ten channeling filters (see figure 1(d)).

Zoom In Zoom Out Reset image size

We also tested the so-called sparse DOG, using profiles as defined in Abbey and Barrett (2001). However, as this channel model is a special case of the DOG and yields similar results, we dropped it from further consideration.

All channels are designed to have non-zero values on a 71 × 71 pixel window centered at the defect location. The window size is chosen such that average defect spectral support FWHM is approximately 1/4 cycles/pixel.

Letting u_i, i = 1, 2, ..., M, denote vectors obtained by lexicographic ordering of the channels' spatial response functions, and letting M represent the number of channels (in our experiments M = 4, 24, 18 or 10 corresponding to BP, GP, LG and DOG channeling operator, respectively), the channeling operator U is defined as:

$$\begin{equation} {\bf U} = [ {{\bf u}_{\rm 1} ,{\bf u}_{\rm 2} , \ldots ,{\bf u}_M } ]^{ T} . \end{equation} \tag{ 1 }$$

Without loss of generality let us assume that the channels are normalized as follows:

$$\begin{equation} {\bf u}_i^{T} {\bf u}_i = 1,\quad \;i = 1,2, \ldots ,M; \end{equation} \tag{ 2 }$$

thus the channel outputs are given by:

$$\begin{equation} {\bf x} = {\bf Uf}. \end{equation} \tag{ 3 }$$

Here it is worth noting that only the BP channel model has non-overlapping (independent) channels, i.e., UU^T = I; in the other models, the channel outputs are correlated.

As explained later, an internal-noise model is used in the CHO to enhance prediction accuracy for HO performance (Burgess and Colborne 1988, Lu and Dosher 1999). Specifically, a noise vector with normal distribution of zero mean and covariance K_int, ε ∼ N(0, K_int), is injected into all of the channel outputs (Abbey and Barrett 2001), which become:

$$\begin{equation} x_i = {\bf u}_i^{T} {\bf f} + \varepsilon _i ,\qquad i = 1, \ldots ,M. \end{equation} \tag{ 4 }$$

The channeling operator is followed by a Hotelling observer, a linear classifier that computes a test statistic for choosing between two hypotheses in a cardiac perfusion-defect-detection task—defect present, H₁, and defect absent, H₀—based on the observed feature vector x as:

$$\begin{equation} t({\bf x}) = {\bf w}^{ T} ( {{\bf x} + \varepsilon } ) = \sum\limits_{i = 1}^M {w_i \big( {{\bf u}_i^{ T} {\bf f} + \varepsilon _i } \big),} \end{equation} \tag{ 5 }$$

where

$$\begin{equation} {\bf w} = {\bf K}^{ - 1} {\bf U}\Delta \bar{\bf f}, \end{equation} \tag{ 6 }$$

$\bar {\bf f}_j = \langle {\bf f} \rangle _{H_j }$, $\Delta \bar{\bf f} = \bar{\bf f}_1 - \bar {\bf f}_0$, and K = K_ext + K_int, in which the external-noise covariance matrix K_ext describing noise originating in the data rather than in the visual system, is given by

$$\begin{equation} {\bf K}_{{\rm ext}} = \frac{1}{2}\sum\limits_{j = 1}^2 {\bf U} [ {\langle {[{\bf f} - \bar {\bf f}_j ][{\bf f} - \bar {\bf f}_j ]^{T} } \rangle _{H_j } } ]{\bf U}^{{{ T} } } , \end{equation} \tag{ 7 }$$

where $\langle \cdot \rangle_{H_j }$ denotes conditional expectation under hypothesis H_j. Further details about the optimality of the CHO can be found in Barrett and Myers (2004). In a BKS scenario K_ext will also incorporate statistical information about background variability.

Note that the test statistic, a cascade of two linear operators—a channeling operator and an observer—can be expressed as:

$$\begin{equation} t({\bf x}) = t({\bf Uf}) = {\bf w}^{ T} {\bf Uf} = {\bf w}_{\bf f}^{T} {\bf f}. \end{equation} \tag{ 8 }$$

Therefore, the CHO effectively applies to the image f an image-domain spatial template w_f, defined as:

$$\begin{equation} {\bf w}_{\bf f}^{T} = {\bf w}^{T} {\bf U} = ( {( {{\bf K}_{{\rm ext}} + {\bf K}_{{\rm int}} } )^{ - 1} {\bf U}\Delta \bar {\bf f}} )^{T} {\bf U}. \end{equation} \tag{ 9 }$$

In experiments presented later the external-noise covariance matrix K_ext and Δf are substituted with sample estimates $\tilde {\bf K}_{{\rm ext}}$ and $\Delta \tilde{\bf f}$, exploiting a subset of the available data.

For a comparison between image-domain spatial templates such as those in equation (9), and HOs' image-domain spatial templates, see (Abbey and Eckstein 2002, Castella et al 2007).

Burgess and Colborne (1988) and Lu and Dosher (1999) showed that HOs exhibit inconsistencies that can be described by a so-called decision-variable noise model, which can be described mathematically by injecting noise, with normal distribution of zero mean and variance $\sigma _\gamma ^2$, into the test-statistic variable as follows:

$$\begin{equation} t({\bf x}) = {\bf w}^T ( {{\bf x} + \varepsilon } ) + \gamma . \end{equation} \tag{ 10 }$$

In a detection study, image quality is assessed by the degree to which the HO (approximated by the NO) can correctly perform the task. This performance is typically quantified by using metrics from decision theory, notably the AUC (Barrett et al 1998, Abbey and Barrett 2001). In this setting, the AUC can be expressed simply in terms of a signal-to-noise ratio (SNR) as follows:

$$\begin{equation} {\rm AUC} = \frac{1}{2}\left[ {1 + {\rm erf}\left( {\frac{{{\rm SNR}}}{{\sqrt 2 }}} \right)} \right], \end{equation} \tag{ 11 }$$

where

$$\begin{equation} {\rm SNR} = \frac{{\langle {t({\bf x})} \rangle _{H_1 } - \langle {t({\bf x})} \rangle _{H_0 } }}{{[ {{{\rm var}} \langle {t({\bf x})} \rangle _{H_{0,} H_1 } }]^{\frac{1}{2}} }}. \end{equation} \tag{ 12 }$$

One can show that:

$$\begin{equation} \langle {t({\bf x})} \rangle _{H_1 } - \langle {t({\bf x})} \rangle _{H_0 } = ( {{\bf U}\Delta \bar{\bf f}} )^T {\bf K}^{ - 1} {\bf U}\Delta \bar{\bf f}, \end{equation} \tag{ 13 }$$

and, if the variances under the two hypotheses are equal, then the variance of the decision variable t(x) is given by:

$$\begin{eqnarray} &&\sigma _{t(\bf x)}^2 = {\rm var}\langle {t({\bf x})} \rangle _{H_0 ,H_1 } = {\bf w}^T [ {{\bf K}_{{\rm ext}} + {\bf K}_{{\rm int}} } ]{\bf w} + \sigma _\gamma ^2 \nonumber\\ &&\hphantom{\sigma _{t(x)}^2 } = \sigma _{{\rm ext}}^2 + \sigma _{{\rm int}}^2 + \sigma _\gamma ^2 , \end{eqnarray} \tag{ 14 }$$

where $\sigma _{{\rm ext}}^2$ is the variance of decision variable t(x) due to the external (data) noise, $\sigma _{{\rm int}}^2$ is the variance due to the channels' internal noise ε and $\sigma _\gamma ^2$ is the variance of the decision variable's internal noise γ. Note that, for the linear CHO, injection of the channels' internal noise is equivalent to addition of decision-variable noise with variance $\sigma _{{\rm int}}^2$.

Using equations (8) and (9) and steps similar to those shown in Abbey and Barrett (2001) or Gallas and Barrett (2003), one can show that:

$$\begin{equation} {\rm SNR} = [ {( {{\bf U}\Delta \bar{\bf f}} )^T ( {{\bf K}_{{\rm ext}} + {\bf K}_{{\rm int}} } )^{ - 1} ( {{\bf U}\Delta \bar{\bf f}} )} ]^{\frac{1}{2}} \frac{1}{{\sqrt {1 + \frac{{\sigma _\gamma ^2 }}{{\sigma _{{\rm ext}}^2 + \sigma _{{\rm int}}^2 }}} }}. \end{equation} \tag{ 15 }$$

Note that in the SNR and AUC calculations, $\Delta \bar {\bf f}$ and K_ext will be re-estimated for every reconstruction method separately and only K_int, $\sigma _\gamma ^2$ or $\sigma _{{\rm int}}^2$ (whichever exists in the particular internal-noise model being assessed) will be evaluated across different reconstruction methods in the proposed train-test paradigm.

The CHO internal-noise model can be interpreted as a regularized linear regression, as explained next. Let us assume that we wish to use the following channelized linear regression model:

$$\begin{equation} i_j = {\bf w}_{{\rm reg}}^T {\bf Uf}_j ,\qquad j = 1,2,\ldots,N,\end{equation} \tag{ 16 }$$

where f_j is the jth image, represented as a vector and N is the total number of images and $i_j^{}$ is regression output.

Now for this linear regression model one can employ penalized least-squares estimation, which minimizes differences between the image label I_j and regression output i_j, to find the optimal value ${\bf w}_{{\rm reg}}^*$, i.e.:

$$\begin{equation} {\bf w}_{{\rm reg}}^* = {\rm arg}\;\mathop {{\rm min}}\limits_{{\bf w}_{{\rm reg}} } \sum\limits_{j = 1}^N {\big( {I_j - {\bf w}_{{\rm reg}}^T {\bf Uf}_j } \big)^2 } + \lambda {\bf w}_{{\rm reg}}^T {\bf Aw}_{{\rm reg}}, \end{equation} \tag{ 17 }$$

in which A is a matrix defining the regularization term $\lambda {\bf w}_{{\rm reg}}^T {\bf Aw}_{{\rm reg}}$ and λ is the regularization parameter.

Using this model it is easy to show (see the appendix) that:

$$\begin{equation} {\bf w}_{{\rm reg}}^* = \left( {{\bf U}\sum\limits_{j = 1}^N {{\bf f}_j {\bf f}_j^T } {\bf U}^T + \lambda {\bf A}} \right)^{ - 1} {\bf U}\sum\limits_{j = 1}^N {{\bf f}_j I_j } .\end{equation} \tag{ 18 }$$

Now substituting for I_j

$$\begin{equation} I_j = \left\{\begin{array}{@{}ll@{}} 1 & {\bf f}_j \in H_1\\ -1 & {\bf f}_j \in H_0, \end{array}\right. \end{equation} \tag{ 19 }$$

and assuming that:

$$\begin{equation} \sum\limits_j {{\bf f}_j } = {\bf 0}, \end{equation} \tag{ 20 }$$

yields

$$\begin{equation} {\bf w}_{{\rm reg}}^* = \frac{1}{2}\left( {{\bf \tilde K}_{{\rm ext}} + \frac{\lambda }{N}{\bf A}} \right)^{ - 1} {\bf U}\Delta {\tilde{\bf f}}, \end{equation} \tag{ 21 }$$

which has the same form as (6) with ${\bf \tilde K}_{{\rm ext}}$ and $\Delta {\bf \tilde f}$ being sample estimates of K_ext and $\Delta {\bf \bar f}$. Note that the image pixel values are (theoretically) non-negative since they represent concentrations of radiotracer, therefore the assumption in equation (20) must be forced upon images by image centering (for example). However this centering neither has an effect on the CHO performance as calculated according to the SNR in equations (5)–(12), nor on HO performance (since the images are rescaled before displaying them on a computer screen).

The previous derivation helps us to understand the CHO with internal-noise mode and its interpretation.

• (1)  
  The CHO is a regularized linear regression that predicts the true hypothesis label I rather than the human confidence rating S. By utilizing this consideration, the internal-noise covariance ${\bf K}_{{\mathop{\rm int}} }$ acts to stabilize the matrix inversion in equation (6). K_ext itself is invertible but usually varies wildly between different data subsets. Note that equation (19) can be modified by substituting I_j with S_j so that the model fits the human confidence rating. A somewhat similar consideration is presented in Abbey and Eckstein (2002), Castella et al (2007).
• (2)  
  The channels' internal noise modulates the regularization term for the channel templates, which consequently regularizes the image-domain templates. This is also evident when comparing image-domain templates (see figures 5–8 and compare unregularized models 1, 8, 9, 10, models without internal channel noise, and regularized models 2, 3, 4, 5, 6, 7, 11, models with internal channel noise).
• (3)  
  There may be some benefits of imposing regularization directly on the image-domain templates rather than on the channels. This will be explored in future work.
• (4)  
  In a BKE task, the CHO does not take into account the image background, only the defect signature $\Delta {\bf \bar f}$. In a BKS scenario, K_ext will incorporate the statistical properties of the background.

This work does not aim to understand the human visual system; instead the goal is to compare noise models in search of the best possible predictor of HO performance in a defect-detection task, within the CHO framework. For a broader view of the visual system and its modeling, readers can refer to Burgess and Colborne (1988), Peli (1996), (2001), Barten (1999), Barrett and Myers (2004) and Zhang et al (2007).

Next, ten existing models are reviewed and evaluated, and a new model is proposed and tested (model 6).

The following are brief descriptions of each of the models considered, the first being an absence of internal noise (model 1). Models 2–7 have internal noise (K_int ≠ 0) but no decision-variable noise ($\sigma _\gamma ^2 = 0$). Models 8–10 have decision-variable noise ($\sigma _\gamma ^2 \ne 0$), but no internal noise (K_int = 0). Model 11 is a 'combined' model that incorporates both internal noise and decision-variable noise.

The number of model parameters that must be tuned to optimize performance is specified for each of the models described next.

2.6.1. Model 1: no internal noise

This model, in which no internal noise or decision-variable noise is added, will serve as a baseline for comparison (Myers and Barrett 1987, Yao and Barrett 1992):

$$\begin{equation} {\bf K}_{{\rm int}} = {\bf 0}\quad {\rm and}\quad \sigma _\gamma ^2 = 0.\end{equation} \tag{ 22 }$$

In this case, $\sigma _{t({\bf x})}^2 = \sigma _{{\rm ext}}^2$, and SNR is denoted by SNR_IO which is calculated as ${\rm SNR}_{\rm IO} = \sqrt {( {{\bf U}\Delta {\bf \bar f}} )^T {\bf K}_{{\rm ext}}^{ - 1} {\bf U}\Delta {\bf \bar f}}$:

Number of model parameters to tune: 0.

2.6.2. Model 2: quantization noise

In (Burgess 1985) the authors suggested that quantization of image intensity by a display device can be seen as a source of internal noise. In this internal-noise model, which was evaluated in (Narayanan et al 2002), ${\bf K}_{{\mathop{\rm int}} }$ is a diagonal matrix, the elements of which are given by:

$$\begin{equation} [ {{\bf K}_{{\mathop{\rm int}} } } ]_{i,i} = \frac{{Q^2 }}{{12}}, \end{equation} \tag{ 23 }$$

where

$$\begin{equation} Q = \frac{{\max \langle {\bf f} \rangle _{H_0 \cup H_1 } - \min \langle {\bf f} \rangle _{H_0 \cup H_1 } }}{{256}}. \end{equation} \tag{ 24 }$$

Note. It can be shown (Grubbs and Weaver 1947, Stark and Brankov 2004) that Q is proportional to the variance of f.

Number of model parameters to tune: 0.

2.6.3. Model 3: uniform-variance internal noise

In this noise model a constant variance σ² (Oldan et al 2004, Gilland et al 2006) is added to each channel, so that ${\bf K}_{{\mathop{\rm int}} }$ is a diagonal matrix, the elements of which are given by:

$$\begin{equation} [ {{\bf K}_{{\mathop{\rm int}} } } ]_{i,i} = \sigma ^2 . \end{equation} \tag{ 25 }$$

Note that, despite appearances, model 2 is a not special case of this model, because Q is a function of the data variance.

Number of model parameters to tune: 1.

2.6.4. Model 4: non-uniform internal-noise variance, proportional to external-noise variance

In this model the injected noise has channel variances proportional to the external-noise variances (Abbey and Barrett 2001, Oldan et al 2004), so that ${\bf K}_{{\mathop{\rm int}} }$ is a diagonal matrix having elements given by:

$$\begin{equation} [ {{\bf K}_{{\mathop{\rm int}} } } ]_{i,i} = p[ {{\bf K}_{{\rm ext}} } ]_{i,i} . \end{equation} \tag{ 26 }$$

Number of model parameters to adjust: 1.

2.6.5. Model 5: uniform internal-noise variance, proportional to the maximum external-noise variance

In this model the injected noise variances are proportional to the maximum variance of the channels' external noise (Barrett et al 1998) so that ${\bf K}_{{\mathop{\rm int}} }$ has diagonal elements given by:

$$\begin{equation} [ {{\bf K}_{{\mathop{\rm int}} } } ]_{i,i} = p\mathop {\max }\limits_j [ {{\bf K}_{{\rm ext}} } ]_{j,j}. \end{equation} \tag{ 27 }$$

Number of model parameters to tune: 1.

2.6.6. Model 6: non-uniform internal-noise variance, proportional to external-noise standard deviation

In this paper we propose the following new model, entirely motivated by the decision-variable internal-noise model 9, described later. In Model 9 the decision-variable noise variance is proportional to the decision-variable standard deviation due to external noise whereas here, in model 6, the channels' internal noise has variances proportional to the standard deviations of the channels' external noise, so ${\bf K}_{{\mathop{\rm int}} }$ is a diagonal matrix having elements given by:

$$\begin{equation} [ {{\bf K}_{{\mathop{\rm int}} } } ]_{i,i} = p\sqrt {[ {{\bf K}_{{\rm ext}} }]_{i,i} }. \end{equation} \tag{ 28 }$$

Number of model parameters to tune: 1.

2.6.7. Model 7: non-uniform compound noise

In this model ${\bf K}_{{\mathop{\rm int}} }$ is a diagonal matrix having elements given by Kulkarni et al (2007):

$$\begin{equation} [ {{\bf K}_{{\mathop{\rm int}} } }]_{i,i} = p_1 [ {{\bf K}_{{\rm ext}} }]_{i,i} + p_2 \;\mathop {\max }\limits_j [ {{\bf K}_{{\rm ext}} }]_{j,j}. \end{equation} \tag{ 29 }$$

Number of model parameters to tune: 2.

2.6.8. Model 8: constant variance decision-variable noise

In this model (Nagaraja 1964, Zhang et al 2007) decision-variable noise has constant variance

$$\begin{equation} \sigma _\gamma ^2 = \sigma ^2 . \end{equation} \tag{ 30 }$$

Number of model parameters to tune: 1.

2.6.9. Model 9: decision-variable variance proportional to the external-noise standard deviation

This model, suggested in Burgess and Colborne (1988) and evaluated in Zhang et al (2007), has the decision-variable noise variance proportional to the decision-variable standard deviation due to external noise.

$$\begin{equation} \sigma _\gamma ^2 = p\sigma _{{\rm ext}} . \end{equation} \tag{ 31 }$$

Number of model parameters to tune: 1.

2.6.10. Model 10: decision-variable variance proportional to the external-noise variance

This model was suggested in Zhang et al (2004), wherein the internal-noise variance is proportional to the external-noise variance:

$$\begin{equation} \sigma _\gamma ^2 = p\sigma _{{\rm ext}}^2 . \end{equation} \tag{ 32 }$$

Number of model parameters to tune: 1.

This model is equivalent to injecting internal noise with covariance matrix proportional to the external-noise covariance matrix; that is,

$$\begin{equation} \sigma _{t(\bf x)}^2 = {\bf w}^t ( {{\bf K}_{{\rm ext}} + p{\bf K}_{{\rm ext}} } ){\bf w} = \sigma _{{\rm ext}}^2 + p\sigma _{{\rm ext}}^2 , \end{equation} \tag{ 33 }$$

It is easy to show that:

$$\begin{equation} \frac{{{\rm SNR}}}{{{\rm SNR}_{{\rm IO}} }} = \frac{1}{{\sqrt {( {1 + p} )} }}\end{equation} \tag{ 34 }$$

where SNR_IO is as defined in section 2.6.1. This ratio is usually defined as a relative observer efficiency that Burgess et al (Burgess 1985, Burgess and Colborne 1988, Park et al 2007) calculated to be in the range 0.4–0.8; therefore,p ∈ [2.5, 7.25].

2.6.11. Model 11: combination

In Eckstein et al (2003) the authors proposed a noise model that combines internal and decision-variable noise models so that ${\bf K}_{{\mathop{\rm int}} }$ has diagonal elements as:

$$\begin{equation} [ {{\bf K}_{{\mathop{\rm int}} } } ]_{i,i} = p_1 [ {{\bf K}_{{\rm ext}} } ]_{i,i}, \end{equation} \tag{ 35 }$$

with the decision-variable noise variance given by

$$\begin{equation} \sigma _\gamma ^2 = p_2 \sigma _{{\rm ext}}^2 . \end{equation} \tag{ 36 }$$

Number of model parameters to tune: 2.

We will use the following criteria to evaluate performance of the CHO using the various internal-noise models.

• (1)  
  Mean-squared error (MSE). MSE is a widely used metric that reflects model-fitting accuracy measured by the squared distance between estimated values and target values. This measure is a good indication of model accuracy, but is not sufficient to serve by itself as the only model selection criterion, as explained next.
• (2)  
  Kendall's tau rank correlation coefficient (Ktau). (Kendall 1948) The Kendall tau coefficient measures the degree of correspondence between two sets of rankings, ranging from −1 (anticorrelated) to +1 (perfectly correlated). In our evaluation we use the Kendall tau coefficient to assess the degree of correspondence between the rankings of the reconstruction algorithms as predicted by AUC values calculated using NOs, with that of the HOs. Note that Ktau measures the extent to which, as one variable increases, the other variable tends to increase, without requiring a linear relationship. In practice, when evaluating image-processing methods, the rank-ordering of performance is usually of great interest. However we were unable to use the Kendall tau to optimize internal-noise models because this matric is highly nonlinear and insensitive to large changes of internal-noise model parameters.
• (3)  
  Model parameter stability (MPS). For a specific diagnostic task, it is desirable that the optimal internal-noise parameters not vary significantly between data sets. The presence of such variability for a given model would suggest that the model is unstable and, therefore, not useful for practical applications. We use the ratio of internal model parameters to quantify the stability when changing from one data to another. A value of one for this ratio suggests good repeatability (stability); the value can range from zero to infinity. To our knowledge, this aspect of CHO behavior has not been explored previously, although in Zhang et al (2004) the authors noted that the performance of model 10 can depend significantly on the choice of the data set. In cases where internal-noise model has two parameters a ratio of each parameter is taken and an average is reported.
• (4)  
  Pearson correlation coefficient (PCC). Just as the model parameters should not vary significantly between data sets, the image-domain spatial template, w_f, as defined in equation (9) should also remain relatively consistent. This may not be true in general, but for the experimental data used in this paper, the images are not significantly different, as can be seen in figure 2. This is even more evident if one considers $\Delta {\bf \bar f}$, the defect images, given in the same figure. If the image-domain spatial template is not consistent it may indicate that the model has suffered from overfitting to a given data set, and has failed to capture intrinsic properties of the HO. We quantify this aspect of model stability by using the PCC to compare the obtained spatial templates. PCC between two spatial templates is defined as the covariance between pixel values among two spatial templates, divided by the product of their standard deviations. PCC ranges from −1 (anticorrelated) to +1 (perfectly correlated). This coefficient has been used successfully to test brain-image analysis procedures for reproducibility (Strother et al 2002).

Zoom In Zoom Out Reset image size

In summary, we argue that an ideal internal-noise model should yield the following behavior. It should produce (1) AUC values that are close to HO AUC as judged by MSE, (2) high values for the rank correlation coefficient (Ktau ∼ 1), (3) stable model parameters (MPS ∼ 1) and (4) stable templates (PCC ∼ 1).

One may argue that a reasonably useful model might not require all these properties. For example, one may view property 2 (Ktau) as more important than property 1 (MSE). To the best of our knowledge there has not been a reported study exploring this issue, and no guidance has been proposed as to which metric is the most appropriate. However, as we argue in the next section, using solely the Ktau metric is not suitable, and we suggest that all presented model selection criteria be used.

The process of model training (parameter tuning) consists of finding values for the internal-noise model parameters that optimize an optimality criterion. In early experiments we used Ktau rank ordering, as well as a combination of MSE and Ktau, as the optimality criteria for internal-noise parameter tuning; however, model accuracy and correlation with the HO obtained by this approach were not as good as those achieved by using the MSE criterion alone. As we pointed out earlier, the Kendall tau coefficient is extremely nonlinear and insensitive to large changes of the internal-noise model parameters; as such, it is not appropriate for tuning of model parameters, so we instead completed the studies using MSE as the optimality criterion.

In this work, we optimized the internal-noise parameter first by exhaustive search on a coarse grid ranging over 14 orders of magnitude from 10⁻⁷ to 10¹⁰ to ensure that the best match between the AUC of HO and CHO with internal-noise model is not missed. This search was further refined, on a finer grid, by focusing to a range that spans four orders of magnitude where the minimum is located. At each successive iteration the search range was reduced by an order of magnitude for a total of ten iterations. Usually, the method reached a stable solution after five iterations.

In our experiments we used a previously published HO study (Narayanan et al 2002), in which the MCAT phantom (Pretorius et al 1999) was used to generate average activity and attenuation maps, including respiratory motion, the wringing motion of the beating heart, and heart chamber contraction. The maps were sampled on a grid of 128 × 128 × 128 with a pixel size of 0.317 cm. Projections (128 × 128 images over 60 angles spanning 360) were generated by Monte Carlo methods using SIMIND (Ljungberg and Strand 1989), simulating the effects of non-uniform attenuation, photon scatter and distance dependent resolution corresponding to a low-energy high-resolution collimator. These projections were then resampled on a 64 × 64 grid over 60 angles. The simulated perfusion defect was located in the territory supplied by the left-anterior descending artery and had an angular extent of 45. The uptake levels for the perfusion defects were set at 65% of the normal uptake in the left ventricular walls in order to obtain nontrivial detection results. The image noise level is that of a typical clinical study with 0.5M counts from the heart region.

In our evaluation study, we used images reconstructed using the OSEM (Hudson and Larkin 1994) algorithm, with one or five effective iterations, incorporating attenuation correction and resolution recovery. These images were low-pass filtered with three-dimensional Gaussian filters with different FWHM of 0, 1, 2, 3, 4, or 5 pixels (see example images in figure 2). A single short-axis slice was extracted and interpolated to a 160 × 160 pixels image. Note that the combinations of filter FWHM and number of iterations yields 12 distinct reconstruction strategies.

Two medical physicists evaluated the defect visibility in a SKE environment (which also assumes location-known-exactly) for images at every combination of the number of iterations and FWHM of the filter. For each parameter combination of the reconstruction algorithm, a total of 100 noisy image realizations were scored by the observers (50 with defect present and 50 with defect absent) on a six-point scale following a HO training session involving an additional 60 images. The estimated AUC was calculated for each setting by using ROCKIT (Metz et al 1998) (see figure 3(a)).

Zoom In Zoom Out Reset image size

Note that HO AUC curves have a nonlinear shape as a function of the reconstruction parameter (here spatial smoothing), thus complicating the task of matching HO AUC.

We also report p-values, i.e., the probability of obtaining by chance AUC values as extreme as the one that was actually observed, under the null hypothesis that there is no difference between methods (see figure 3(b)). The reported p-values indicate that it is sufficient to use two observers and 100 images if the goal is to determine whether one reconstruction methods AUC is statistically different from the other (Obuchowski et al 2004). Therefore, the data set used in this study is sufficient for numerical model tuning and testing.

For each noise model, we performed model evaluation by means of MSE, Ktau, MPS and PCC using two different comparisons.

Comparison 1: fitting accuracy. Here we tested each noise model using the same type of images as in model optimization, but with different noise realizations. A NO's ability to fit to HO data is a necessary, but not sufficient, condition for a NO to be useful. Specifically, there is no need to apply a NO on images reconstructed in the same way as those used in the NO training phase, since HO performance for this reconstruction method is already available from the HO study. Such testing would satisfy a general train-testing paradigm, but it is not sufficient for real-life NO applications.

Further, a NO trained to predict HO performance only on images reconstructed by one algorithm may be not accurate in predicting performance for a different reconstruction algorithm, defeating the very purpose of this approach.

To indicate results obtained as part of comparison 1, the names of the metrics are prefixed by the letter 'F' (short for 'fitting'; i.e., F-MSE and F-Ktau). The reported values are averaged over six FWHM.

We used the following training procedure. For each reconstruction method, AUC was calculated according equations (11) using half of the available noise realizations for every value of the filter FWHM, with one iteration of OSEM. The internal-noise parameters were adjusted by exhaustive search to maximize agreement between the HOs' AUC and the estimated AUC. The internal-noise parameters thus obtained were then applied to form predictions from the remaining noise realizations, yielding six AUC values. We then repeated the experiment with five iterations of OSEM, yielding an additional six AUC values. These 12 AUC values were compared to that of the HOs, and an average MSE and Ktau is reported.

Comparison 2: generalization accuracy. As we pointed out earlier, an important purpose of a NO is to provide an estimate of lesion-detection performance as a measure of image quality for reconstruction methods not yet evaluated by a HO ROC study. Therefore, to be useful, a NO must accurately predict HO performance over a wide range of image-reconstruction parameter settings for which HO data are not available. Thus, the NO must exhibit good generalization properties.

In this comparison, we studied a kind of train-test generalization that is the most representative of the practical use of a NO. In this experiment, after tuning of the internal-noise model parameters for a broad set of images, the NO was then tested on a different, but equally broad, set of images.

Specifically we tuned the internal-noise model parameters using data for every value of the filter FWHM, and one iteration of OSEM. In tuning, the parameters were adjusted by exhaustive search so as to minimize the average MSE between the six AUC values of the HOs and six AUC values of the CHO AUC, thus maximizing agreement between human and model observer. The internal-noise parameters thus obtained were then applied to analyze the remaining data; that is, every value of the filter FWHM and five iterations of OSEM, yielding six AUC values. Next, the roles of one and five iterations were reversed, yielding an additional six AUC values. Next, these 12 values were compared to that of the HOs, and average values of MSE and Ktau are reported.

Recall that K_ext and Δf are replaced by sample estimates of these quantities, $\tilde{\bf K}_{{\rm ext}}$ and $\Delta \tilde {\bf f}$, and re-estimated separately for each of the 12 reconstruction methods, and are not the part of generalization evaluation.

To indicate results obtained as part of comparison 2, the names of the metrics are prefixed by the letter 'G' (short for 'generalization'; i.e., G-MSE and G-Ktau). The reported values are averaged over six FWHM values.

MPS is calculated as a ratio of the model parameters optimized in comparison 2. In cases where the internal-noise model has two parameters, the averaged ratio, over two model parameters, is reported.

Finally an average PCC value is reported. The first component in this average is the average PCC over six image-domain spatial templates, w_f, corresponding to different FWHM values, obtained using images reconstructed by five iteration of OSEM. The second component is the average PCC calculated between image templates in iteration one and five. Finally these two numbers are averaged and reported.

Tables 1–4 show, for each channel model (CHO-BP, CHO-GB, CHO-LG and CHO-DOG), the average fitting error and rank correlation (F-MSE and F-Ktau) followed by average generalization error and rank correlation (G-MSE and G-Ktau) and MPS. These numbers are followed average PCC. These evaluations uniformly demonstrate that looking simply at fitting accuracy, measured by F-MSE or F-Ktau, leads to a misleading conclusion as to which model performs the best. In each case, the other criteria (G-MSE and G-Ktau, MPS and PCC), which emphasize generalization performance (which we argue is a necessary attribute of a NO), consistently point to the superiority of a different model than that suggested by the fitting-based performance measures.

The results show consistently that the fitting-based metrics lead to conclusions about which NO is best that are not supported when the NO is tested on new data (to measure generalization performance). Because so many such comparisons are made, and these comparisons consistently demonstrate the same point, we only discuss one example comparison to illustrate the theme. The full results are shown in tables 1–4.

To choose an example, let us consider CHO evaluation when using bandpass filters (as in figure 1(a)) for the channeling operator. In this case, models 7, 8 and 10 show excellent fitting accuracy as measured by F-MSE (table 1), and model 6 shows good performance as measured

by F-Ktau; however, models 5, 6, 7, 9 and 10 prove to have much better generalization performance as measured by G-MSE, while only model 6 has good G-Ktau performance. This example shows clearly that testing on the same type of images (i.e., evaluation by F-MSE), may be misleading, and that model testing should be performed on a distinct data set, using metrics such as G-MSE and G-Ktau. Table 1 also summarizes the stability of the internal-noise model parameter—MPS. Here MPS represents the average ratio of the internal-noise parameters, which ideally should equal 1 if the results are repeatable (which would be desirable). This ratio indicates that models 2, 6 and 10 produce stable parameters.

Finally, let us consider the image-domain templates, w_f, for each internal-noise model, shown in figure 4. First, let us examine the noiseless difference images, $\Delta {\bf \bar f} = {\bf \bar f}_1 - {\bf \bar f}_2$, which appear on the left in figures 4 and 1(e). Note that $\Delta {\bf \bar f}$ looks similar for images obtained by different reconstruction methods. This similarity was measured quantitatively by PCC computed in two ways: 1) as the average PCC for all difference images within images reconstructed by five iteration of OSEM (result is 0.892); 2) as the average PCC between corresponding difference images in iteration 1 and iteration 5 (result is 0.956) yielding an total average of 0.9240. Therefore, it is reasonable to expect that image-domain spatial templates w_f should somewhat resemble $\Delta {\bf \bar f}$ templates or at least have similar PCC of 0.9240. The average PCC values are shown at the bottom of table 1. Here we can conclude that models 3, 5 and 6 have similar PCC as that of $\Delta {\bf \bar f}$ templates. Moreover these templates resemble the difference image, $\Delta {\bf \bar f}$.

In conclusion, the best CHO-BP noise model overall is model 6 since it produces accurate results as measured by G-MSE and G-Ktau, and proves to be a stable model, as measured by MPS and PCC.

To avoid repetition, rather than fully explain every comparison, we will only summarize the key findings for the other channel models; however, the reasoning in each case proceeds similarly to the preceding discussion for the case of BP filters.

Zoom In Zoom Out Reset image size

CHO-GB. Fitting measures F-MSE and F-Ktau would misleadingly identify model 11 as a good one; however, after considering G-MSE, G-Ktau, MPS and PCC, model 6 emerges as the best, followed by model 7.

CHO-LG. F-MSE and F-Ktau would misleadingly identify model 11 as a good one; however, G-MSE, G-Ktau, MPS and PCC show that model 6 is best, followed by model 5.

CHO-DOG. F-MSE and F-Ktau would identify correctly model 6, but also suggests that models 4 and 9 as good candidates; however, after considering G-MSE, G-Ktau, MPS and PCC, model 6 is found to be the best model, followed by model 4.

The initial aim of this work was not to propose a new internal-noise model, but simply to present possible criteria for evaluation of models and a principled approach to the train-test scheme. However, this study also led us to a new internal-noise model, which turned out to provide the best results in terms of generalization accuracy and stability. In this model the variances of the injected channel noise are proportional to the standard deviations of the external noise. This model proved consistently to be among the best models for all four channeling operators; therefore, as a final comparison, we present results in figure 8, showing AUC curves for the optimized model 6 and all four channeling operators. Error bars represent plus or minus one standard deviation. Generalization performance is shown in both figures; that is, the models are tested on data reconstructed in a different way than the data used for training.

This comparison shows similarly good performance of CHO-BP, CHO-GB and CHO-DOG, which is expected after examining the estimated image-domain templates in figures 4, 5 and 7. The performance of CHO-DOG is slightly better in terms of generalization G-MSE and G-Ktau (see tables). The performance of CHO-LG does not seem to be as good as that of the other three methods, as one would expect by examining the estimated templates in figure 6.

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size

Zoom In Zoom Out Reset image size