An Analytic Expression for the Binormal Partial Area under the ROC Curve

doi:10.1016/j.acra.2012.09.009

Academic Radiology

Volume 19, Issue 12, December 2012, Pages 1491-1498

https://doi.org/10.1016/j.acra.2012.09.009 Get rights and content

Rationale and Objectives

The partial area under the receiver operating characteristic (ROC) curve (pAUC) is a useful summary measure for diagnostic studies. Unlike most summary measures that are functions of the ROC curve, researchers have not been aware of an analytic expression available for computing the pAUC for an ROC curve based on a latent binormal model. Instead, the pAUC has been computed using numerical integration or a rational polynomial approximation. Our purpose is to provide analytic expressions for the two forms of pAUC.

Materials and Methods

We discuss the two fundamentally different types of pAUC. We present analytic expressions for both types, provide corresponding proofs, and illustrate their application with an example comparing the performances of spin echo and cine magnetic resonance imaging for detecting thoracic aortic dissection.

Results

We provide an example of using the pAUC as the outcome in a multireader multicase analysis. We find that using the pAUC results in a more significant finding.

Conclusions

We have provided analytic expressions for both types of pAUC, making it easier to compute the pAUCs corresponding to binormal ROC curves.

Introduction

In diagnostic radiology, receiver operating characteristic (ROC) curves are commonly used to quantify the accuracy with which a reader (typically a radiologist) can discriminate between images from nondiseased (or normal) and diseased (or abnormal) cases. Although the ROC curve concisely describes the tradeoffs between sensitivity and specificity, typically accuracy is summarized by a summary index that is a function of the ROC curve. Commonly used summary indices include the area under the ROC curve (AUC), the partial area under the ROC curve (pAUC), sensitivity for a given specificity, and specificity for a given sensitivity. See Zou et al (1) for a concise introduction to ROC analysis.

A common method for estimating the ROC curve is likelihood estimation under the assumption of a latent binormal model 2, 3, 4, 5; alternatively, a generalized linear model approach can also be used 6, 7 based on the binormal model assumption. Under the latent binormal model assumption the ROC curve can be described by two parameters. Except for the pAUC, analytic expressions have been routinely employed for expressing the indices previously mentioned as a function of the binormal ROC curve parameters. It is generally believed that the pAUC, assuming a latent binormal model, cannot be expressed as an analytic expression. For example, Pepe (8) states: “Unfortunately, a simple analytic expression does not exist for the pAUC summary measure. It must be calculated using numerical integration or a rational polynomial approximation.” Similarly, Zhou et al (9) state: “This partial area as it is known, is evaluated by numerical integration (McClish, 1989).” Although these methods can be programmed, having a simple expression for the pAUC would be much more convenient.

It is generally not known that Pan and Metz (10) provided analytic expressions for the two forms of pAUC. However, the expressions they provided were incorrect and they did not provide proofs for their results. More importantly, it is generally not known that Thompson and Zucchini (11) provided a correct analytic expression for one form of pAUC as well as the proof. In fact, we only became aware of this latter result during the final stage of submitting this article. The purpose of this article is to bring to the attention of the reader the result provided by Thompson and Zucchini, extend their result to the second form of pAUC, and illustrate use of both pAUC expressions with a real data set that compares the relative performance of single spin-echo magnetic resonance imaging (SE MRI) to cinematic presentation of MRI (CINE MRI) for the detection of thoracic aortic dissection. In addition, we provide proofs for both results that are more accessible to radiology researchers and clinicians than the proof given by Thomas and Zucchini.

Section snippets

Two Different pAUCs

Let FPF and TPF denote false- and true-positive fractions for a given classification threshold such that an image with a test result equal or greater than the threshold is classified as diseased, and otherwise nondiseased. That is, FPF is the probability that a test result for a nondiseased subject exceeds the threshold and TPF is the probability that a test result for a diseased subject exceeds the threshold. The ROC curve is a plot of TPF versus FPF for all possible thresholds. FPF and TPF

Results

The ROC curves computed for the example data set are presented in Figure 2. Table 1 presents the corresponding binormal parameter estimates for a and b and estimates and standard errors for AUC, ${pAUC}_{FPF}$ for FPF intervals (0.0, 0.2) and (0.0, 0.1), and ${pAUC}_{TPF}$ for TPF intervals (0.8, 1.0) and (0.9, 1.0). The pAUCs have been normalized by dividing by the length of the defining interval; thus, the pAUC values represent average sensitivity or specificity over the corresponding defining FPF or TPF

Discussion

For the two types of pAUCs, we derived analytic expressions under the assumption of a latent binormal model. Previously it was believed that analytic expressions did not exist, even though Thompson and Zucchini (11) had stated and proved Equation 2, and thus numerical methods have been used to solve for pAUC values. The formulas presented in this article greatly simplify computation of pAUCs.

We illustrated use of these expressions with a real data set in which, using a multireader analysis, we

Acknowledgments

I thank Carolyn Van Dyke, MD, for sharing her data set for the example. I thank the reviewers for helpful suggestions that clarified the presentation.

References (28)

D.D. Dorfman et al.
Maximum likelihood estimation of parameters of signal-detection theory and determination of confidence intervals—rating method data
J Math Psychol
(1969)
D.D. Dorfman et al.
Degeneracy and discrete receiver operating characteristic rating data
Acad Radiol
(1995)
X.C. Pan et al.
The “proper” binormal model: parametric receiver operating characteristic curve estimation with degenerate data
Acad Radiol
(1997)
D.D. Dorfman et al.
Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design
Acad Radiol
(1998)
S.L. Hillis et al.
Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis
Acad Radiol
(2008)
K.H. Zou et al.
Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models
Circulation
(2007)
D.D. Dorfman
RSCORE II
C.E. Metz et al.
Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data
Stat Med
(1998)
M.S. Pepe
An interpretation for the ROC curve and inference using GLM procedures
Biometrics
(2000)
T.A. Alonzo et al.
Distribution-free ROC analysis using binary regression techniques
Biostatistics
(2002)

M. Pepe

The statistical evaluation of medical tests for classification and prediction

(2003)

X.-H. Zhou et al.

Statistical methods in diagnostic medicine

(2011)

M.L. Thompson et al.

On the statistical analysis of ROC curves

Stat Med

(1989)

D.K. McClish

Analyzing a portion of the ROC curve

Med Decision Making

(1989)

Cited by (14)

Diagnostic accuracy of technologies for glaucoma case-finding in a community setting
2015, Ophthalmology
Citation Excerpt :
Initial diagnostic accuracy estimates of each index test to detect glaucoma suspect/definite POAG combined and definite POAG were evaluated using the predefined cutoffs for abnormality to generate sensitivity, specificity, and likelihood ratios with 95% confidence intervals (CIs). To compare index test performance within a clinically relevant range for detection of a low-prevalence disease, we determined the sensitivity at 90% specificity and normalized the partial area under the receiver operating characteristic curves (AUROC) to determine the average sensitivity27 between 90% and 100% specificity. To test for any statistically significant differences between sensitivity at a set specificity and partial AUROC curve estimates, the Wald test was used.28
To assess case-finding performance of the Frequency Doubling Technology Perimeter (FDT) (Carl Zeiss Meditec, Inc., Dublin, CA), Moorfields Motion Displacement Test (MMDT) (Moorfields Eye Hospital, London, UK), iVue optical coherence tomography (OCT) (Optovue Inc., Fremont, CA), and ocular response analyzer (ORA) (Reichert Ophthalmic Instruments, Depew, NY), alone or combined, for primary open-angle glaucoma (POAG).
Cross-sectional, observational, community-based study.
A total of 505 subjects aged ≥60 years recruited from a community setting using no predefined exclusion criteria.
Subjects underwent 4 index tests conducted by a technician unaware of subjects’ ocular status. FDT and MMDT were used in suprathreshold mode. iVue OCT measured ganglion cell complex and retinal nerve fiber layer (RNFL) thickness. Reference standard was full ophthalmic examination by an experienced clinician who was masked to index test results. Subjects were classified as POAG (open drainage angle, glaucomatous optic neuropathy, and glaucomatous field defect), glaucoma suspect, ocular hypertension, or non-POAG/nonocular hypertension.
Test performance evaluated the individual as the unit of analysis. Diagnostic accuracy was assessed using predefined cutoffs for abnormality, generating sensitivity, specificity, and likelihood ratios. Continuous data were used to derive estimates of sensitivity at 90% specificity and partial area under the receiver operating characteristic curve (AUROC) plots from 90% to 100% specificity.
From the reference standard examination, 26 subjects (5.1%) had POAG and 32 subjects (6.4%) were glaucoma suspects. Sensitivity (95% confidence interval) at 90% specificity for detection of glaucoma suspect/POAG combined was 41% (28–55) for FDT, 35% (21–48) for MMDT, and 57% (44–70) for best-performing OCT parameter (inferior quadrant RNFL thickness); for POAG, sensitivity was 62% (39–84) for FDT, 58% (37–78) for MMDT, and 83% (68–98) for inferior quadrant RNFL thickness. Partial AUROC was significantly greater for inferior RNFL thickness than visual-function tests (P < 0.001). Post-test probability of glaucoma suspect/POAG combined and definite POAG increased substantially when best-performing criteria were combined for FDT or MMDT, iVue OCT, and ORA.
Diagnostic performance of individual tests gave acceptable accuracy for POAG detection. Low specificity of visual-function tests precludes their use in isolation, but case detection improves by combining RNFL thickness analysis with visual function tests.
Multivariate normally distributed biomarkers subject to limits of detection and receiver operating characteristic curve inference
2013, Academic Radiology
Biomarkers are of ever-increasing importance to clinical practice and epidemiologic research. Multiple biomarkers are often measured per patient. Measurement of true biomarker levels is limited by laboratory precision, specifically measuring relatively low, or high, biomarker levels resulting in undetectable levels below, or above, a limit of detection (LOD). Ignoring these missing observations or replacing them with a constant are methods commonly used although they have been shown to lead to biased estimates of several parameters of interest, including the area under the receiver operating characteristic (ROC) curve and regression coefficients.
We developed asymptotically consistent, efficient estimators, via maximum likelihood techniques, for the mean vector and covariance matrix of multivariate normally distributed biomarkers affected by LOD. We also developed an approximation for the Fisher information and covariance matrix for our maximum likelihood estimations (MLEs). We apply these results to an ROC curve setting, generating an MLE for the area under the curve for the best linear combination of multiple biomarkers and accompanying confidence interval.
Point and confidence interval estimates are scrutinized by simulation study, with bias and root mean square error and coverage probability, respectively, displaying behavior consistent with MLEs. An example using three polychlorinated biphenyls to classify women with and without endometriosis illustrates how the underlying distribution of multiple biomarkers with LOD can be assessed and display increased discriminatory ability over naïve methods.
Properly addressing LODs can lead to optimal biomarker combinations with increased discriminatory ability that may have been ignored because of measurement obstacles.
Receiver operating characteristic analysis: Utility, reality, covariates, and the future
2013, Academic Radiology
Mixtures of receiver operating characteristic curves
2013, Academic Radiology
Citation Excerpt :
Therefore, estimation of the AUC for the p-mixture ROC curve requires only the estimation of the AUCs of its component ROC curves. There is no closed form expression for the pAUC of a binormal curve (26); however, it can be written as a function of the cumulative density function of the standard bivariate normal (27). It follows that there is no closed form expression for the pAUC of the mixture ROC curve either, but this is not a burden: an estimate of pAUC(p) can be obtained by numerically integrating the component ROC curves or using the bivariate formulation (27).
Receiver operating characteristic (ROC) curves are ubiquitous in the analysis of imaging metrics as markers of both diagnosis and prognosis. While empirical estimation of ROC curves remains the most popular method, there are several reasons to consider smooth estimates based on a parametric model.
A mixture model is considered for modeling the distribution of the marker in the diseased population motivated by the biological observation that there is more heterogeneity in the diseased population than there is in the normal one. It is shown that this model results in an analytically tractable ROC curve which is itself a mixture of ROC curves.
The use of creatine kinase–BB isoenzyme in diagnosis of severe head trauma is used as an example. ROC curves are fit using the direct binormal method, ROCKIT software, and the Box-Cox transformation as well as the proposed mixture model. The mixture model generates an ROC curve that is much closer to the empirical one than the other methods considered.
Mixtures of ROC curves can be helpful in fitting smooth ROC curves in datasets where the diseased population has higher variability than can be explained by a single distribution.
An additive selection of markers to improve diagnostic accuracy based on a discriminatory measure
2013, Academic Radiology
Citation Excerpt :
Following the proposed selection procedure, a subset of algorithms can be selected and measurements from these algorithms can be combined to generate a composite marker which tends to have better accuracy than an individual algorithm. As one referee points out, our proposed algorithm can be easily extended to marker selection in terms of the partial AUC (pAUC) (30) and/or multireader multicase (MRMC) ROC. In the first case, we can provide clinically relevant values of test sensitivity or specificity and replace full AUC in discriminatory score of the marker by the pAUC.
The estimation of the area under the receiver operating characteristic (ROC) curve (AUC) often relies on the assumption that the truly positive population tends to have higher marker results than the truly negative population. The authors propose a discriminatory measure to relax such an assumption and apply the measure to identify the appropriate set of markers for combination.
The proposed measure is based on the maximum of the AUC and 1-AUC. The existing methods are applied to estimate the measure. The subset of markers is selected using a combination method that maximizes a function of the proposed discriminatory score with the number of markers as a penalty in the function.
The properties of the estimators for the proposed measure were studied through large-scale simulation studies. The application was illustrated through a real example to identify the set of markers to combine.
Simulation results showed excellent small-sample performance of the estimators for the proposed measure. The application in the example yielded a reasonable subset of markers for combination.
Sampling the Latest Work in Receiver Operating Characteristic Analysis. What Does It Mean?
2012, Academic Radiology

View all citing articles on Scopus

: Stephen Hillis was supported by the National Institute of Biomedical Imaging and Bioengineering grants R01EB000863 and R01EB013667.

^∗: Deceased.

View full text

Receiver operating characteristic analysisAn Analytic Expression for the Binormal Partial Area under the ROC Curve

Rationale and Objectives

Materials and Methods

Results

Conclusions

Introduction

Section snippets

Two Different pAUCs

Results

Discussion

Acknowledgments

J Math Psychol

Acad Radiol

Acad Radiol

Acad Radiol

Acad Radiol

Receiver-operating characteristic analysis for evaluating diagnostic tests and predictive models

Circulation

RSCORE II

Maximum likelihood estimation of receiver operating characteristic (ROC) curves from continuously-distributed data

Stat Med

An interpretation for the ROC curve and inference using GLM procedures

Biometrics

Distribution-free ROC analysis using binary regression techniques

Biostatistics

The statistical evaluation of medical tests for classification and prediction

Statistical methods in diagnostic medicine

On the statistical analysis of ROC curves

Stat Med

Analyzing a portion of the ROC curve

Med Decision Making

Receiver operating characteristic analysis
An Analytic Expression for the Binormal Partial Area under the ROC Curve