The appropriateness of Bland-Altman’s approximate confidence intervals for limits of agreement

A percentile is a numerical measure that represents the reference point below which a given percentage of values in the target population fall. Because of the conceptual simplicity and context-free feature, percentiles are widely used for determining the relative magnitude and substantial importance of quantitative measurements in all scientific fields. For example, the children health conditions are often assessed by their weight and height in comparison to the national averages and percentiles found in the growth charts. Also, reference limits are extensively applied in medicine and related fields to identify informative range of measurement from a reference population. The most typical reference limits contain the central 95% of the values in the population of interest. As an important application, the Bland and Altman [1, 2] 95% limits of agreement are comprised of the 2.5th percentile and 97.5th percentile for the distribution of the difference between paired measurements.

The practical usage of percentiles is often represented by referring to a normal distribution. In this prominent case, the normal percentile is a linear function of the mean and standard deviation of the designated population. Note that the sample mean and sample variance are complete and sufficient statistics for the population mean and variance. Although estimation of normal percentile is not discussed in most standard texts, it is straightforward to obtain the minimum variance unbiased estimator of a normal percentile. However, the dominance property does not extend to other principles in decision theoretic analyses such as the mean square error criterion. Among others, Royston and Mathews [3] conducted a comparison of potential point estimators of normal percentiles with respect to bias and mean square error. More advanced and theoretical investigations of normal percentile estimators can be found in Keating, Mason, and Balakrishnan [4], Keating and Tripathi [5], Parrish [6], Rukhin [7], and Zidek [8, 9].

In view of the stochastic nature in statistical inference, it is more informative to construct confidence intervals for the target parameters than to provide a single estimate about their values. General expositions and comprehensive guidelines of interval estimation are available in Hahn [10, 11], Hahn and Meeker [12], and Vardeman [13]. Accordingly, various interval methods of normal percentiles have been described from different perspectives. The exact interval procedure of normal percentiles has been documented in the literature, for example, see Hahn and Meeker [12], Johnson, Kotz, and Balakrishnan [14], and Owen [15]. Moreover, the one-sided confidence intervals of normal percentiles have a close link to the one-sided tolerance bounds of a normal distribution as noted in David and Nagaraja [16], Krishnamoorthy and Mathew [17], and Odeh and Owen [18].

Notably, Bland and Altman [1, 2] suggested the 95% limits of agreement for evaluating the differences between measurements by two methods. The endpoints of the Bland-Altman 95% limits of agreement are the 2.5th percentile and 97.5th percentile for the distribution of the difference between paired measurements. To reflect the uncertainty due to sampling error, approximate interval formulas were presented for estimating the two individual percentiles. The large number of citations revealed that the Bland-Altman analysis has become the major technique for assessing agreement between two methods of clinical measurement. But the recent work of Carkeet [19] and Carkeet and Goh [20] provided detailed discussions in favor of exact confidence interval over the approximate procedure considered in Bland and Altman [1, 2], especially when the sample sizes are small. Further considerations and reviews of measuring agreement in method comparison studies are available in Barnhart, Haber, and Lin [21], Choudhary and Nagaraja [22], and Lin et al. [23].

Although the practical implementation of the exact interval procedure is well presented in Carkeet [19], the explication of the differences between the exact and approximate methods mainly concentrated on the relative magnitudes and symmetric/asymmetric bounds of the resulting confidence limits. On the other hand, the endpoints of the Bland-Altman 95% limits of agreement are usually viewed as a pair of bound for measuring agreement in method comparison studies. Accordingly, Carkeet [19] and Carkeet and Goh [20] focused on the comparison of the approximate confidence intervals for upper and lower limits of agreements as a pair and the exact two-sided tolerance intervals for a normal distribution. Therefore, the distinctive advantage of the exact interval procedures and the potential limitation of the approximate confidence intervals for the individual upper and lower limits of agreement were not fully addressed in Carkeet [19] and Carkeet and Goh [20]. It is of practical importance to conduct a detailed appraisal of the accuracy and discrepancy between the exact and approximate interval procedures for an individual limit of agreement under a wide range of model configurations. The problem of obtaining a single confidence interval to cover both limits of agreement simultaneously is more involved and a detailed discussion of this topic is beyond the scope of the present study.

In addition to the abovementioned studies, a numerical comparison of several interval estimation methods of normal percentiles was presented in Chakraborti and Li [24]. They adopted a standardized minimum variance unbiased estimator as the pivotal quantity and proposed both exact and approximate confidence intervals of normal percentiles. Their simulation study showed that the expected width and coverage probability of the suggested exact and approximate methods are nearly identical to that of the procedure described in Lawless ([25], p. 231). Despite the analytic arguments and empirical findings in Chakraborti and Li [24], the following two attentions toward their illustration should be noted. First, although it was demonstrated that Lawless’s [25] confidence intervals are the same as the existing formulas in Owen [15] and Odeh and Owen [18], they did not discuss the theoretical implications between their exact method and the established exact procedure. Second, in contrast to the asymmetry of the exact confidence intervals, the approximate confidence intervals of Chakraborti and Li [24] are equidistant around the minimum variance unbiased estimate. Note that the two endpoints of a two-sided confidence interval can also be interpreted as the limits of one-sided confidence interval. Thus, the performance of the two limits of Chakraborti and Li’s [24] approximate interval method should be further evaluated with respect to the equal-tailed property. The analytic and numerical results in Chakraborti and Li [24] are not detailed enough to clarify these fundamental issues. It is prudent to elucidate these vital aspects of their methods to be accepted as a feasible technique.

To enhance the adoption of appropriate techniques for interval estimation and research design, this paper has two objectives. The first is to appraise the statistical features of interval estimation procedures for normal percentiles. Theoretical justifications are presented to illuminate the statistical connections between different pivotal quantities for obtaining exact confidence intervals. Furthermore, comprehensive empirical assessments are provided to show the seemingly accurate approximate methods with equidistant around the principal estimators have problematic confidence limits. The second goal is to provide sample size procedures for precise interval estimation of normal percentiles. The required precision of a confidence interval is evaluated with the magnitude of expected width, and the assurance probability of interval width within a designated threshold. In view of the general availability of statistical software packages SAS and R, computer algorithms are developed to facilitate the implementation of the suggested confidence interval and sample size computations.

Assume X₁, …, X _N are a sample from a N(μ, σ²) population with unknown mean μ and variance σ² for N > 1. The sample mean \( \overline{X} \) and sample variance S² are defined as \( \overline{X}=\sum \limits_{i=1}^N{X}_i/N \) and \( {S}^2=\sum \limits_{i=1}^N{\left({X}_i-\overline{X}\right)}^2/\left(N-1\right) \), respectively. The 100pth percentile of the distribution N(μ, σ²) is denoted by θ, whereand z _p is the 100pth percentile of the standard normal distribution N(0, 1). To estimate the percentile θ, the intuitive formulais a biased estimator because E[S] < σ. As noted in Royston and Mathews [3], the minimum variance unbiased estimator iswhere c = (ν/2)^1/2Γ(ν/2)/Γ{(ν + 1)/2} and ν = N – 1. Note that c is an adjusting factor so that cS is an unbiased estimator of σ or E[cS] = σ. Moreover, it can be shown that the variance and mean square error of the two estimators are Var[\( \widehat{\uptheta} \) _B ] = {1 + N\( {z}_p^2 \)(1–1/c²)}(σ²/N), MSE[\( \widehat{\uptheta} \) _B ] = {1 + 2N\( {z}_p^2 \)(1–1/c)}(σ²/N), and Var[\( \widehat{\uptheta} \) _MU ] = MSE[\( \widehat{\uptheta} \) _MU ] = {1 + N\( {z}_p^2 \)(c²–1)}(σ²/N). Because c is slightly larger than 1 for N > 1, further examinations assure the contrasting dominance phenomena: Var[\( \widehat{\uptheta} \) _MU ] > Var[\( \widehat{\uptheta} \) _B ] and MSE[\( \widehat{\uptheta} \) _MU ] > MSE[\( \widehat{\uptheta} \) _B ]. The relative numerical performance of \( \widehat{\uptheta} \) _B , \( \widehat{\uptheta} \) _MU , and alternative estimators of θ can also be found in Royston and Mathews [3].

To obtain confidence intervals for θ, standard derivations show thatwhere t(ν, –z _p N^1/2) is a noncentral t distribution with degrees of freedom ν and noncentrality parameter –z _p N^1/2 (Johnson, Kotz, & Balakrishnan [14], Chapter 31). Accordingly, T* yields a pivotal quantity for constructing confidence intervals of normal percentiles. An upper 100(1 – α)% one-sided confidence interval of θ is expressed as {\( \widehat{\uptheta} \) _L , ∞} and the lower confidence limit iswhere t_1 − α(ν, –z _p N^1/2) is the 100(1 – α)th percentile of the distribution t(ν, –z _p N^1/2) and t_1 − α(ν, –z _p N^1/2) = −t_α(ν, z _p N^1/2) for 0 < α < 1. Also, a lower 100(1 – α)% one-sided confidence interval of θ is {−∞, \( \widehat{\uptheta} \) _U } and the upper confidence limit has the form

Furthermore, a 100(1 – α)% two-sided confidence interval of θ with equal tail probability can be readily obtained as {\( \widehat{\uptheta} \) _L , \( \widehat{\uptheta} \) _U } where

and

Supplementary SAS/IML and R computer programs are provided to take advantage of the embedded statistical functions for calculating the exact confidence intervals.

In addition, it may be more appealing to modify the point estimators \( \widehat{\uptheta} \) _B and \( \widehat{\uptheta} \) _MU to acquire the alternative pivotal quantitiesfor deriving the confidence intervals of θ, respectively. It is easy to see that T _B  = T* + z _p N^1/2 and T _MU  = T* + z _p cN^1/2. Therefore, T _B and T _MU differ from T* only in the location shift. Because the terms z _p N^1/2 and z _p cN^1/2 do not depend on the unknown parameters, T _B and T _MU give the same one- and two-sided confidence intervals for θ described in Eqs. 5–7. As a generalization of the simple location shifts between different pivotal quantities, the prescribed application of pivotal quantity for exact interval estimation extends to any linear function of T*. For example, Lawless [25] constructed the confidence intervals of normal percentiles through the quantity

Evidently, T _L can be expressed as a linear transformation of T* by T _L  = (T* + z _p N^1/2)/N^1/2. Assume q_{L, 1 − α} is the 100(1 – α)th percentile of T _L , it is readily established that q_{L, 1 − α} = {t_1 − α(v, −z _p N^1/2) + z _p N^1/2}/N^1/2. Although the result in Lawless ([25], p. 231) is written in a different form, the quantity T _L also leads to the same exact confidence interval {\( \widehat{\uptheta} \) _L , \( \widehat{\uptheta} \) _U } for θ.

On the other hand, Chakraborti and Li [24] considered the standardized quantityfor interval estimation of θ, where a = 1 + N\( {z}_p^2 \)(c²–1). Their method relies on direct computations with the derived probability density function and cumulative distribution function of T _ST . Therefore, a special purpose algorithm is required to compute the quantiles of T _ST and to obtain the suggested confidence intervals of θ. Note that T _ST is a linear function of T* in terms of T _ST  = (T* + z _p cN^1/2)/a^1/2. Hence, if q_{ST, 1 − α} denotes the 100(1 – α)th percentile of T _ST , it has the identical linear transform with the 100(1 – α)th percentile of T* or q_{ST, 1 − α} = {t_1 − α(v, −z _p N^1/2) + z _p cN^1/2}/a^1/2. As noted earlier, the actual value t_1 − α(ν, –z _p N^1/2) can be obtained with the cumulative distribution function of a noncentral t distribution in major statistical packages such as SAS and R. Hence with the general availability of software systems and the underlying linear relationship between T _ST and T*, direct calculation is not required to compute the percentile q_{ST, 1 − α}. More importantly, using the standard pivotal procedure and the prescribed linear transformation of T*, the pivotal quantity T _ST leads to the same interval estimators of θ with T* and the other three pivotal measures T _B , T _MU , and T _L . Although the pivotal quantity T _L was also examined in Chakraborti and Li [24], the resulting interval estimators of T _L and T _ST are viewed as two distinct procedures. However, the numerical assessments in Chakraborti and Li [24] reported that the performances of the two interval procedures of T _L and T _ST are almost identical. The important connections between the pivotal quantities and the resulting confidence intervals of θ should be properly recognized. Essentially, the prescribed explication illuminates the conceptual equivalence between the five pivotal quantities T^*, T _B , T _MU , T _L , and T _ST for constructing confidence intervals of θ.

Along with the exact confidence interval procedure of normal percentiles, Chakraborti and Li [24] also described an approximate interval estimator by assuming T _ST has a t distribution with degrees of freedom ν:

Thus, an approximate 100(1 – α)% two-sided equal tail confidence interval {\( \widehat{\uptheta} \) _AL , \( \widehat{\uptheta} \) _AU } of θ is immediately constructed asandwhere τ _AL  = z _p cN^1/2 – t_{1 − α/2}(ν)a^1/2, τ _AU  = z _p cN^1/2 + t_{1 − α/2}(ν)a^1/2, and t_{1 − α/2}(ν) is the 100(1 – α/2)th percentile of the distribution t(ν). Although the two-sided confidence interval is only an approximation, the simulation study of Chakraborti and Li [24] revealed that {\( \widehat{\uptheta} \) _AL , \( \widehat{\uptheta} \) _AU } is very competitive with the exact interval estimator {\( \widehat{\uptheta} \) _L , \( \widehat{\uptheta} \) _U } with respect to the coverage probability and interval width.

On the other hand, to construct confidence intervals of limits of agreement or percentiles, Bland and Altman [2] argued that Var[S] ≐ σ²/(2ν) and Var[\( \widehat{\uptheta} \) _B ] ≐ bσ²/N where \( b=1+{z}_p^2/2 \). With the approximation, they suggested the simplified pivotal quantity

Accordingly, the widely used confidence intervals of Bland and Altman [2] can be derived from T _BA and they are written as {\( \widehat{\uptheta} \) _BAL , \( \widehat{\uptheta} \) _BAU } whereandwith τ _BAL  = z _p N^1/2 – t_{1 − α/2}(ν)b^1/2 and τ _BAU  = z _p N^1/2 + t_{1 − α/2}(ν)b^1/2. For the particular case of α = 0.05, the general expressions reduce to the confidence intervals for the two endpoints of the 95% limits of agreement considered in Bland and Altman [2]:andrespectively, because z_0.025 = − 1.96, z_0.975 = 1.96, and b = 2.92.

For the blood pressure data presented in Bland and Altman [2] with the sample size N = 85, the sample mean difference (observer minus machine) \( \overline{X} \) = − 16.29 mmHg, and the standard deviation of the differences S = 19.61, the 95% confidence intervals of the exact and two approximate methods for the 2.5th percentile are {\( \widehat{\uptheta} \) _L , \( \widehat{\uptheta} \) _U } = {− 62.9501, − 48.3770}, {\( \widehat{\uptheta} \) _AL , \( \widehat{\uptheta} \) _AU } = {− 62.1035, − 47.5754}, and {\( \widehat{\uptheta} \) _BAL and \( \widehat{\uptheta} \) _BAU } = {− 61.9536, − 47.4961}, respectively. For the interval estimation of the 97.5th percentile, the resulting exact and two approximate 95% confidence intervals are {\( \widehat{\uptheta} \) _L , \( \widehat{\uptheta} \) _U } = {15.7970, 30.3701}, {\( \widehat{\uptheta} \) _AL , \( \widehat{\uptheta} \) _AU } = {14.9954, 29.5235}, and {\( \widehat{\uptheta} \) _BAL , \( \widehat{\uptheta} \) _BAU } = {14.9161, 29.3736}, respectively. Although the differences between these estimates may not be substantial, it is vital to point out that the confidence limits of the 2.5th percentile are in the ascending order of \( \widehat{\uptheta} \) _L < \( \widehat{\uptheta} \) _AL < \( \widehat{\uptheta} \) _BAL and \( \widehat{\uptheta} \) _U < \( \widehat{\uptheta} \) _AU < \( \widehat{\uptheta} \) _BAU . Whereas the confidence limits of the 97.5th percentile have a reversed situation: \( \widehat{\uptheta} \) _BAL < \( \widehat{\uptheta} \) _AL < \( \widehat{\uptheta} \) _L and \( \widehat{\uptheta} \) _BAU < \( \widehat{\uptheta} \) _AU < \( \widehat{\uptheta} \) _U . This inherent relationship between the three interval procedures is further justified as the usual occurrence in the simulation study.

In general, the actual distribution of the pivotal quantity T* is skewed, especially when sample size is small and p deviates considerably from 0.5. This implies that the interval procedure should adopt asymmetric confidence intervals for θ. Notably, the exact two-sided interval estimates {\( \widehat{\uptheta} \) _L , \( \widehat{\uptheta} \) _U } are not equidistant from the sample mean except for the special case p = 0.5. In contrast, the approximate confidence intervals {\( \widehat{\uptheta} \) _AL , \( \widehat{\uptheta} \) _AU } of Chakraborti and Li [24] is equidistant about the unbiased estimate \( \widehat{\uptheta} \) _UB . Therefore, the interval procedure is presumably inappropriate and the two confidence limits \( \widehat{\uptheta} \) _AL and \( \widehat{\uptheta} \) _AU are methodologically inaccurate when one-sided coverage probabilities are considered. But the numerical investigations in Chakraborti and Li [24] did not cover these fundamental issues. Similarly, the confidence intervals {\( \widehat{\uptheta} \) _BAL , \( \widehat{\uptheta} \) _BAU } of Bland and Altman [2] are symmetric around the estimate \( \widehat{\uptheta} \) _B and thus also suffer the same shortcoming as the intervals {\( \widehat{\uptheta} \) _AL , \( \widehat{\uptheta} \) _AU } of Chakraborti and Li [24].

Note that the lower and upper confidence limits of a 100(1 – α)% two-sided confidence interval are equivalent to the lower and upper confidence limits of the 100(1 – α/2)% one-sided upper and lower confidence intervals, respectively. To demonstrate the potential drawback of the approximate interval procedures of Chakraborti and Li [24] and Bland and Altman [2], a simulation study was conducted to evaluate the coverage performance of their one- and two-sided confidence intervals. Although the approximate interval method of Bland and Altman [2] has been examined in Carkeet and Goh [20] under a different perspective, the particular method is included in the following appraisal for the sake of completeness and with the intention to explicate additional properties that were not reported before.

Specifically, Monte Carlo simulation studies of 10,000 iterations were performed to compute the simulated coverage probability of the exact and approximate confidence intervals for the percentiles of a standard normal distribution N(0, 1). The designated sample size has six different magnitudes: N = 10, 20, 30, 50, 100, and 200. Also, a total of eight percentile probabilities are examined: p = 0.025, 0.05, 0.10, 0.20, 0.80, 0.90, 0.95, and 0.975. For each replicate, the lower and upper confidence limits {\( \widehat{\uptheta} \) _L , \( \widehat{\uptheta} \) _U }, {\( \widehat{\uptheta} \) _AL , \( \widehat{\uptheta} \) _AU }, and {\( \widehat{\uptheta} \) _BAL , \( \widehat{\uptheta} \) _BAU } were computed to construct the 95 and 97.5% one-sided confidence intervals and the corresponding 90 and 95% two-sided confidence intervals. The simulated coverage probability was the proportion of the 10,000 replicates whose confidence interval contained the population normal percentile. Then, the adequacy of the one- and two-sided interval procedures is determined by the error = simulated coverage probability – nominal coverage probability. The results are summarized in Tables 1, 2, 3 and 4 for the exact and approximate confidence intervals with two-sided confidence coefficient 1 – α = 0.90 and 0.95, respectively.

It can be seen from the resulting errors of the three types of confidence intervals that the exact approach performs extremely well for all 96 cases presented in Tables 1, 2, 3 and 4. For the two approximate methods of Chakraborti and Li [24] and Bland and Altman [2], the coverage probabilities of their two-sided interval remain rather close to the nominal confidence levels. However, the corresponding approximate one-sided interval procedures do not preserve the same desired accuracy unless the sample size is large. Due to different degree of presumed simplifications, the interval procedure of Bland and Altman [2] is inferior to that of Chakraborti and Li [24], especially for small sample sizes. To enhance the explication, the simulated coverage probabilities of the 97.5% one-sided confidence intervals for N = 10 are plotted in Fig. 1. Despite the attractive coverage behavior of the approximate two-sided confidence intervals, the errors of the upper confidence intervals tend to be negative for small p while those associated with large p are consistently positive. The situations of the lower confidence intervals reveal exactly the opposite patterns. In other words, the corresponding lower and upper confidence limits are generally too large for the 2.5th, 5th, 10th and 20th normal percentiles and are mostly too small for the 80th, 90th, 95th, and 97.5th normal percentiles. Consequently, the two endpoints of the two-sided confidence intervals generally do not meet the assumption of equal-tailed error rates for the two approximate interval methods. A mere coverage probability assessment of the approximate two-sided confidence intervals may obscure the potential biases of the confidence limits based on the t(ν) approximations described in Eqs. 11 and 13. It is inappropriate to claim that a two-sided interval procedure is accurate on the basis of a combination of some noticeable under- and over-estimated confidence limits. Instead, the exact interval procedure should be used in preference to the approximate methods of Bland and Altman [2] and Chakraborti and Li [24].

In view of the wide application in medical studies, this article aims to explicate the theoretical and empirical features of interval procedures of percentiles. An integrated discussion is presented to address the similarities and differences of exact and approximate confidence intervals constructed with various pivotal quantities described in the literature. Although there are distinct selections of pivotal quantities, it is shown that they yield the same exact confidence intervals. Notably, the exact interval procedure requires the use of the cumulative distribution function of a noncentral t distribution. The difficulty of applying the exact approach has been alleviated because of the availability of specialized routines in popular software packages. In contrast, the approximate interval methods are computationally simple and do not require specialized software because they only involve the quantiles of a regular t distribution. However, the approximate confidence intervals carry the symmetry property of a t distribution whereas the noncentral t distribution is skewed so that the resulting exact confidence intervals are not equidistant around the primary statistic.

Despite the positive findings in previous research, detailed numerical assessments are presented to reveal the underlying drawbacks of the approximate methods under the notion that the endpoints of a two-sided confidence interval have a corresponding interpretation as a lower or upper confidence limit of a one-sided confidence interval. Essentially, the simplicity and symmetry of an approximate confidence interval generally do not maintain the assumption of equal-tailed error rates for the two individual endpoints. For the purpose of planning percentile studies so that the results will help confirm meaningful reference targets, sample size procedures for precise interval estimation of normal percentiles are described under the precision criteria of expected width and assurance probability. To enhance the applicability of the exact interval approach and corresponding sample size methodologies, computer codes are also presented to perform the required computations.