Evaluation of surrogate measures of insulin sensitivity - correlation with gold standard is not enough

The term insulin sensitivity refers to the body’s sensitivity to the effects of insulin and is an umbrella term for various physiological processes. Individuals with low insulin sensitivity require larger amounts of insulin in order to keep blood glucose stable. Impaired insulin sensitivity is a key abnormality underlying the development of type 2 diabetes as well as several other clinical states. Measuring insulin sensitivity is of importance in identifying individuals at risk of developing diabetes and to evaluate diabetes-focused interventions. A number of measures of varying complexity have emerged for this purpose. The hyperinsulinemic euglycemic clamp, HEC (described in de Fronzo et al. [1], among others) is considered the reference method for the measurement of insulin sensitivity and is referred to as the gold standard. In the clamp technique, insulin and glucose are both continuously infused into the bloodstream to maintain plasma insulin concentration at a constant high level and plasma glucose concentration at a constant basal level. M _LBM , the mean glucose infusion rate reached at steady state normalized per kilogram lean body mass, is a measure of insulin sensitivity. The HEC is a costly, time consuming and invasive method requiring trained staff. There is consequently a need for alternative so called surrogate measures, especially for large scale epidemiological studies.

The surrogate measures are grouped into two families of indices, the OGTT-based indices (OGTT = oral glucose tolerance test) and the fasting indices (see Table 1). The OGTT-based indices (e.g. Matsuda, Stumwoll) are based on changes in plasma concentrations of insulin and glucose during an OGTT. Some of the fasting indices, e.g. the homeostasis model of insulin resistance (HOMA-IR), the quantitative insulin sensitivity check index (QUICKI) and the fasting insulin resistance index (FIRI), are based only on fasting plasma concentrations of glucose and insulin, while others include additional biomarkers. A more complete table of surrogate measures can be found in e.g. Otten et al. [2].

There is a wide range of publications discussing the pros and cons of the surrogate measures from various perspectives. Singh and Saxena [3], Gutch et al. [4], Otten et al. [2], among others, present overviews and descriptions of many of the surrogate measures. Borai et al. [5] give an excellent discussion on the choice of suitable measure from the perspective of the nature of the study. Depending on the purpose and situation, some publications concentrate on certain populations, while others attempt more general comparisons. The conclusion is often that one measure is the best choice for a specific population under study.

The most common way of evaluating surrogate measures appears to be by using correlation coefficients with the HEC ([6‐14], among others). Other comparative analyses, such as ROC (receiver operating characteristic) analysis are also used (Rössner et al. [15] and Ruige et al. [16]). These methods are however less frequently used. In particular, the fasting surrogate indices HOMA-IR and QUICKI are often compared. Different authors propose that one or the other is the one to use, or even that both should be used.

This study has two main purposes. The first is to scrutinize the method of using correlation with the HEC to evaluate surrogate measures. We suggest additional aspects to consider in an evaluation. The second purpose is to show that some of the surrogate measures are mathematically equivalent. We show some consequences of this equivalence and question the comparison between the measures, and most importantly the use of several measures within the same study. Here we would like to concur with Rössner et al. [15] in their aim to “limit further comparisons between these fasting indices”.

We define the two most common measures of correlation, and discuss their characteristics and relation. In the next section we discuss why correlation measures alone are not suitable for method comparison. Further, we suggest other aspects worth considering. Next, we focus on three common surrogate measures; HOMA-IR, QUICKI and FIRI. We argue that they are one-to-one transformations of each other and discuss the consequences of this relationship. Finally some conclusions end the paper.

A correlation coefficient measures the extent to which two variables tend to change together, both regarding strength and direction of the relationship. A number of different measures of correlation exist, the two most common being the Pearson’s correlation coefficient and the Spearman’s rank correlation coefficient.

The Pearson’s correlation coefficient measures the strength and direction of the linear relationship between two variables. It ranges from − 1 (perfectly linear negative relationship) to 1 (perfectly linear positive relationship). The Pearson’s correlation coefficient for a sample of data is defined as

where (x _i ,y _i ) are observations of variables (X _i ,Y _i ).

Standard tests and confidence intervals for the Pearson’s correlation coefficient rely on the assumption that data come from a bivariate normal distribution, possibly with unequal variances for X and Y, but with equal variances for all range of X- and Y-values. It is a measure which is sensitive to outliers, hence one or two “odd” measurements might influence the coefficient severely (Fig. 1c). Furthermore, Pearson’s correlation coefficient is by definition a measure of linearity, Hence, the coefficient is 1 in Fig. 1a, while it is only 0.78 in Fig. 1b in spite of a perfect monotone relation.

The dependence on linearity and outliers is reduced by ranking the variables before applying Formula (1), which leads to the Spearman’s rank correlation coefficient.

The Spearman’s rank correlation coefficient is a non-parametric measure of association between two variables. It measures the strength of a monotonic relationship between paired data. It ranges from − 1 (perfect monotonic decreasing relationship) to 1 (perfect monotonic increasing relationship). Note that a linear relationship is always monotonic (Fig. 1a), whereas a monotonic relationship is not necessarily linear (Figs. 1b, d).

A majority of studies base their evaluation of surrogate measures on strength of linearity with the HEC, i.e. on Pearson correlation. Less emphasis is put on strength of a joint trend not necessarily linear, i.e. on Spearman correlation. The choice of correlation coefficient should be based on the type of association that is of relevance. Does a trend necessarily have to be linear?

Correlational meta-analyses combining different coefficients sometimes follow an approach proposed by Rupinski and Dunlap [17] for converting Spearman correlations (ρ) to Pearson correlations (r):

One should be aware that this formula relies on the assumption of a bivariate normal distribution. Most insulin sensitivity measures have skewed distributions, why this assumption is likely to be violated. Converting a Spearman correlation coefficient to a Pearson correlation coefficient for data which does not fulfill this assumption may give unreliable results.

Investigators often wish to estimate insulin sensitivity with a simple, cheap and low-invasive method. Unfortunately, the true value of insulin sensitivity is not available to calibrate on. Thus, the usual practice is to evaluate the surrogate measure by comparison with an established technique, for instance the HEC. Correlation is the most commonly used method for this comparison. Correlation tells us something about the surrogate measure’s relationship (linear or monotonic) with the reference method. However, the information we gain from correlation is limited.

There are a number of factors influencing the correlation coefficient that one should be aware of. The range of measurements has a considerable effect on the size of the correlation coefficient. A larger range of measurements will generally give a higher correlation. Expressed differently, correlation will increase if the between subject variability increases, which may seem counter-intuitive. Furthermore, in particular regarding Pearson correlation, outliers have a high impact on the coefficient, as was seen in Fig. 1c. In addition, standard methods used for testing and estimation are influenced by heterogeneous variance.

We advice against relying too heavily on correlation. Instead we would like to encourage the use of additional methods of evaluation. In the remainder of this section we focus on additional aspects worth considering in an evaluation of a surrogate measure of insulin sensitivity.

In a more complete evaluation of any clinical measure, different aspects of variability should be considered; for instance repeatability, reproducibility and bias. Repeatability is a measure of how well a measurement can be repeated under identical conditions and is derived from the within subject variability of replicates. Reproducibility is a measure of how well a measurement can be replicated under differing conditions, e.g. with different observers, times and laboratories. A highly reproducible measure enables comparison between studies.

As is becoming apparent, the definition of a perfect surrogate measure is not straightforward, a question addressed by Berger [18] among others. What makes a perfect surrogate depends on the purpose of the study. In the terminology of Buyse and Molenberghs [19], a perfect surrogate at the trial level is one which enables prediction of the treatment effect on the reference method from the treatment effect on the surrogate. A surrogate is perfect at the individual level when there is perfect association between the surrogate and the reference method, after correction for the treatment effect.

Several quantities have been proposed in the validation of a surrogate with respect to a reference method for a specific treatment. Freedman et al. [20] propose quantifying the proportion explained (PE), the proportion of the effect of treatment on the reference method that can be explained by the treatment effect on the surrogate. Buyse and Molenberghs [19] propose quantifying the relative effect (RE), the effect of treatment on the reference method relative to the effect of treatment on the surrogate. It should be noted that a large number of observations generally are needed for these validation procedures.

In summary, we find that correlation with the gold standard is not enough to evaluate a surrogate measure of insulin sensitivity. Although a valuable contribution to an evaluation, other aspects should be investigated, for instance repeatability, reproducibility and discriminatory ability.

As mentioned previously, a large number of studies can be found in the literature comparing different surrogate measures. The evaluation is predominantly based on correlation with the HEC or some other reference method. The fasting surrogate measures HOMA-IR and QUICKI are frequently subject to this type of comparison. Some studies find that QUICKI has a higher correlation with the HEC, while others conclude that HOMA-IR is better. However, HOMA-IR, QUICKI (and FIRI) are equivalent in a mathematical sense. If two measures have a one-to-one correspondence, that is if each value for one has a unique counterpart in the other, then one of them is redundant in the sense that it can be transformed into the other. This is the case for HOMA-IR, FIRI and QUICKI. Pairwise, they are each strictly monotone functions of each other for any values of fasting insulin and glucose (see formulas in Table 2). As a result of the one-to-one correspondence a number of analysis methods will give equal results, which we demonstrate below. This should be kept in mind when evaluating these surrogate measures.