Simpson's Paradox, Lord's Paradox, and Suppression Effects are the same phenomenon – the reversal paradox

This article discusses three statistical paradoxes that pervade epidemiological research: Simpson's paradox, Lord's paradox, and suppression. These paradoxes are not just tantalising puzzles of purely academic interest; potentially, they have serious implications for the interpretation of evidence from observational studies. Scenarios which are associated with and can be explained by these paradoxes are discussed. A concise explanation of these paradoxes and an historical overview is also provided. Simulated data based upon the foetal origins of adult diseases hypothesis [1, 2] are used to illustrate how the three paradoxes are different manifestations of one phenomenon – the reversal paradox – depending on whether the outcome and explanatory variables are categorical, continuous or a combination of both; this renders the issues and remedies for any one to be similar for all three. All statistical analyses were performed within SPSS 15.0 (SPSS Inc, Chicago, USA).

The 'foetal origins of adult disease' hypothesis (FOAD), which has evolved into the 'developmental origins of health and disease' (DOHaD) hypothesis [1, 2], was proposed to explain the associations observed between low birth weight and a range of diseases in later life. These associations have been interpreted as evidence that growth retardation in utero has adverse long-term effects on the development of vital organ systems which predispose the individuals to a range of metabolic and related disorders in later life. Nevertheless, although an inverse association between birth weight and disease in later life was found in some studies, this relationship was only established in many studies after the current body size variables such as body mass index (BMI), body weight and/or body height were adjusted for in the regression analysis. As body sizes may be in the causal pathway from birth weight to health outcomes in later life, the justification of this adjustment of current body sizes has been questioned recently [3‐8].

Using the inverse relationship between birth weight and systolic blood pressure in later life as an example, Figure 1 shows the directed acyclic graphs [9‐11] for the possible relationships between the three observed variables: birth weight, current body weight and systolic blood pressure. In Figure 1a, current body weight is on the causal pathway from birth weight to systolic blood pressure, so current body weight is not a genuine confounder and should not be adjusted for. In Figure 1b, there is no relationship between birth weight and current body weight, and therefore the latter is not a confounder for the relationship between birth weight and blood pressure either. However, this model cannot explain the observed positive correlations between birth weight and current body weight in many epidemiological studies. In Figure 1c, current body weight is a confounder because it is ancestor to both birth weight and blood pressure in the directed acyclic graph [9‐11]. Obviously, this scenario is implausible in reality because current body weight cannot affect birth weight. In Figure 1d, the observed positive correlation between birth weight and current body weight is due to an unobserved confounder, UC, which affects both birth weight and current body weight. Also, there is no path from birth weight and current body weight [7], i.e. if UC could be identified and measured, birth weight and current body weight would be independent, conditional on UC [12]. More complex causal diagrams for the three variables are possible by incorporating more unobserved variables in the model. However, the four scenarios in Figure 1 are sufficient for our discussion in this study, so we do not pursue them further.

Figure 1a,c and 1d all explain the observed correlation structure amongst birth weight, current body weight and blood pressure equally well, and it is not possible to judge which one is true based upon the observed data. For example, researchers may argue current body weight is a genuine confounder in Figure 1d and therefore should be adjusted for [7]. This can only be confirmed when the unobserved confounder (be parental, genetic, or environmental factors) is identified and the conditional independence between birth weight and current weight is satisfied.

Nevertheless, the adjustment of current body weight in the statistical analysis will change the estimated relationship between birth weight and blood pressure, as the adjusted relationship is a conditional relationship. Differences between the unadjusted and adjusted (i.e. unconditional and conditional) relationships frequently cause confusion in the interpretations of statistical analyses and they also give rise to three statistical paradoxes, which we shall explain in the next section.

Simpson's paradox [13], or Yule's paradox [14], is a well known statistical phenomenon. It is observed when the relationship between two categorical variables is reversed after a third variable is introduced to the analysis of their association, or alternatively where the relationship between two variables differs within subgroups compared to that observed for the aggregated data. Although first discussed by Karl Pearson in 1899 [15], it is George Udny Yule, once Pearson's assistant, who provides a detailed assessment of this problem in 1903 [14].

Lord's paradox was named after two short articles in the psychology literature by Frederick M Lord regarding the use of analysis of covariance (ANCOVA) within non-experimental studies [21, 22]. In contrast to Simpson's paradox, little discussion of Lord's paradox can be found in the statistical and epidemiological literature [23], though social scientists have shown a great interest in this phenomenon [24‐28]. Lord's paradox refers to the relationship between a continuous outcome and a categorical exposure being reversed when an additional continuous covariate is introduced to the analysis. One specific example is that the additional covariate is a measure made at baseline within a longitudinal study, where the outcome is the same variable measured some time later (e.g. following an intervention). Therefore, the aim is to measure change in the outcome by adjusting for the baseline measurements, and the categorical covariate might be the exposure/control groups – this is the familiar design for ANCOVA. This controversy was first discussed in 1910 between Karl Pearson and Arthur C Pigou when they debated the role of parental alcoholism and its impact on the performance of children [29].

Of the three paradoxes, suppression effects within multiple regression are probably the least recognised amongst clinical and epidemiological researchers, though the suppression phenomenon has been extensively discussed by statisticians [32‐34] and methodologists from the social sciences [35, 36]. The classical definition of suppression is that a potential covariate that is unrelated to the outcome variable (i.e. has a bivariate correlation of zero) increases the overall model fit within regression (as assessed by R², for instance) when this covariate is added to the model. This seems counter-intuitive and needs some explanation.

Suppose y is the outcome variable, and x₁and x₂ are two covariates (i.e. 'explanatory' variables). Denote the bivariate Pearson correlation between y and x₁ as r_{y 1}; the correlation between y and x₂ as r_{y 2}; and the correlation between x₁ and x₂ as r₁₂. Within multiple regression, where y = b₀ + b₁x₁ + b₂x₂, the standardized partial regression coefficients of b₁ and b₂ for x₁ (β₁) and x₂ (β₂), respectively, are given by [37]:

Now suppose that y is adult blood pressure (BP), x₁ birth weight (BW), and x₂ adult current weight (CW). Many studies have shown the bivariate correlation (r_y1) between BP (y) and birth weight (x₁) to be negative though weak [38, 39], whilst others show this to be positive [40]; for illustrative purposes only, assume that r_y1is zero. Many studies show the bivariate correlation (r_y2) between BP (y) and current weight (x₂) to be positive [41]. When BP is regressed on birth weight and current weight, the model fit assessed by R² becomes [37]:

Since r_y1is equal to zero, equation (2) becomes:

Since 1 − r 12 2 will always be smaller than 1, R 2 = r y 2 2 1 − r 12 2 will always be greater than r y 2 2. By including x₁ in the regression model, more variance of y is 'explained', i.e. the predictability of the model increases. However, this seems counterintuitive, since the zero bivariate correlation between y and x₁ (r_y1= 0) indicates that no more variance in y can be explained by x₁. So where does the additional 'explained variance' in y come from when x₁ is entered in the regression model? The answer is that the additional explained variance in y comes from x₂.

Although x₁ is not correlated with y, it is positively correlated with x₂, which in turn is positively correlated with y. When x₁ is entered in the model, it 'suppresses' the part of x₂ that is uncorrelated with y, thereby increasing overall predictability. In other words, the role of x₁ in the model is to suppress (reduce) the noise (the uncorrelated component of x₂) within the correlation between y and x₂, as though any uncertainty in x₂ 'predicting' y is 'explained' by x₁.

It is not only R² that is increased; the coefficient for x₂, β 2 = r y 2 1 − r 12 2, becomes greater than r_{y 2}. Furthermore, although r_{y 1}is equal to zero, β₁ is not zero and becomes negative: β 1 = − r 12 r y 2 1 − r 12 2. In general, the greater the positive correlation between x₁ and x₂, the greater the absolute value of β₁ and β₂. However, having r_{y 1}equal zero (or being negative) is not necessary to observe suppression; r_{y 1}may be positive and x₁ may still be a suppressor [35].

It was Paul Horst, in 1941, who first explored this curious phenomenon within educational research [42], and in the last few decades, many statisticians have been interested in this topic [33‐35]. There are still very few discussions within the clinical and epidemiological literature regarding the impact of suppression (i.e. the impact on the changes in the regression coefficients and R²) on the interpretation of non-randomised studies whilst making statistical adjustment for covariates within regression [12, 43].

The reversal paradox is often used as the generic name for Simpson's paradox, Lord's Paradox, and suppression (see Table 4). Whilst the original definition and naming of the reversal paradox was derived from the notion that the direction of a relationship between two variables might be reversed after a third variable is introduced, this nevertheless may generalise to scenarios where the relationship between two variables is enhanced, not reduced or reversed, after the third variable is introduced (as with many studies on the foetal origins hypothesis).

In non-randomised studies, the reversal paradox can often occur due to 'controlling' for what is typically termed a confounder, even though a clear definition of what is meant by 'confounder' is rarely provided (contingent on understanding its role in the biological/clinical process being modelled). Differences in the strength or even direction of any association between outcome and exposure might give rise to contradictory interpretations regarding potential causal relationships. Furthermore, it is very difficult, if not impossible, to compare results across studies where many varied attempts are made to control for different confounders, especially in the absence of any consistent reasoning given for the choice of confounders. In some situations, statistical adjustment might introduce bias rather than eliminate it [45].

It might be suggested that the adjustment of current weight in our foetal origins example can be viewed as estimations of direct and indirect effects, such as those in path analysis or structural equation modelling. Recall Figure 1a, the path from birth weight to BP is to estimate the direct effect of birth weight → BP, and then the path from birth weight → current weight → BP is to estimate the indirect effect. For instance, in the model 3 of Table 3, the regression coefficient for birth weight, -3.708, is the direct effect, and the indirect effect is derived from 0.465 (the regression coefficient for current weight in model 3) multiplied by 11.976 (the simple regression coefficient for birth weight when current weight is regressed on birth weight) = 5.569. The total effect is therefore -3.708 + 5.569 = 1.861, which is the simple regression coefficient for birth weight in the model 1 of Table 3. Our reservation with interpreting the results from model 3 as the partition of the total effect into direct and indirect effect is that many variables, such as current height and current BMI, can be put in between birth weight and BP, and it can be claimed that there is more than one indirect effect. Furthermore, any body size measured after birth, for example, body weight at year one, year two etc, can be adjusted for in the model and presumably used to estimate the indirect effects and direct effect. Whilst the total effect of birth weight on BP is not affected by the numbers of intermediate body size variables in the model, the estimation of 'direct' effect differs when different intermediate variables are adjusted for. Unless there is experimental evidence to support the notion that there are indeed different paths of direct and indirect effects from birth weight to BP, we are cautious of using such terminology to label the results from multiple regression, as with model 3. In other words, to determine whether the unconditional or conditional relationship reflects the true physiological relationship between birth weight and blood pressure, experiments in which birth weight and current weight can be manipulated are required in order to estimate the impact of birth weight on blood pressure.

Although the three statistical paradoxes occur in different types of variables, they share the same characteristic: the association between two variables can be reversed, diminished, or enhanced when another variable is statistically controlled for. Understanding the concepts and theory behind these paradoxes will provide insights into some of the controversial or contradictory results from previous research. Prior knowledge and theory play an important role in the statistical modelling of non-randomised data. Incorrect use of statistical models might produce consistent, replicable, yet erroneous results.