Sequences of regressions and their independences

Abstract

Ordered sequences of univariate or multivariate regressions provide statistical models for analysing data from randomized, possibly sequential interventions, from cohort or multi-wave panel studies, but also from cross-sectional or retrospective studies. Conditional independences are captured by what we name regression graphs, provided the generated distribution shares some properties with a joint Gaussian distribution. Regression graphs extend purely directed, acyclic graphs by two types of undirected graph, one type for components of joint responses and the other for components of the context vector variable. We review the special features and the history of regression graphs, prove criteria for Markov equivalence and discuss the notion of a simpler statistical covering model. Knowledge of Markov equivalence provides alternative interpretations of a given sequence of regressions, is essential for machine learning strategies and permits to use the simple graphical criteria of regression graphs on graphs for which the corresponding criteria are in general more complex. Under the known conditions that a Markov equivalent directed acyclic graph exists for any given regression graph, we give a polynomial time algorithm to find one such graph.

Appendix: Details of regressions for the chronic pain data

Tables 1–8 show the results of linear least-squares regressions or logistic regressions, one at a time, for each of the response variables and for each component of a joint response separately. At first, each response is regressed on all its potentially explanatory variables given by their first ordering. The tables give the estimated constant term and for each variable in the regression, its estimated coefficient (coeff), the estimated standard deviation of the coefficient (s _coeff), as well as the ratio z _obs=coeff/s _coeff. These ratios are compared with 2.58, the 0.995 quantile of a random variable Z having a standard Gaussian distribution, for which Pr(|Z|>2.58)=0.01. In backward selection steps, the variable with the smallest observed value |z _obs| is deleted from a regression equation, one at a time, until the threshold is reached.

Table 1 Response: Y, success of treatment; linear regression including a quadratic term

Table 2 Response: Z _a , intensity of pain after treatment; linear regression

Table 3 Response: X _a , depression after treatment; linear regression

Table 4 Response: Z _b , intensity of pain before; linear regression

Table 5 Response: X _b , depression before; linear regression

Table 6 Response: U, chronicity of pain; linear regression

Table 7 Response: A, site of pain; logistic regression

Table 8 Response: V, previous illnesses; linear regression

The procedure defines a selected model, unless one of the excluded variables has a contribution of $|z '_{\mathrm{obs}}|>2.58$ when added alone to the selected directly explanatory variables; then such a variable needs also to be included as an important directly explanatory variable. This did not happen in the given data set.

The tables show for linear models also R ², the coefficient of determination, both for the full and for the selected model. Multiplied by 100, it gives the percentage of the variation in the response explained by the model.

In the linear regression of Z _a on X _a and on the directly explanatory variables of both Z _a and X _a , that is, on Z _b ,X _b ,A, the contribution of X _a leads to z _obs=3.51, which coincides—by definition—with z _obs computed for the contribution of Z _a in the linear regression of X _a on Z _a and on Z _b ,X _b ,A. Hence the two responses are correlated even after considering the directly explanatory variables and a dashed line joining Z _a and Z _b is added to the well-fitting regression graph in Fig. 8.

In the linear regression of Z _b on X _b and on the directly explanatory variables of both Z _b and X _b , that is, on U,A,V,B, the contribution of X _b leads to z _obs=2.64. Hence the two responses are associated after considering their directly explanatory variables and there is a dashed line joining Z _b and X _b in the regression graph of Fig. 8.

The relatively strict criterion, for excluding variables, assures that all edges in the derived regression graph correspond to dependences that are considered to be substantive in the given context. Had instead a 0.975 quantile been chosen as threshold, then one arrow from A to Y and another from U to X _a would have been added to the regression graph. Although this would correspond to a better goodness-of-fit, such weak dependences are less likely to become confirmed as being important in follow-up studies.

The subgraph induced by Z _a ,Z _b ,X _a ,X _b of the regression graph in Fig. 8 corresponds to two seemingly unrelated regressions. Even though separate least-squares estimates can in principle be severely distorted, for the present data, the structure is so well-fitting in the unconstrained multivariate regression of Z _a and X _a on Z _b , X _b , U,V,A,B, that is, in a simple covering model, that none of these potential problems are relevant.

With C={U,V,A,B}, this is evident from the observed covariance matrix of Z _a ,X _a given Z _b ,X _b ,C, denoted here by $\tilde{\varSigma}_{aa|bC}$ and the observed regression coefficient matrix $\tilde{\varPi}_{a|b.C}$ being almost identical to the corresponding maximum likelihood estimators $\hat{\varSigma}_{aa|bC}$ and $\hat{\varPi}_{a|b.C}$.

The former can be obtained by sweeping or partially inverting the observed covariance matrix of the eight variables with respect to Z _b ,X _b ,C and the latter by using an adaption of the EM-algorithm, due to Kiiveri (1987), on the observed covariance matrix of the four symptoms, corrected for linear regression on C. In this way, one gets

The assumed definition of the joint distribution in terms of univariate and multivariate regressions assures that the overall fit of the model can be judged locally in two steps. First, one compares each unconstrained, full regression of a single response with regressions constrained by some independences, that is, by selecting a subset of directly explanatory variables from the list of the potentially explanatory variables. Next, one decides for each component pair of a joint response whether this pair is conditionally independent given their directly explanatory variables considered jointly. This can again be achieved by single univariate regressions, as illustrated above for the joint responses Z _a and X _a .