Study-level estimation of log-odds-ratio, log-risk-ratio, and risk difference
Consider
K studies that used a particular individual-level binary outcome. Study
i (
\(i = 1, \ldots , K\)) reports
\(X_{iT}\) and
\(X_{iC}\), the numbers of events in the
\(n_{iT}\) subjects in the Treatment arm and the
\(n_{iC}\) subjects in the Control arm. It is customary to treat
\(X_{iT}\) and
\(X_{iC}\) as independent binomial variables:
$$\begin{aligned} X_{iT}\sim {\textrm{Bin}}(n_{iT},p_{iT})\qquad {\text {and}}\qquad X_{iC}\sim {\textrm{Bin}}(n_{iC},p_{iC}). \end{aligned}$$
(1)
The log-odds-ratio for Study
i is
$$\begin{aligned} \theta _{i} = \log _{e} \left( \frac{p_{iT}(1 - p_{iC})}{p_{iC}(1 - p_{iT})}\right) \qquad \text {estimated by} \qquad \check{\theta }_{i} = \log _{e} \left( \frac{\check{p}_{iT}(1 - \check{p}_{iC})}{\check{p}_{iC}(1 - \check{p}_{iT})}\right) , \end{aligned}$$
(2)
where
\(\check{p}_{ij}\) is an estimate of
\(p_{ij}\).
As inputs, a two-stage meta-analysis uses estimates of the
\(\theta _i\) (
\(\hat{\theta }_i\)) and estimates of their variances (
\(\hat{v}_i^2\)). It is helpful to have an unbiased estimator of
\(\theta\). For a binomial random variable
\(X \sim {\textrm{Bin}}(n,p)\), we denote the maximum-likelihood (ML) estimator of
p by
\(\tilde{p} = X/n\) . Böhning and Viwatwongkasem [
13] studied estimators of
p of the form
\((X + a)/(n + 2a)\). Use of
\(a = 0.5\) and hence
\(\hat{p} = (X + 0.5)/(n + 1)\) eliminates
O(1/
n) bias and provides the least biased estimator of LOR [
14]. We use
\(\hat{p}\) when estimating LOR, but also, for comparison, retain the use of
\(\tilde{p}\) in standard methods.
The (conditional, given the
\(p_{ij}\) and
\(n_{ij}\)) asymptotic variance of
\(\hat{\theta }_i\), derived by the delta method, is
$$\begin{aligned} v_{i}^2 = \textrm{Var}(\hat{\theta }_{i}) = \frac{1}{n_{iT} {p}_{iT} (1 - {p}_{iT})} + \frac{1}{n_{iC} {p}_{iC} (1 - {p}_{iC})}, \end{aligned}$$
(3)
estimated by substituting
\(\hat{p}_{ij}\) for
\(p_{ij}\). This estimator of the variance is unbiased in large samples, but Gart et al. [
14] note that it overestimates the variance for small sample sizes. They also give approximate conditional higher moments of LOR.
The log-risk-ratio (LRR) for Study
i is
$$\begin{aligned} \rho _{i} = \log _{e}(p_{iT}) - \log _{e}(p_{iC}) \quad \text {estimated by} \quad \hat{\rho }_{i} = \log _{e}(\check{p}_{iT}) - \log _{e}(\check{p}_{iC}), \end{aligned}$$
(4)
where
\(\check{p} = (X + 1/2) / (n + 1/2)\) provides an unbiased (to order
\(O(n^{-2})\)) estimate of
\(\log (p)\) [
15]. An unbiased (to
\(O(n^{-3})\)) estimate of the variance of
\(\hat{\rho }\) [
15] is
$$\begin{aligned} \widehat{\textrm{Var}}(\hat{\rho }) = \frac{1}{X_T + 1/2} - \frac{1}{n_T + 1/2} + \frac{1}{X_C + 1/2} - \frac{1}{n_C + 1/2}, \end{aligned}$$
(5)
and Pettigrew et al. [
15] also give approximate conditional higher moments for
\(\log (\hat{p})\).
The risk difference (RD) for Study
i is
$$\begin{aligned} \Delta _{i} = p_{iT} - p_{iC} \quad \text {estimated by} \quad \hat{\Delta }_{i} = \tilde{p}_{iT} - \tilde{p}_{iC}. \end{aligned}$$
(6)
Its variance is
$$\textrm{Var}(\hat{\Delta }_i) = p_{iT} (1 - p_{iT}) / n_{iT} + p_{iC} (1 - p_{iC}) / n_{iC},$$
estimated by substituting \(\tilde{p}\) for p. The moments of the binomial distribution directly yield the conditional higher moments of RD.
All three binary effect measures (LOR, LRR, and RD) have the form
\(\eta = h(p_T) - h(p_C)\). This relation facilitates calculation of conditional moments of
\(\hat{\eta }\) from the moments of
h(
p). Additional file 1 gives the details. Even when they are not available in closed form, our simulations yield an exact calculation of conditional central moments of
\(\hat{\eta }\) for all three effect measures, similar to the implementation of Kulinskaya and Dollinger [
11] for LOR.
Random-effects model and the Q statistic
We consider a generic random-effects model: For Study i (\(i = 1,\ldots ,K\)) the estimate of the effect is \(\hat{\theta }_i \sim G(\theta _i, v_i^2)\), where the effect-measure-specific distribution G has mean \(\theta _i\) and variance \(v_i^2\), and \(\theta _i \sim N(\theta , \tau ^2)\). Thus, the \(\hat{\theta }_i\) are unbiased estimates of the true conditional effects \(\theta _i\), and the \(v_i^2 = \textrm{Var}(\hat{\theta }_i | \theta _i)\) are the true conditional variances.
Cochran’s
Q statistic is a weighted sum of the squared deviations of the estimated effects
\(\hat{\theta }_i\) from their weighted mean
\(\bar{\theta }_w = \sum w_i\hat{\theta }_i / \sum w_i\):
$$\begin{aligned} Q=\sum w_i (\hat{\theta }_i - \bar{\theta }_w)^2. \end{aligned}$$
(7)
In [
2]
\(w_i = 1/\hat{v}_i^2\), the reciprocal of the
estimated variance of
\(\hat{\theta }_i\), hence the notation
\(Q_{IV}\). In what follows, we examine
\(Q_F\), discussed by DerSimonian and Kacker [
6] and further studied by Kulinskaya et al. [
7], in which the
\(w_i\) are arbitrary positive constants. In
\(Q_F\) we specify
\(w_i = \tilde{n}_i = n_{iC} n_{iT} / n_i\), the effective sample size in Study
i (
\(n_i = n_{iC} + n_{iT}\)).
Define
\(W = \sum w_i\),
\(q_i = w_i / W\), and
\(\Theta _i = \hat{\theta }_i - \theta\). In this notation, and expanding
\(\bar{\theta }_w\), Eq. (
7) can be written as
$$\begin{aligned} Q = W \left[ \sum q_i (1 - q_i) \Theta _i^2 - \sum \limits _{i \not = j} q_i q_j \Theta _i \Theta _j \right] . \end{aligned}$$
(8)
We distinguish between the conditional distribution of
Q (given the
\(\theta _i\)) and the unconditional distribution, and the corresponding moments of
\(\Theta _i\). For instance, the conditional second moment of
\(\Theta _i\) is
\(M_{2i}^c = v_i^2\), and the unconditional second moment is
\(M_{2i} = \textrm{E}(\Theta _i^2) = \textrm{Var}(\hat{\theta }_i) = \textrm{E}(v_i^2) + \tau ^2\).
Under the generic REM, it is straightforward to obtain the first moment of
\(Q_F\) as
$$\begin{aligned} \textrm{E}(Q_F) = W \left[ \sum q_i (1 - q_i) \textrm{Var}(\Theta _i) \right] = W \left[ \sum q_i (1 - q_i) (\textrm{E}(v_i^2) + \tau ^2) \right] . \end{aligned}$$
(9)
This expression is similar to Eq. (4) in [
6]; DerSimonian and Kacker use
\(v_i^2\) instead of its unconditional mean
\({\textrm{E}}(v_i^2)\).
Kulinskaya et al. [
7] also provide expressions for the second and third moments of
\(Q_F\), but these moments require higher moments of
\(\Theta\), up to the fourth and the sixth moments, respectively. The variance of
Q is given by
$$\begin{aligned} W^{-2} \textrm{Var}(Q) = \sum \limits _i q_i^2 (1-q_i)^2 (M_{4i} - M_{2i}^2) + 2 \sum \limits _{i \not = j} q_i^2 q_j^2 M_{2i} M_{2j}, \end{aligned}$$
(10)
where
\(M_{4i} = {\textrm{E}}(\Theta _i^4)\) is the fourth (unconditional) central moment of
\(\hat{\theta }_i\).
In the standard REM for LOR, we would assume
\({\textrm{logit}}(p_{iT}) = {\textrm{logit}}(p_{iC}) + \theta _i\) for
\(\theta _i \sim N(\theta ,\tau ^2)\). The intercept
\(p_{iC}\) may also be random (i.e.,
\(p_{iC}\sim H(\cdot )\)). Further,
\(p_{iC}\) and
\(p_{iT}\) may be correlated. Similarly, in the standard REM for RD, we would assume
\(p_{iT} = p_{iC} + \Delta _i\) for
\(\Delta _i \sim N(\Delta ,\tau ^2)\). However, the distribution of
\(\Delta _i\) needs to be restricted, to ensure that probabilities lie within (0,1). Before this is resolved, we cannot derive unconditional moments of RD. Similarly, for LRR, we assume that
\(p_{iT} = \exp (\log p_{iC} + \rho _i)\), and the distribution of
\(\rho _i\) needs to be restricted to keep values of
\(p_{iT}\) within (0,1). Bakbergenuly et al. [
16] give a detailed discussion.
Fortunately, in the fixed-intercept models (i.e., when the \(p_{iC}\) are fixed), assuming also homogeneity of effects (\(\tau ^2 = 0\)), the unconditional and conditional moments of each binary effect measure coincide. Therefore, the conditional moments of Q are sufficient to obtain a moment-based approximation to the distribution of \(Q_F\) under homogeneity.
Approximations to the null distributions of \(Q_F\) and \(Q_{IV}\)
For meta-analysis of mean differences (MD), Kulinskaya et al. [
7] considered the distribution of
\(Q_F\), a quadratic form in normal variables, which has the form
\(Q = \Theta ^{T}A\Theta\) for a symmetric matrix
A of rank
\(K - 1\). Because, for MD, the vector
\(\Theta\) has a multivariate normal distribution,
\(N(\mu ,\Sigma )\), the distribution of
\(Q_F\) can be evaluated by the algorithm of Farebrother [
17] (after determining the eigenvalues of
\(A \Sigma\) and some other inputs). If the variances in
\(\Sigma\) are the true variances, Farebrother’s algorithm evaluates the exact distribution of
Q. In practice (as in our simulations), it is necessary to plug in estimated variances. Encouragingly, the resulting approximation is quite accurate for MD. Kulinskaya et al. [
7] also considered a two-moment approximation and a three-moment approximation. The three-moment approximation regularly encountered numerical problems, so we do not include it here.
For the binary effect measures, \(Q_F\) is a quadratic form in asymptotically normal variables. The Farebrother algorithm may provide a satisfactory approximation for larger sample sizes, though it may not behave well for small n. To apply it, we again plug in conditional or unconditional estimated variances. (In the fixed-intercept models, the two coincide under the null, \(\tau ^2 = 0\).) We investigate the quality of that approximation, which we denote by F SSW, and the two-moment approximation (2M SSW), which is based on the gamma distribution. For each of these two approximations, we investigate two approaches to estimating \(p_{iT}\) to plug into the calculation of the second and fourth central moments of an effect measure, \(\hat{\eta }_i\). The “naïve” approach estimates \(p_{iT}\) from \(X_{iT}\) and \(n_{iT}\). For the “model-based” approach, we observe that each of LOR, LRR, and RD has the form \(\eta = h(p_T) - h(p_C)\), which facilitates calculation of conditional moments of \(\hat{\eta }\) from the moments of h(p). We obtain estimated moments from the relation \(\widehat{h(p_{iT})} = \widehat{h(p_{iC})} + \bar{\eta }\) for a fixed-weights mean effect \(\bar{\eta }\). Thus, we study four new approximations to the null distribution of \(Q_F\): F SSW naïve, F SSW model, 2M SSW naïve, and 2M SSW model.
The null distribution of
\(Q_{IV}\) is usually approximated by the chi-square distribution with
\(K - 1\) degrees of freedom. For the binary effect measures, as also for both MD and SMD, this approximation is not accurate for small sample sizes [
9]. For RD and LOR, Kulinskaya et al. [
18] and Kulinskaya and Dollinger [
11], respectively, provided an improved approximation to the null distribution of
\(Q_{IV}\) based on a two-moment gamma approximation; we denote the approximation for LOR by KD and that for RD by KDB. Biggerstaff and Jackson [
19] used the Farebrother approximation to the distribution of a quadratic form in normal variables as the “exact” distribution of
\(Q_{IV}\). Jackson et al. [
20] extended this approach to a
Q with arbitrary weights in a meta-regression setting. When
\(\tau ^2 = 0\), the Biggerstaff and Jackson [
19] approximation to the distribution of
\(Q_{IV}\) is the
\(\chi ^2_{K - 1}\) distribution.