Background
Methods
The inconsistency test
Estimation of variance
Simulation study
Empirical evidence to inform simulation scenarios
Simulation scenarios
Number of studies
| |
Balanced direct comparisons | KAB = KAC = KBC = 1, …, 7 |
Imbalanced direct comparisons | KAB = 1, KAC = 4, KBC = 7 (and KAB = 1, KAC = 4, KBC = 3 for the typical loop) |
Treatment effects
| |
Comparison AB | ORAB = 0.73 |
Comparison AC | ORAC = 1 |
Comparison BC | ORBC = exp{log(ORAC) - log(ORAB) + IFABC} |
Inconsistency in the network
| |
Inconsistency Factor | IFABC = {0, 0.3, 0.45, 0.6, 1} |
Heterogeneity in the network
| |
Subjective outcome | τ2 ~ LN(-2.13, 1.582) |
All-cause mortality outcome | τ2 ~ LN(-4.06, 1.452) |
Trial arm size
| |
Small | n ~ U(20, 50) |
Moderate | n ~ U(50, 150) |
Large | n ~ U(150, 300) (and n ~ U(120, 160) for the typical loop) |
Frequency of events
| |
Average risk for frequent events | |
Average risk for rare events | |
Approaches to estimate the variances of the direct pairwise summary effects
| |
Inverse variance method | |
Knapp-Hartung method |
Results
Type I error
Balanced scenario(KAB= KAC= KBC= K) | Imbalanced scenario | |||||||
---|---|---|---|---|---|---|---|---|
K = 1 | K = 2 | K = 3 | K = 4 | K = 5 | K = 6 | K = 7 | KAB=1 | |
KAC=4 | ||||||||
KBC=7 | ||||||||
Type I error (IF = 0) | ||||||||
n ~ U(20,50) | 0.07 | 0.07 | 0.06 | 0.04 | 0.05 | 0.05 | 0.04 | 0.06 |
n ~ U(50,150) | 0.10 | 0.07 | 0.06 | 0.06 |
0.05
|
0.06
| 0.04 |
0.08
|
n ~ U(150,300) | 0.13 |
0.07
| 0.05 | 0.06 | 0.06 | 0.04 | 0.05 | 0.06 |
Power (IF = 0.6) | ||||||||
n ~ U(20,50) | 0.13 | 0.15 | 0.18 | 0.23 | 0.27 | 0.33 | 0.37 | 0.16 |
n ~ U(50,150) | 0.25 | 0.30 | 0.42 | 0.52 |
0.62
|
0.70
| 0.76 |
0.32
|
n ~ U(150,300) | 0.42 |
0.54
| 0.70 | 0.79 | 0.84 | 0.88 | 0.89 | 0.49 |
Coverage Probability (IF = 0.6) | ||||||||
n ~ U(20,50) | 0.96 | 0.96 | 0.97 | 0.98 | 0.97 | 0.97 | 0.97 | 0.97 |
n ~ U(50,150) | 0.95 | 0.96 | 0.97 | 0.96 |
0.96
|
0.96
| 0.96 |
0.95
|
n ~ U(150,300) | 0.93 |
0.95
| 0.94 | 0.94 | 0.96 | 0.95 | 0.95 | 0.95 |
Statistical power
Heterogeneity | No heterogeneity | |||||||
---|---|---|---|---|---|---|---|---|
IF = 0.3 | IF = 0.45 | IF = 0.6 | IF = 1 | IF = 0.3 | IF = 0.45 | IF = 0.6 | IF = 1 | |
Frequent Events | ||||||||
IVDL | 0.17 | 0.26 | 0.36 | 0.59 | 0.20 | 0.38 | 0.52 | 0.77 |
KHDL | 0.19 | 0.27 | 0.37 | 0.60 | 0.27 | 0.44 | 0.58 | 0.80 |
Rare Events | ||||||||
IVDL | 0.10 | 0.15 | 0.21 | 0.38 | 0.09 | 0.16 | 0.25 | 0.49 |
KHDL | 0.13 | 0.18 | 0.24 | 0.41 | 0.16 | 0.23 | 0.33 | 0.55 |
Heterogeneity | No heterogeneity | |||||||
---|---|---|---|---|---|---|---|---|
IF = 0.3 | IF = 0.45 | IF = 0.6 | IF = 1 | IF = 0.3 | IF = 0.45 | IF = 0.6 | IF = 1 | |
Frequent Events | ||||||||
IVDL | 0.10 | 0.15 | 0.23 | 0.42 | 0.13 | 0.23 | 0.38 | 0.68 |
KHDL | 0.11 | 0.17 | 0.24 | 0.42 | 0.19 | 0.31 | 0.44 | 0.73 |
Rare Events | ||||||||
IVDL | 0.08 | 0.10 | 0.14 | 0.25 | 0.07 | 0.11 | 0.17 | 0.35 |
KHDL | 0.11 | 0.12 | 0.16 | 0.28 | 0.12 | 0.17 | 0.25 | 0.44 |
Coverage probability and bias
Heterogeneity | No heterogeneity | |||||||||
---|---|---|---|---|---|---|---|---|---|---|
IF = 0 | IF = 0.3 | IF = 0.45 | IF = 0.6 | IF = 1 | IF = 0 | IF = 0.3 | IF = 0.45 | IF = 0.6 | IF = 1 | |
Frequent Events | ||||||||||
IVDL | 0.90 | 0.94 | 0.94 | 0.94 | 0.93 | 0.96 | 0.98 | 0.97 | 0.97 | 0.97 |
KHDL | 0.89 | 0.93 | 0.93 | 0.93 | 0.91 | 0.92 | 0.95 | 0.94 | 0.94 | 0.93 |
Rare Events | ||||||||||
IVDL | 0.93 | 0.96 | 0.96 | 0.97 | 0.96 | 0.97 | 0.98 | 0.99 | 0.98 | 0.96 |
KHDL | 0.91 | 0.95 | 0.95 | 0.95 | 0.94 | 0.92 | 0.96 | 0.96 | 0.95 | 0.94 |
Heterogeneity | No heterogeneity | |||||||||
---|---|---|---|---|---|---|---|---|---|---|
IF = 0 | IF = 0.3 | IF = 0.45 | IF = 0.6 | IF = 1 | IF = 0 | IF = 0.3 | IF = 0.45 | IF = 0.6 | IF = 1 | |
Frequent Events | ||||||||||
IVDL | 0.92 | 0.96 | 0.96 | 0.96 | 0.95 | 0.97 | 0.98 | 0.98 | 0.97 | 0.97 |
KHDL | 0.91 | 0.95 | 0.96 | 0.95 | 0.94 | 0.93 | 0.96 | 0.95 | 0.95 | 0.93 |
Rare Events | ||||||||||
IVDL | 0.95 | 0.96 | 0.97 | 0.98 | 0.98 | 0.97 | 0.98 | 0.98 | 0.99 | 0.99 |
KHDL | 0.93 | 0.95 | 0.96 | 0.96 | 0.96 | 0.93 | 0.96 | 0.96 | 0.96 | 0.95 |
Characteristics of the inconsistency test in a ‘typical’ loop of evidence
Type I error | Power | Coverage probability | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
IF = 0 | IF = 0.3 | IF = 0.45 | IF = 0.6 | IF = 1 | IF = 0 | IF = 0.3 | IF = 0.45 | IF = 0.6 | IF = 1 | |
All-cause mortality outcome (median (τ2) = 0.02) | ||||||||||
IVDL | 0.05 | 0.14 | 0.23 | 0.38 | 0.75 | 0.95 | 0.97 | 0.99 | 0.98 | 0.95 |
KHDL | 0.11 | 0.21 | 0.32 | 0.46 | 0.78 | 0.89 | 0.94 | 0.93 | 0.92 | 0.90 |
Subjective outcome (median (τ2) = 0.11) | ||||||||||
IVDL | 0.07 | 0.14 | 0.23 | 0.34 | 0.63 | 0.94 | 0.96 | 0.96 | 0.97 | 0.95 |
KHDL | 0.12 | 0.20 | 0.29 | 0.41 | 0.65 | 0.88 | 0.93 | 0.93 | 0.92 | 0.91 |
Discussion
-
The assumption of consistency in network meta-analysis is often evaluated performing a z-test within each loop of evidence.
-
The inconsistency test has low power for the ‘typical’ loop (comprising 8 trials and about 2000 participants) found in published networks. This study suggests that the probability to detect inconsistency when present is between 14% and 21% depending on the estimation method.
-
Power is positively associated with sample size and frequency of the outcome, and negatively associated with the underlying extent of heterogeneity.
-
Using the Knapp-Hartung method to estimate uncertainty around meta-analytic effects is slightly more powerful than the inverse variance approach.
-
Type I error converges to the nominal level as the total number of individuals included in the loop increases while coverage is close to the nominal level for most studied scenarios.
-
We recommend that investigators a) employ a variety of methods to evaluate inconsistency, b) interpret the magnitude of the estimated inconsistency factor and its confidence interval c) adopt a sceptical stance towards statistically non-significant test results unless the loop of evidence has many data d) always consider the comparability of the studies in terms of potential effect modifiers to infer about the possibility of inconsistency