Sample size and power considerations in network meta-analysis

When both direct and indirect evidence is available, it may often be advantageous to combine the two statistically[2, 4‐7, 30, 33]. For example, if only two small trials have investigated two active interventions A and B head to head, but 20 trials have compared A or B with placebo, the indirect evidence will be able to add much power and precision to the comparative estimate of A and B. The combination of indirect and direct evidence requires advanced statistical regression techniques (i.e., network meta-analysis) that are beyond the scope of this article[4, 6, 30]. However, in the context of sample size and power considerations, it suffices to understand that indirect evidence, when combined with direct evidence, increases the power and precision of treatment effect estimates[4, 6, 7, 9, 30, 33]. The extent to which is does so can be evaluated readily by using the methods we describe below.

In this section we introduce three methods for gauging how much statistical precision an indirect estimate provides when no direct evidence is available. In particular, we describe how to approximate the amount of information required in a direct (head-to-head) meta-analysis to produce the same precision as that in the available indirect evidence. Simply put, what direct meta-analysis sample size would provide a similar degree of information? We dub this the effective sample size of the indirect evidence or, interchangeably, the effective indirect sample size. We describe three different methods for approximating the effective indirect sample size. Each of these methods differs with respect to simplicity and validity (the simpler one being the least valid), so we outline the simplicity-validity trade-offs at the end of the section.

A simple approach to gauging the degree of power and precision available in indirect evidence is to approximate how many trials are required in an indirect comparison to produce a matching degree of power and precision from a single head-to-head trial. This type of approximation is possible under the simple assumptions that the variances (of the mean) are equal for each trial and that no heterogeneity is present. Glenny et al. showed that when the number of trials is the same in both of two comparisons informing the indirect evidence (e.g., two trials of A vs. C and two trials of B vs. C), it takes four trials in the indirect evidence to produce the same precision as one direct head-to-head trial[8]. In indirect comparisons, however, it is common that one comparison will include more trials than the other. When this happens, the above 1:4 precision ratio no longer holds true. For example, if the number of trials is twice as high in one comparison (i.e., a 1:2 trial count ratio), the indirect comparison will need exactly 4.5 trials to produce the same precision as one head-to-head trial (see mathematical derivation in Appendix 1.a). In reality, however, to maintain a ratio of 1:2 in the trial count, one would need six trials (2:4) in the indirect comparison to produce the same precision as one head-to-head trial. To produce the same precision as two head-to-head trials, one would need 2 × 4.5 = 9 trials, which allows maintaining the 1:2 ratio with three trials in one comparison and six in the other (i.e., 3:6 as the trial count ratio). Table 1 presents the approximate number of trials required in an indirect comparison under different scenarios where the number of trials in the two comparisons is unbalanced. The mathematical derivations for all exact precision ratios are presented in Appendix 1.a.

The cells in underlined italics indicate where the indirect evidence produces the exact precision of the corresponding number of trials. The remaining cells indicate where the indirect evidence produces precision slightly above that of the corresponding number of head-to-head trials.

In some cases the required number of trials in an indirect comparison for a specific trial count ratio produces a precision corresponding to more than that of a stated number of single head-to-head trials. For example, with a trial count ratio of 1:3, one would require 2 × 5.33 = 10.66 indirect comparison trials to produce the precision of two head-to-head trials. However, since we cannot have fractions of trials, we take the closest integer above 10.66 where the trial count ratio is maintained: 12 trials with a trial count ratio of 3:9.

Table 1 can readily be used for quickly and easily checking how many head-to-head trials the indirect evidence ‘effectively’ corresponds to. That is, if the indirect evidence produces the precision of, say, three trials, we can think of the evidence as being as strong as a meta-analysis of three head-to-head trials. For example, if one has an indirect comparison with 4 trials comparing A with C, and 12 trials comparing B with C, the precision of the indirect comparison corresponds to a meta-analysis of 3 trials directly comparing A with B (Table 1). It should be noted that Table 1 is only valid to the extent that trial sample sizes and trial population variances are similar across trials, as well as the extent to which heterogeneity is absent or ignorable.

Another relatively simple approach to gauging the degree of power and precision from an indirect comparison is to consider the collection of trials included in each comparison as one (large) clinical trial. From a sample size perspective, following similar mathematic derivations as the above trial count perspective, the relationship between the precision of an indirect comparison and the precision of a direct meta-analysis turns out the same (see Appendix 1.b for mathematical derivations). For example, to produce the same precision as a head-to-head meta-analysis including 1,000 patients, one would need a total of 4,000 (4 × 1,000) patients in the indirect comparison, provided the number of patients is the same for the two comparisons (2,000:2,000). Taking the sample size perspective comes with the flexibility of a possible reversal of the calculation. For example, an indirect comparison with 500 patients in the A vs. C comparison and 500 patients in the B vs. C comparisons would produce the same precision as a direct comparison with 250 patients [(500 + 500)/4]. Likewise, in a scenario with 1,000 patients in comparison A vs. C, and 10,000 patients in comparison B vs. C, the exact precision ratio is 12.1 (see Table 1), and so the effective direct meta-analysis sample size would be (1,000 + 10,000)/12.1 = 909.

Often, the sample sizes in the two comparisons do not line up to produce the exact precision ratio presented in Table 1 and Table 2. Letting n _AC and n _BC denote the sample sizes for the comparisons of A vs. C and B vs. C, respectively, a more general formula for the effective indirect sample size is (see Appendix 1.b).

For the above example, the effective sample size using this formula is therefore1000 × 10000 / 1000 + 10000 = 909 .

The above simple approach to calculate the effective indirect sample size does not consider the possibility that statistical heterogeneity exists across trials. When heterogeneity is present, the effect estimates that go into the indirect estimate incur a higher degree of variation, and so the effective indirect sample size corresponding to a head-to-head meta-analysis will be smaller than with the above simple approach. In line with already-established heterogeneity corrections for meta-analysis required sample sizes[23, 24, 28], we put forward that the actual number of patients in each of the comparisons informing the indirect estimate can be penalized by the additional variation explained by heterogeneity. In line with previous proposals, we penalize for the ‘lack of homogeneity’[23, 24] using the popular measure of heterogeneity, I ² [34], as a basis for the penalization.

Consider the example where a meta-analysis of A vs. C includes 6,000 patients and a meta-analysis of B vs. C includes 8,000 patients, and assume the estimated degree of heterogeneity for A vs. C is 50% (I _AC ²  = 50%) and 25% for B vs. C (I _BC ²  = 25%). Then the lack of homogeneity is 100%-50% = 50% for A vs. C and 100%-25% = 75% for B vs. C. We penalize the actual sample size by multiplying the actual sample size by the lack of homogeneity, so that the penalized sample size of A vs. C is 50% × 6,000 = 3,000, and the penalized sample size for B vs. C is 75% × 8,000 = 6,000. The total penalized number of patients in the indirect comparison is then 3,000 + 6,000 = 9,000, the patient count ratio is 1:2 (same as 3,000:6,000), the precision ratio is 4.5 (see Table 1), and so the effective heterogeneity-corrected sample size in this indirect comparison is 9,000/4.5 = 2,000.

Following the above example, the general formula for a heterogeneity-corrected effective sample size for indirect evidence is

where n _AC and n _BC are the actual sample sizes (before correction) in the meta-analyses of A vs. C and B vs. C, respectively, and where the precision ratio is based on the heterogeneity-corrected sample sizes (see Table 1 and Table 2). In the appendix we provide a general formula for the precision ratio.

As with the above example of non-penalized sample sizes, the penalized sample sizes may not always line up to match the precision ratio given in Table 1 and Table 2. The more general formula for the heterogeneity-corrected effective sample size is (see Appendix 1.b).

One immediate limitation of the above-proposed sample size heterogeneity correction is the fact that I ² s are typically unreliable and unstable in meta-analyses including a limited number of trials and will depend on the effect metric used[34‐36]. In most cases, it will therefore be preferable to simply assume some plausible degree (percentage) of heterogeneity derived from a mix of clinical considerations and the I² estimate at hand. Typically, an assumption of 25% or 50% heterogeneity will be reasonable in the context of sample size considerations[22]. Table 2 illustrates the effective sample size for various indirect comparison scenarios including a total of 10,000 patients, under different combinations of heterogeneity corrections (one including no correction).

Statistical information, also sometimes referred to as Fisher information, is a more complex statistical measure for gauging the degree of precision present in a data set. For pairwise meta-analyses, the statistical information is equal to the inverse of the pooled variance (i.e., one divided by the variance), which is also the measure of precision[37]. For indirect comparisons, the statistical information (precision) is equal the inverse of the pooled indirect variance. That is, with variances V _AC and V _BC for comparisons A vs. C and B vs. C, the indirect variance is V _AC  + V _BC , so indirect statistical information is 1/(V _AC  + V _BC ). Because the variance incorporates heterogeneity and is dependent on the number of trials and the sample size, no further calculations or adjustments are needed.

A major disadvantage of statistical information in meta-analysis is that it operates on a scale that no statistically non-sophisticated reader could easily grasp[38]. The statistical information may however be useful in the context of sample size requirements since it is possible to calculate the required statistical information (analogous to the required sample size) and compare the actual statistical information present in an indirect comparison with such a yardstick for sufficient power.

Each of the above approaches comes with strengths and limitations. These are outlined in Table 3.

So far we have only described situations where the indirect estimate is obtained through one common comparator. However, it is worth noting that indirect evidence may often come from scenarios where the link between two treatments of interest must go through two or more comparators. For example, if we wish to compare A with D and the following three comparisons are available—A vs. B, B vs. C and C vs. D—then the link between A and D goes through both B and C. The indirect variance is now a sum of three (rather than two) direct variances,V AB + V BC + V CD . We would therefore expect comparably smaller precision. Calculating the effective number of trials in the two common comparator example, we find that with an equal number of trials, say 3:3:3, the precision of the indirect comparison corresponds to that of only one direct trial, that is, an exact precision ratio of 1:9. With a more unbalanced number of trials, say 8:1:8, we get an exact precision ratio of 1:21.

Such consistently large direct to indirect precision ratios indicate that indirect comparisons with two (or more) comparators in the link will typically add very little precision. For this reason we dub them ‘weak links.’ In the context of combining direct and indirect evidence as we address below, it seems that weak links will typically only add an ignorable small amount of precision to the final estimate and thus, for simplicity, may be ignored for sample size and power considerations in indirect comparisons and network meta-analysis.

A recent MTC explored the efficacy of five different interventions for smoking cessation, low-dose NRT (<22 mg nicotine patches), which is an over-the-counter intervention, and four newer and more expensive interventions, combination NRT (i.e., patch plus gum), high-dose NRT (>22 mg patches), buproprion and varenicline[31].The published MTC provides a useful data-set to illustrate the issues we have raised in this article. Low-dose NRT is already well established and well known to yield about a 50% higher success rate than inert control interventions (i.e., relative risk of approximately 1.50). To consider any of the four newer treatments worthwhile, one could argue for the need to demonstrate at least an additional 20% efficacy. These considerations make up the basis for the required sample size considerations. Keep in mind that the published MTC has much higher rates of effective sample size than the assumptions we make in this example.[31]

The median smoking cessation success rate at 6 months for patients in the inert control group is 15.0% across the 79 included trials reporting at this time point. Assuming a 1.5 relative risk, we would expect a 22.5% success rate with low-dose NRT. An additional 20% relative efficacy would suggest a 26.0% success rate (with any of the newer treatments). Accepting a maximum type I error of α = 5% and a power of (1-β) = 90%, the required sample size to demonstrate that any of the newer interventions are at least 20% better than low-dose NRT is 6,303. To assess the strength of the evidence contained in the treatment network, we calculate the effective sample size (number of patients) for each of the newer treatments compared with low-dose NRT. We do this with and without taking heterogeneity into account. We subsequently calculate the information fraction and power based on the effective sample size. Figure 1 presents the number of trials, number of patients and degree of heterogeneity in the comparisons informed by head-to-head evidence. Figure 2 presents the sources of direct and indirect evidence for the four comparisons of interest (newer treatments vs. low-dose NRT).

For low-dose NRT vs. combination NRT, the direct evidence includes 1,664 patients (and no heterogeneity). Indirect evidence exists with inert control as the common comparator. The comparison of low-dose NRT and inert control includes 19,929 patients, but with 63% heterogeneity, so the heterogeneity penalized sample size is 19,929 × (1–0.63) = 7,374. The comparison of combination NRT vs. inert control includes 1,848 patients (and no heterogeneity). The effective indirect sample size without heterogeneity penalization (n _indirect ) and the effective indirect sample size with heterogeneity penalization (n _indirect-Pen ) are therefore

and

Adding this to the direct evidence sample size, we get an effective total sample size of

and

These two total effective sample sizes correspond to an information fraction of 53% and 50% (half of the require sample size has been accumulated) and a statistical power of 66% and 63%. Table 4 presents the direct effective indirect and total sample sizes and the corresponding information fractions and statistical power for all four comparisons of the newer treatments vs. low-dose NRT. The calculations for the remaining three comparisons are presented in Appendix 2.

In the network meta-analysis of interventions for smoking cessation, high-dose NRT and varenicline were both significantly better than low-dose NRT and demonstrated effect estimates larger than the a priori considered minimally important difference. For these comparisons, high-dose NRT is supported by 88% power, and varenicline is supported by 76% power. Considering that the true effects of high dose NRT and varenicline over low dose NRT are much higher than the 20% increase that was assumed for these calculations, the true power of these comparisons are also much higher. For example, if the MTC actual effects for varenicline (i.e., a 38% increase in smoking cessation compared with low dose NRT) are used, the statistical power to detect this difference then exceeds 99%. Combination NRT and buproprion both yielded effect estimates very similar to low-dose NRT. Considering that the two are supported by 66% and 95% power, respectively, and that none of the two effect estimates appear superior, it should be reasonable to infer that the two interventions do not offer any noteworthy benefit over low-dose NRT.