Network meta-analysis: users’ guide for pediatricians

A well-formulated clinical question will typically follow the PICO format (P: population, I: intervention, C: comparator, O: outcomes) [29]. NMA uses the same format except that “I” and “C” (intervention and comparisons), include all the interventions compared against each other. Successful definitions of each element of the PICO are required to determine the studies eligible for the review and develop a priori hypotheses to address possible heterogeneity.

Although, the scope of the research question can vary from narrow to broad, it is essential that for any paired comparison within the NMA, it is plausible that we will, for each outcome of interest, observe similar effects across all patient populations being addressed [30, 31]. Eligibility criteria can be wide enough to permit the possibility of differences in effect across the included patients, interventions, and outcomes. For instance, effects may differ – among eligible studies- in more or less severely affected patients; across high and low doses and across shorter and longer follow-up.

An NMA that assessed the efficacy of asthma treatments strategies included all children with chronic asthma [12]. The definition of chronic asthma was not based on the Global Initiative for Asthma (GINA) asthma guidelines staging [32], nor did the authors present data on disease severity, or attempt subgroup analyses. The broad inclusion criteria and lack of subgroup hypotheses fail to address the differences in disease severity that might lead to differences in treatment response [33].

Another example relates to differences in the measurement of outcome [27, 34]. Two systematic reviews in asthma began with the goal of conducting an NMA; only one was successful. The first study evaluated the effectiveness of the various inhalation regimens on FEV1 improvement [18]. The systematic review revealed large variations in the way the 23 trials measured and reported FEV1. This heterogeneity prevented the review team from performing an NMA. The second NMA assessed the efficacy of treatments on reducing exacerbation [12]. Severe exacerbation was defined as patients needing hospital admission, a visit to the emergency department or a standard course of systemic corticosteroids. In this case, outcomes were reported similarly across trials and the authors presented pooled estimates.

A comprehensive systematic search that identifies all pertinent available studies minimizes the risk of spurious findings from unrepresentative selection of studies. Since many reviews articles have demonstrated the inadequacy of searching only one database [35‐37], an optimal search include all relevant electronic databases (e.g., Medline, Embase, Psycinfo, CENTRAL, CINAHL) [38]. Ideally, a search of the grey literature will minimize the risk of publication bias.

Subsequently, the team selects eligible studies [38]. The report should provide evidence of the reproducibility of assessment of study eligibility through review by at least 2 independent assessors, and present a figure summarizing each selection step in the eligibility determination process (identification of titles and abstracts; culling of titles and abstracts; review of full texts; final determination of eligibility) [39].

Certainty in effect estimates represents how trustworthy are the results and their conclusions [31]. Within any network, it is likely that the quality of the evidence differs across paired comparisons: high quality evidence may reveal that one treatment is superior to another, whereas we may have only low quality evidence regarding the relative merit of other treatments.

Making that rating requires a sequence of judgments relying on assessments of the quality of the direct and indirect evidence. Three articles were published on 2014 by the Grades of Recommendation, Assessment, Development and Evaluation (GRADE) working group, the Cochrane Collaboration, and the ISPOR-AMCP-NPC good practice task force [30, 31, 40] that extend quality of evidence assessment of meta-analysis to NMA. Following the GRADE approach, the overall confidence starts as high for direct, indirect, and network estimates that are derived from RCTs [31]. The evidence can be rated down from high to moderate, low, or very low quality based on the presence and magnitude of any of the 5 domains: Risk of bias (RoB), indirectness, imprecision, inconsistency, and publication bias [31].

Many prior published NMAs have not explicitly addressed all the recommended elements. Fortunately, however, some present the information required for a reader to make the necessary judgments. If the information is not available, then the credibility of the NMA is compromised [31].

Consider, for instance, the GRADE profile for the direct evidence of an NMA of antidepressant medications for improving depression symptoms in children (Table 2) [41]. The evidence certainty for Fluoxetine versus placebo was rated as very low as a result of high RoB, imprecision, and inconsistency. The Imipramine versus placebo comparison was rated as moderate, the only concern being imprecision. With this variation in evidence certainty, making sense of the results requires ratings of evidence quality for each pairwise comparison.

There are two statistical approaches to perform NMA: a frequentist and a Bayesian approach [41, 42]. The frequentist approach is what clinicians will generally see in individual RCTs and conventional meta-analyses. The additional major aspect of Bayesian approaches is the specification of prior probabilities of treatment effects before beginning the data analysis and combining these priors and their precision with the estimate from the data to produce a posterior probability and its credible interval. Results in NMA are presented as effect estimates, typically odds ratios (ORs), or RR, hazard ratio (HR), mean difference (MD) with their 95% confidence interval (CI) (frequentist approach) or credible interval (CrI) (Bayesian approach), both of which describe the range of plausible truth around the point estimate.

Ideally, NMAs will present direct, indirect, and network estimates for each paired comparison. When, however, there are large numbers of comparisons, this becomes a challenging task. For example, the mechanical ventilation modes for RDS in preterm infants NMA included 16 different ventilation modes, yielding 120 comparisons- this probably requires an online appendix [9]. Ways to deal with this profusion of comparisons is to present effect estimates in a league table (all possible pairwise interventions compared to each other by cross-matching the interventions on the raw with those in the column), forest plots (all pairwise interventions compared to one reference intervention, or to the least efficacious intervention such as placebo), or evidence comparisons (direct, indirect, and NMA) for each intervention compared to one reference [13, 41, 43, 44].

RoB conveys the likelihood that limitations in design or conduct of studies will result in estimates of treatment effect that vary systematically from the truth. The greater the RoB, the more appropriate it becomes to rate down the quality of the evidence [45, 46].

For assessing the RoB, authors may use an instrument such as the Cochrane RoB tool for RCTs [38]. This instrument assesses six elements: randomization sequence generation, concealment of allocation, blinding of participants, personnel and outcome assessors, completeness of follow-up, selective outcome reporting, and presence of other biases.

In the NMA of strategies for preventing asthma exacerbations, the authors used the Cochrane instrument to assess RoB [12] and judged all trials to be at low RoB. Although the authors did not provide an overall RoB judgment per comparison, it is possible -although tedious- for the pediatrician to make this rating if the NMA authors have presented ratings of RoB for each study in a table or figure. In this case, it is not a problem: since all studies were at low RoB, there is no need to rate down for RoB for any comparison.

The lack of adequate power to inform a particular outcome leads to imprecision [47]. One standard for assessing precision is to consider whether differences between intervention and control exclude chance (i.e. statistically significant). This has two limitations: first results may exclude no effect, but may not exclude an effect too small to be important; second, using this criterion, one would always rate down for precision if results were not statistically significant, no matter how narrow the CI or CrI.

Therefore, we suggest an alternative standard. To assess imprecision, one can consider whether decisions regarding choice of therapy will differ if the upper and lower CI or CrI represents the truth. Another way of thinking about this approach is to consider whether the CI or CrI excludes a minimally important difference (MID). The MID is a measure of the smallest change in the value of a patient-reported outcome, typically applied to outcomes such as quality of life measures [48].

For example, in the NMA of ventilation modes for infants, the comparison of synchronized-intermittent mechanical ventilation with volume-guarantee (SIMV+VG) versus high-frequency-jet-ventilation (HFJV), the point estimate suggests that SIMV+VG reduced mortality (HR = 0.23) [9]. However, the 95%CrI ranged from an extremely large reduction in mortality (HR = 0.03, reduction in hazard by 97%) to an almost doubling of hazard (HR = 1.46). Since the treatment choice will be different at each CrI end, the evidence quality is reduced for imprecision.

On the other hand, for the comparison SIMV+VG versus SIMV with pressure-support ventilation (SIMV+PSV), mortality was lower with SIMV+VG (HR = 0.12; 95%CrI 0.01, 0.86). Here, even the upper suggests a 14% reduction in hazard with SIMV+VG. Therefore, in this instance, there is no need to rate down the quality of the evidence for imprecision. Although, the width of the CrI may still be considered large and thus could be considered imprecise for outcomes such as hospital length of stay, any but the smallest reduction in mortality is critical. The judgment of importance is critically dependent on the absolute difference, in this case the absolute mortality risk difference: for instance, for 27 weeks infants with baseline mortality risk of 10%, the absolute mortality risk reduction with SIMV+VG versus SIMV+PSV would approximately be 9% if the point estimate of the HR (0.12) were accurate, and approximately 1.4% if the upper boundary of the CrI (0.86) represented the truth. The magnitude of the absolute difference is greater for even younger infants with higher mortality, and less for older infants with lower mortality (Table 3) [49‐51].

In a complementary approach, authors can, for each direct comparison, assess imprecision by calculating the optimal information size (OIS), the number of patients or events needed for adequately powered individual study to avoid spurious findings [47]. This, however, ignores the contribution of the indirect comparisons to the network estimate. Methods to incorporate indirect estimates of OIS to NMA are under development [26].

Whenever a closed loop is present (Fig. 1.2, and Table 6) there is a possibility that the available direct and indirect comparisons will yield very different effect estimates, a condition we refer to as incoherence, or “inconsistency” used by other authorities [26, 27, 43, 53, 54]. Incoherence can arise for reasons similar to those that can explain heterogeneity and intransitivity (Table 4).

One can assess incoherence through inspecting the point estimates and the degree of CI or CrI overlap between direct and indirect evidence. In addition, investigators may conduct statistical tests that addresses whether chance can explain difference between direct and indirect comparisons [55, 56]. Unexplained incoherence requires rating down evidence quality.

In the asthma NMA, the direct evidence comparing ICS-L versus leukotriene receptor antagonists (LTRA) suggested a large reduction in exacerbation favoring ICS-L (OR = 0.38; 95%CrI 0.21, 0.68), and the network estimate showed a significant reduction (OR = 0.56, 95%CrI 0.39, 0.76) [12] – from which, one might infer that the indirect estimate showed a substantially smaller effect or, depending on the amount of indirect evidence, none at all. If the authors had provided the indirect estimate and its CrI, one could make the judgment regarding the degree of incoherence. The authors’ statement that they found no incoherence in the network on the basis of statistical tests is somewhat reassuring.

Like conventional meta-analysis, when heterogeneity is high, NMA can use techniques of subgroup analysis and meta-regression to try and explain heterogeneity by identifying modifiers of treatment effects [57, 58]. For example in the NMA addressing adverse events associated with antidepressant medications in children and adolescents [41], the OR for adverse events associated with sertraline use compared to placebo 2.94 (95%CrI 0.94,17.19, I² = 79.3). The authors performed a subgroup analysis based on age and found increased adverse events with sertraline compared to placebo; for children age < 13 years (OR = 12.64, 95%CrI 2.72, 678.43), and in children age > 13 years (OR = 0.59, 95%CrI 0.15, 6.03).

A somewhat less satisfactory way of exploring heterogeneity is to omit studies and determine if the omission influences results. For example, in the mechanical ventilation NMA, the authors examined the robustness of the analysis by excluding 2 studies that included only newborns with gestational age 25–26 weeks [9]. The results showed no changes in the effect estimates.

When direct and indirect evidence vary, and the network estimate is between the two and rated down for incoherence, what estimate is the clinician going to believe? The GRADE approach suggests using the effect estimates from the highest quality evidence, which most commonly will be the direct estimate [31]. Other authorities would argue that, having committed oneself to an NMA, one should always use the network estimates.

For example, the pediatric antidepressants medications NMA included a comparison of Fluoxetine versus Placebo (Table 7) [41]. In this comparison, one can infer from the information presented a rating of the quality of the direct evidence as very low, the indirect evidence as moderate, and the network estimate as very low quality. In this case, following the GRADE approach, the clinician is better off using the effect estimates from the indirect evidence.

Publication bias results from missing studies [59]. This is because some studies, particularly those with negative results, may never be published. A low risk NMA for publication bias will demonstrate comprehensive search for studies, present symmetrical funnel, and demonstrate insignificant statistical test for publication bias [38]. This assessment requires, however, at least 10 studies. If publication bias is very likely rating down the evidence is warranted.

Methods are available that allow NMA authors to rank treatments from best to worst [26, 60]. They are often expressed as probabilities that treatments are 1st, 2nd, 3rd etc. best, either in tables (Table 8) or graphically (rankograms). Surface under the ranking (SUCRA) summarizes the information from the rankograms as a single number. Ranking need be made for each outcome –a treatment that is best for one outcome (e.g. benefit) may be worst for another (e.g. harm) [60].

Although intuitively appealing, there are a number of reasons why clinicians should not routinely choose a treatment with the higher rankings [61]. First, a treatment that is best in one outcome (e.g., a benefit outcome) may be the worst in another (e.g., a harm outcome). Second, issues such as cost and a clinician’s familiarity with use of a particular treatment may also bear consideration. Third, rankings do not take into account the magnitude of differences in effects between treatments (a first ranked treatment may be only slightly, or a great deal better than the second ranked treatment). Fourth, chance may explain apparent difference between treatments; the use of a measure of uncertainty such as credible intervals for the SUCRA or p-value might help to consider the precision of these probabilities [62]. Finally, and most important, the evidence on which rankings are based may be very low quality, and therefore untrustworthy [61].

Although the first ranking may be secure, others are not: the asthma NMA showed that the treatment ranks for 2nd, 3rd, and 4th orders were ICS-L, ICS-H, ICS + LTRA (Table 8) for improving symptom-free days [12]. However, the probability for each treatment were close: 0.38, 0.33, 0.24 respectively, the NMA estimates were imprecise, and of low quality evidence. Therefore, the treatment ranks for the 2nd, 3rd, and 4th orders are untrustworthy.