Empirical evidence of study design biases in nutrition randomised controlled trials: a meta-epidemiological study

The sample consists of 33 SRs of RCTs which were identified through a previous meta-epidemiological study [12]. For this project, we searched for SRs of RCTs in the Cochrane Database of Systematic Reviews published between 01 January 2010 and 31 December 2019, which investigated the effects of nutritional interventions (e.g. dietary pattern, micronutrients) on patient-relevant outcomes (e.g. mortality, cardiovascular disease [CVD]). Details of the inclusion and exclusion criteria, as well as the search strategy, are displayed in Additional file 1: Appendix S1-S2.

For each identified Cochrane review, we chose a maximum of six outcomes (maximum three binary and three continuous outcomes) for each given intervention, based on the ranking in the Summary of Findings tables (from top to bottom). Furthermore, we selected only the meta-analyses that contain at least one trial judged as ‘low risk of bias’ and another for ‘unclear or high risk of bias’ for a specific methodological characteristic according to the RoB assessment provided by the original SR authors.

We extracted the following data for each included Cochrane review: name of the first author, year of publication, type of intervention, description of outcome, effect estimates (risk ratio [RR] or mean difference [MD] with the corresponding 95% confidence interval [CI]), and the number of included RCTs. For continuous outcomes, we calculated the standard deviations [SD], and the effect measures were converted to standardised mean differences [SMD] with the transformation formula SMD = MD/SD.

For each RCT included in the eligible meta-analyses of each considered Cochrane review, we additionally extracted all effect estimates with their corresponding 95% CI and the judgements (low, unclear, high) for each methodological trial characteristic of the Cochrane tool [6] (random sequence, generation and allocation concealment, blinding, incomplete outcome data, and selective reporting) as provided by the original SR authors. Although dietary compliance was not suggested in the RoB domains of the Cochrane tool, we extracted judgements if available, due to its importance for nutrition RCTs [13]. Judgements for each methodological trial characteristic were extracted from the risk of bias summary figure and/or the ‘characteristics of studies’ tables in the appendix.

Two authors extracted data using a piloted standardised data collection form. Data extraction was performed by one reviewer (IR) and checked by a second reviewer (JS). Divergences were resolved by consensus, or by consulting a third reviewer (LS).

We standardised the direction of effect of the outcomes so that binary effect estimates < 1 (or continuous outcomes < 0) are always expressing a beneficial effect.

Relative effect estimates (quantified as the ratio of risk ratios [RRR] or difference in standardised mean differences [DSMD]) between RCTs with high/unclear and those with low RoB were measured for each methodological trial characteristic. To do so, we re-calculated the effect estimate [RR or MD] for each group of trials included in the same meta-analysis, by pooling first only those RCTs which were judged to be at low RoB according to the methodological trial characteristic under investigation and second only those RCTs with an unclear or high RoB judgement (see Additional file 2: Fig. S1). By using RCTs with low RoB judgements as the reference group, we examined the pooled estimate to see whether there was a relatively larger or smaller estimate coming from RCTs with high/unclear RoB judgements. For example, a RR from high/unclear RoB RCTs of 0.95 and a RR from low RoB RCTs of 0.90 result in a RRR of 1.06, whereas a RR of 1.00 in low RoB RCTs compared to a RR of 1.06 in high/unclear RoB RCTs is also a RRR of 1.06. Therefore, the RRRs should not be interpreted as larger or smaller treatment effects in one RoB category, but only as differences between the two RoB categories, and the direction of difference depends on the direction of effect of the underlying RCTs.

RCTs with high and unclear RoB judgements were grouped together, since the number of available judgements was small (Table 1). This approach was in line with landmark meta-epidemiological studies [8, 10, 14]. We performed meta-analyses to synthesise the relative effect estimates (average bias). Statistical analyses were conducted across all outcomes and for each methodological characteristic independently. When methodological trial characteristics were defined differently across the Cochrane reviews (e.g. ‘blinding’ in general), we excluded these from the analyses. We performed subgroup analyses with respect to the different types of interventions (e.g. dietary approach, fatty acids, micronutrients), and types of outcomes (e.g. mortality, cardiovascular disease, cancer). Moreover, we performed subgroup analyses comparing the objective (e.g. all-cause mortality) versus pregnancy outcomes versus mostly subjective outcomes according to Savovic and colleagues [9, 10].

We also performed a sensitivity analysis excluding the highly correlated outcomes (Additional file 3: Tables S1-S7). The determination of highly correlated outcomes was based on experts’ opinions and is in line with our previous study [12]. For each intervention, we only chose the outcome that was mentioned first from the Summary of Findings tables of the identified Cochrane reviews. For example, for the intervention of α-linolenic acid, the outcomes CVD, CVD mortality, and coronary heart disease are probably highly correlated. CVD mortality outcome was chosen as it was mentioned first in the Summary of Findings table, while the other two outcomes were excluded. In a post hoc analysis, we conducted a sensitivity analysis including only one outcome per comparison from each included SR. We chose the outcome with the largest number of RCTs included. In case there were two outcomes with the same number of RCTs, we chose the one that was ranked higher in the Summary of Findings table.

We used the random-effects model in meta-analysis to take into account the between-study variability (heterogeneity). We assessed heterogeneity with I² and τ² measures [15, 16]. The τ² statistic was estimated by the restricted maximum-likelihood (REML) method [17], which performs well for both binary and continuous outcomes [18‐20]. We also calculated 95% prediction intervals (PI) to provide the range of possible values for the difference between the characteristics of RCTs that might be observed in future comparisons.

We performed post hoc multivariable meta-regressions to account for confounding variables such as sample size and conflicts of interest. For each methodological trial characteristic, we investigated whether ‘sample size’ or reported ‘conflicts of interest’ are associated with the differences in the effect estimates across the groups of low and high/unclear RoB RCTs.

All statistical analyses were conducted using the R package meta (version 5.2–0) [21].

Of the 33 SRs of RCTs identified in our previous study [22‐54] including 97 meta-analyses, five SRs [25, 32, 35, 41, 49], including 11 meta-analyses, were excluded since they provide only one RCT for any eligible outcome. Moreover, we excluded nine meta-analyses in five eligible SRs [29, 44, 48, 51, 53] since comparisons were not possible for any methodological trial characteristic. Overall, 27 SRs of RCTs (Cochrane reviews) [22‐24, 26‐31, 33, 34, 36‐40, 42‐48, 50, 52‐54] with 77 meta-analyses (n = 927 RCTs) were included. The details of the selection process are displayed in Fig. 1 and Additional file 3: Tables S8-S9 [55‐86].

The meta-analyses included a median of ten RCTs (range: 2 to 64). The interventions evaluated in the identified meta-analyses can be categorised into micronutrients (n = 36), fatty acids (n = 16), dietary approach (n = 14), food groups (n = 5), fibre (n = 4), and food (n = 2).

The intervention was administered in the form of dietary supplements (n = 31), dietary intake (n = 30), or both (n = 16). The outcome clusters included intermediate disease markers (n = 21), CVD (n = 18), pregnancy outcomes (n = 17), all-cause mortality (n = 12), cancer (n = 6), bone health (n = 2), and eye disease (n = 1). Additional file 3: Tables S10-S11 presents the characteristics of all meta-analyses in our sample.

All included SRs assessed the RoB of the primary studies using the criteria of the original version of the Cochrane tool [6]. Among the 77 meta-analyses, 60 were informative for ‘random sequence generation’, 69 for ‘allocation concealment’, 39 for ‘blinding of participants and personnel’, 49 for ‘blinding of outcome assessment’, 60 for ‘incomplete outcome data’, 36 for ‘selective reporting’, and 16 for ‘dietary compliance’. Overall, we included 52 meta-analyses on binary outcomes and 25 on continuous outcomes (Additional file 3: Table S12).

For ‘random sequence generation’, 618 trials (74.4%) included in the meta-analyses were judged as having low RoB by the original review authors (see Table 1). For ‘allocation concealment’, there were 541 trials (58.5%) with a low RoB judgement, for ‘blinding of participants and personnel’ 264 trials (61.4%), ‘blinding of outcomes’ 348 trials (69.6%), ‘incomplete outcome data’ 583 trials (66.9%), and ‘selective reporting’ 369 trials (65.1%). The proportion of trials judged as being at low RoB was lowest for the ‘dietary compliance’ domain (50.9%, 119 trials). In the comparison group enclosing unclear and high RoB trials, most trials (51.6–96.4%) were judged as having an unclear RoB by the original review authors. Most of the trials with a high RoB were found for ‘blinding of participants and personnel’ (78 trials, 18.1%) and ‘incomplete outcome data’ (140 trials, 16.0%).

Tables 2 and 3 present the results of the association of the reported characteristics on intervention effect estimates, from the main analyses, subgroup analyses, and sensitivity analyses. All meta-analyses conducted are illustrated with forest plots in Additional file 2: Figs. S2-S47.

This meta-epidemiological study of 77 meta-analyses synthesised evidence on the average bias associated with reported methodological trial characteristics such as random sequence generation, allocation concealment, blinding, incomplete outcome data, selective reporting, and dietary compliance of RCTs in the nutritional field. The main findings suggest that most characteristics of nutrition RCTs considered in our sample may not exaggerate the average intervention effect estimates. However, we observed that intervention effect estimates may exaggerate on average by 18% in trials rated at high/unclear (versus low) RoB for blinding of outcome assessment. For this methodological trial characteristic, the average bias appears to be larger for subjective outcomes, whereas for all-cause mortality and pregnancy outcomes, no association was observed. Moreover, high/unclear (versus low) RoB judgements for incomplete outcome data may exaggerate intervention effect estimates on average by 9%, which was confirmed in various subgroup and sensitivity analyses. Overall, the statistical heterogeneity was low, which was confirmed by a narrow 95% PI. For a pooled estimate of continuous outcomes, no differences were observed between high/unclear (versus low) RoB methodological trial characteristics, except for selective reporting in the main analysis.

We did not identify any similar empirical study investigating the impact of study design biases on intervention effects across meta-analyses of nutrition RCTs. However, several meta-epidemiological studies have been conducted in the medical research field so far. Savovic and colleagues [14], for instance, showed that intervention effect estimates seemed to be exaggerated in trials with inadequate/unclear (versus adequate) sequence generation (by 11%), allocation concealment (by 7%), or double-blinding (by 13%). This exaggeration was, however, not statistically significant for mortality and other objective outcomes, which is in line with our findings. Similarly, Page and colleagues [8] synthesised evidence from 24 meta-epidemiological studies and observed that certain characteristics of RCTs such as sequence generation, allocation concealment, and double blinding may exaggerate the intervention effect estimates for subjective compared with objective outcomes. Moreover, inadequate/unclear blinding of the outcome assessor (versus adequate blinding) may exaggerate the intervention for subjective outcomes, which is in line with our findings. Regarding the detected exaggeration of intervention effects observed in RCTs with high or unclear RoB judgements for incomplete outcome data, previous studies reported inconsistent findings: Savovic and colleagues [10] found little evidence that intervention effects were exaggerated in trials which were judged to be at high or unclear RoB for incomplete outcome data, whereas Abraha and colleagues [87] observed an exaggeration.

In congruence with our findings, two small meta-epidemiological studies concluded that there was no convincing evidence that trials rated at high/unclear (versus low) risk of bias due to selective reporting have different effect estimates [88, 89].

A recent meta-epidemiological study investigated the impact of blinding and found no evidence for an average difference in estimated treatment effect between trials with and without blinded patients, healthcare providers, or outcome assessors [90]. This study result is in agreement with our findings on double blinding of participants and personnel but in disagreement with our findings on blinding of outcome assessment for subjective outcomes.

Systematic evidence syntheses based on high-quality evidence are highly needed to generate trustworthy dietary guidance. Ideally, these should base on direct evidence from RCTs with a low risk of bias. In the field of nutrition research, however, trialists face a number of methodological challenges that are difficult to resolve [13]. Whereas certain trial characteristics such as random sequence generation, allocation concealment, and blinding of outcome assessment can be performed for most RCTs, irrespective of the research field, others are context-specific and are more difficult to incorporate such as blinding of participants and personnel, incomplete outcome data, and dietary compliance. For instance, people are in general aware of what they consume. Therefore, it is often impossible to blind study participants, except for RCTs with dietary supplements. Moreover, longer-term nutrition research trials often face attrition of participants (40–50% dropout rate is fairly common in free-living populations [91]) and low compliance to a specific dietary regimen is often observed [92].

Nonetheless, in our meta-epidemiological study, we observed that on average, most characteristics of nutrition RCTs may often not exaggerate intervention effect estimates. Of note, our sample is mainly based on interventions with dietary supplements where blinding is possible and more than 60% of the included RCTs were judged as low RoB for blinding of participants and personnel. Moreover, two-third of the RCTs in our sample were judged with low RoB for incomplete outcome data. Attrition seems to be lower in interventions with supplements, as the lifestyle behaviour change introduced by the intervention is small [93].

Regarding dietary compliance, about half of the RCTs (n = 119) included in our sample were judged with a low RoB, and all interventions based on the supplementation (and partly on dietary intake) of fatty acids or micronutrients. Surprisingly, RCTs with high/unclear RoB for dietary compliance showed no differences compared to low RoB trials. This could possibly be explained by the fact that most of the included RCTs were rated as ‘unclear’ (n = 100) and only few (n = 15) at ‘high’ RoB. Furthermore, the number of included studies which were informative about dietary compliance was small. Dietary compliance is not a specific domain in the original version of the Cochrane tool [6]. The authors of the Cochrane reviews included in this study assessed dietary compliance by introducing a new RoB domain [23, 40], within ‘other sources of bias’ [22, 38] or did not provide any details on the assessment in the methods Sect. [24]. Moreover, definitions of ‘dietary compliance’ differ between the included reviews: some authors, for instance, based their judgement on thresholds, whereas others on reported differences in intake or biomarkers between the control and treatment groups (see Additional file 3: Table S26). In the most recent version of the Cochrane tool (RoB 2.0), however, compliance to a specific dietary intervention will be assessed within the domain ‘assessing deviations from intended interventions’ [94].

In order to highlight the methodological weaknesses of included RCTs and address the study design bias, it is crucial for the authors of SRs to take RoB assessments into account in the statistical analysis, in the GRADE assessment, and conclusions of the SRs.

This study has several strengths. First, to the best of our knowledge, no similar study has been conducted in the nutritional field so far. Second, we analysed a large sample of diet-disease effects (n = 77 meta-analyses). Third, we selected meta-analyses of RCTs published as Cochrane reviews, which are internationally recognised as the highest standard in evidence-based healthcare. The high methodological quality of Cochrane nutrition reviews was confirmed recently [95]. Fourth, our study was based on meta-analyses of both binary and continuous outcomes.

However, our study has also several limitations: First, although we pooled a large sample of diet-disease associations, our sample may not be representative of all meta-analyses, and the totality of the evidence of available diet-disease associations might provide different results. Second, some trials may be included multiple times in the main analysis due to the different outcomes. However, in our post hoc sensitivity analyses, we also took a conservative approach by including only one outcome per comparison from each included SR. These sensitivity analyses confirmed all the findings of the main analysis. Third, we did not use the most recent version of the RoB tool [94] since we extracted the RoB judgements as provided by the original review authors. The usage of this tool, which was published in 2019 [94], would have made it impossible to generate an adequate sample. Additional considerations added to the revised tool were thus not considered for our analysis. For example, differences in baseline characteristics or the effect of adhering to intervention are not addressed by the original tool [6]. Fourth, we did not verify the RoB assessment of the included Cochrane reviews including the judgements made for ‘dietary compliance’. Finally, the RoB assessment is based on the reported methodological characteristics of the included primary studies. It remains however unclear, whether trials that were classified as ‘unclear’ might have been in fact adequately conducted. Since we merged both, RCTs with high and unclear risk of bias, we might have misclassified some of the trials which might be at low risk of bias. Due to these limitations, our findings need to be interpreted with caution.