Risk of bias judgments for random sequence generation in Cochrane systematic reviews were frequently not in line with Cochrane Handbook

Randomised controlled trials (RCTs) are considered crucial for objectively testing the efficacy and safety of interventions [1]. They are often used to inform clinical practice and included in systematic reviews to obtain an even higher level of evidence through evidence synthesis. However, a recent systematic review of meta-epidemiological studies indicated that estimates about effects of interventions might be exaggerated in RCTs with inadequate or unclear sequence generation and blinding [2].

Therefore, RCTs can be biased if flaws in their design and conduct lead to overestimation or underestimation of intervention effect estimates. Because of this potential for bias, assessment of the risk of bias (RoB) in trials is one of the usual methodological steps during the preparation of Cochrane systematic reviews. Bias has been defined as any systematic error that can negatively impact estimated effects of interventions and lead to incorrect conclusions about the efficacy and safety of analysed interventions [3].

In Cochrane’s RoB tool, systematic review authors are expected to provide judgment about the level of RoB as low, unclear or high for seven different potential domains of bias: selection bias (biased allocation to interventions) due to inadequate generation of a randomised sequence; selection bias (biased allocation to interventions) due to inadequate concealment of allocations prior to assignment; performance bias due to knowledge of the allocated interventions by participants and personnel during the study; detection bias due to knowledge of the allocated interventions by outcome assessors; attrition bias due to amount, nature or handling of incomplete outcome data; reporting bias due to selective outcome reporting; and other bias due to problems not covered elsewhere. Cochrane authors should provide their RoB assessments in the RoB table, where the assessment for each domain needs to contain two sections: a judgment (i.e., the risk is low, unclear or high) and an accompanying supporting comment that needs to justify the judgment. The first domain analysed in the Cochrane RoB tool is assessing the process of generating a randomisation sequence as a part of the assessment of selection bias [4].

A recent report analysed the evolution of reporting and inadequate methods over time in 20,920 RCTs included in Cochrane reviews [5]. The authors analysed data from RCTs included in all Cochrane reviews published between March 2011 and September 2014, which reported an evaluation of the Cochrane RoB items, including sequence generation. The results indicated that unclear risk for sequence generation was found in 49% of trials, high risk in 4% of all trials and low risk in 48% of trials included in the analysed Cochrane reviews [5]. Additionally, it was found that the proportion of trials with unclear RoB for sequence generation decreased over time, falling from 69% in the 1986–1990 cohort of trials to 31% in the 2011–2014 cohort. The proportion of trials with high RoB for sequence generation fell from 4.6% in 1986–1990 to 3.2% in 2011–2014 [5].

However, it is possible that the way the Cochrane authors judge RoB for sequence generation is highly variable. We have already proved this for RoB domains for allocation concealment [6], blinding of participants and personnel [7], incomplete outcome data [8], selective reporting [9], and other bias [10]. In that case, results presented in the study of Dechartres et al. [5] or similar studies would not be based on consistent ratings of RoB in Cochrane reviews, and improvements shown for certain RoB domains could be misleading.

The objectives of this study were two-fold: to evaluate the rationales based on which judgments related to random sequence generation were made for trials in Cochrane reviews, and to investigate the proportion of inadequate judgments about randomisation based on independent re-assessments using guidance from the Cochrane Handbook.

We analysed 10,290 RCTs included in 718 Cochrane reviews. During analysis, we had to exclude 187 RCTs and 14 reviews (Fig. 1). We excluded 19 RCTs due to customised risk judging (e.g. risk of bias judged as “moderate”). An additional 112 trials were excluded because of missing RoB table or domain associated with random sequence generation. There were 49 RCTs in which we found no specific reason for the missing RoB or domain, while 54 trials were not randomised, three webpage errors caused duplicate entries and six studies only had abstracts (Table 1). We also excluded 56 RCTs with the empty field or N/A (representing “not available”) in the supporting comment cell of the RoB table.

Analysis of judgment and comments about the risk of bias associated with random sequence generation in 10,103 RCTs included in 704 Cochrane reviews indicated that Cochrane authors do not always adhere to recommendations regarding assessment of this particular type of risk of bias and that 12% of judgments for bias associated with this domain were not in line with the recommendations from the Cochrane Handbook. This also implies that one in eight judgments of bias regarding randomisation of participants in Cochrane reviews was not elaborated or that the supporting comment was not detailed enough. The most common discrepancies were observed for the category of trials judged as having high RoB for sequence generation, where 28% of the judgments were not supported with the explanations given in the accompanying supporting comments for the judgment. To the best of our knowledge, this is the first study evaluating rationales for and accuracy of judgments for RoB associated with random sequence generation in Cochrane systematic reviews.

The most frequent discrepancy, according to the parent category, was judging RoB for randomised sequence generation as low when the method of randomisation was not sufficiently described. The most frequent of such supporting comments were related to the mention of central randomisation without further details, baseline balance between the groups where the Cochrane authors assumed that the randomised sequence generation was adequate, self-standing comments about block randomisation or stratification and supporting comments about using envelopes without further details.

Cochrane Handbook explicitly warns: ″Sometimes trial authors provide some information, but they incompletely define their approach, and do not confirm some random component in the process. For example, authors may state that blocked randomization was used, but the process of selecting the blocks, such as a random number table or a computer random number generator, was not specified. The adequacy of sequence generation should then be classified as unclear″ [11].

Cochrane has recently proposed A revised tool to assess risk of bias in randomized trials (RoB 2.0) [12]. In the RoB 2.0 version, baseline characteristics of participants are featured prominently. The RoB 2.0 uses baseline imbalances to signal problems with the randomisation process, and one of the signalling questions in the RoB domain about randomisation process is ″Were there baseline imbalances that suggest a problem with the randomization process?″ [12].

We found 98 instances where random number generation were mentioned without remarks about using a computer, electronic calculators, etc. This category was judged as low risk in 100%, and we left it in the computer parent category, although it did not fulfil strict Cochrane Handbook rules. Otherwise, erroneous levels would rise from 12 to 13% in total and from 20 to 22% for trials judged as having a low risk of bias.

Our findings shed light on the certainty of evidence and reliability of conclusions in Cochrane reviews and a number of meta-epidemiological studies that were based on the RoB assessments from Cochrane reviews. RoB assessment is regularly mentioned in conclusions of Cochrane reviews. It is very common in Cochrane reviews to read that included studies were of high risk of bias, and therefore their results are less reliable. Cochrane authors may conclude that certain findings are based on studies with low RoB, but if one in five of those RoB judgments is flawed, then the conclusions of reviews may be severely compromised. The readers of such reviews get the message that certain recommendations rely on evidence with low RoB and therefore evidence that is more reliable and trustworthy.

In previous studies, authors appear to assume that RoB judgments in Cochrane reviews were apropriately applied and various conclusions that were reached were related to those judgments [5, 13]. A recent study reported that poor reporting and inadequate methods have decreased over time, particularly for sequence generation and allocation concealment, based on the analysis of 20,920 RCTs included in Cochrane reviews [5]. However, our study indicated that this result does not have to be due to better reporting and better methods, but errors in judgment of Cochrane authors.

In our study we found that more than half of the trials included in our sample of Cochrane reviews had unclear risk of bias for generating randomisation sequence, which is in line with a report by Kahan et al., who reported that risk of selection bias is difficult to ascertain in the majority of trials because of poor reporting [14].

Likewise, studies that analysed the association between RoB and effect of interventions could have reached inappropriate conclusions because they trusted the judgment of systematic review authors. Savovic et al. reported that estimates of intervention effects were exaggerated by an average of 11% in clinical trials with inadequate or unclear sequence generation. Their results were based on analysis of 1,973 trials included in 234 meta-analyses [13].

According to the Cochrane Handbook, Cochrane authors can make assumptions but they need to elaborate on them [4]. We considered assumption as the most important factor for such a high proportion of unsupported judgments. We found supporting comments where Cochrane authors just wrote assumed, and judged the trial as having a low risk of bias related to randomisation without explaining why they assume that risk is low.

A limitation of our study is a confined time period in which analysed Cochrane reviews were published. However, we analysed a high number of Cochrane reviews, with a high number of included trials, and these Cochrane reviews were published recently. Therefore, we believe that they are representative of the current state of reporting of the analysed domain in Cochrane systematic reviews. Another limitation is a possible human error due to the interpretation of Cochrane Handbook and manual categorisation of automatically extracted text. To prevent this error, every categorisation made by one author was checked by the first author. Also, we acknowledge that our subcategories, created during manual categorisation, are not recognised by the Cochrane RoB tool. However, we pointed them out just to show the most common mistakes that authors make without entering an in-depth analysis. A single comment can sometimes be subcategorised and the decision for sorting a comment into a subcategory was arbitrary because no ranking on this level was defined. However, this did not change the results of this study because the comment remained inside the same parent category.

Furthermore, we acknowledge the possibility that RoB judgments may have been correct based on all the information available to Cochrane authors, but since the authors did not provide enough details in supporting comments, their judgments appear inadequate. In this study we did not attempt to analyse information available in trials included in Cochrane reviews and to compare whether Cochrane authors may have had more information available than shown in a supporting comment. Nevertheless, our study is useful in pointing out inadequate reporting in RoB tables, so that in the future authors using Cochrane RoB tool may pay more attention to detailed justifications of their judgments.

It has already been shown that the Cochrane RoB has low reliability between individual reviewers, as well as across consensus assessments of reviewer pairs [15]. Da Costa et al. argued that the low reliability of the RoB assessment in systematic reviews can have detrimental effects on decision making and healthcare quality [16]. Interventions such as standardised intensive training on RoB assessment were tested, and the results indicate that such interventions can significantly improve the reliability of the Cochrane RoB tool [17]. Apart from author training, other solutions for improving the reliability of RoB judgments would be more stringent peer reviews and editorial assessments of judgments and supporting comments in the Cochrane RoB table.

Additionally, we analysed only Cochrane reviews, which use Cochrane RoB tool. Recent studies have indicated that the RoB instrument did not adequately capture the risk of bias in RCTs [18, 19]. Future studies on Cochrane RoB 2.0 tool are warranted to see how the new tool compares to the current one.

Authors of Cochrane systematic reviews sometimes make inadequate judgments about the risk of bias related to random sequence generation. Most of these judgments were not elaborated or supported by comments in sufficient detail. Subsequently, this might compromise the reliability of their conclusions. Our results can help authors of both Cochrane and non-Cochrane reviews which use Cochrane RoB tool to avoid making common mistakes when assessing RoB in included trials. Even though systematic reviews are considered the highest level of evidence, our study shows that these types of studies should also be scrutinised to make sure that their every aspect is trustworthy. Interventions for improving the reliability of Cochrane risk of bias assessment should be considered.