Background
Assessment of the risk of bias (RoB) in included studies is an integral part of preparing Cochrane systematic reviews. Bias is any systematic error that can negatively affect the estimated effects of interventions and lead authors to wrong conclusions about efficacy and safety of analyzed interventions [
1].
Cochrane reviews use Cochrane’s RoB tool, whose aim is to enable better appraisal of evidence and ultimately lead to better healthcare [
2]. Cochrane’s standard RoB tool has seven domains. First domain addresses random sequence generation as a potential source of selection bias, assessing potentially biased allocation to interventions due to inadequate generation of a randomized sequence. Second domain analyzes allocation concealment, which can also lead to selection bias. The third domain is devoted to blinding of participants and personnel; it is associated with performance bias due to the knowledge of the allocated interventions by participants and personnel during the study. Fourth domain addresses blinding of outcome assessment; if done inadequately, it can lead to detection bias due to the knowledge of the allocated interventions by outcome assessors. Fifth domain analyzes the presence of incomplete outcome data, which can yield attrition bias due to amount, nature or handling of incomplete outcome data. The sixth domain is devoted to selective reporting, which can cause reporting bias due to selective outcome reporting. And finally, there is the seventh domain of Cochrane RoB assessment called “other bias”, which is used to note bias occurring due to any additional problems that were not covered by the first six domains [
3].
The Cochrane Handbook provides some examples of other potential threats to validity, such as design-specific risk of bias in non-randomized trials, baseline imbalance between groups of participants, blocked randomization in trials that are not blinded, differential diagnostic activity, study changes due to interim results, deviations from the study protocol, giving intervention before randomization, inappropriate administration of an intervention or having co-intervention(s), contamination due to drug pooling among participants, insufficient delivery of intervention, inappropriate inclusion criteria, using instruments that are not sensitive for specific outcomes, selective reporting of subgroups and fraud [
3].
This list of potential other sources of bias mentioned in the Cochrane Handbook is limited, and it would, therefore, be useful to explore potential additional sources of ‘other bias’. By consulting a more comprehensive list of potential other biases, the systematic review might recognize certain problems in included studies that might not otherwise consider a potential source of bias.
The aim of this study was to define which issues authors of Cochrane reviews describe as “other bias”, to determine the prevalence of various categories of other bias and to quantify qualitative data which support the assessment of other bias.
Discussion
In this study, we analyzed 768 Cochrane systematic reviews, with 11,369 included trials. We found that Cochrane authors used numerous different categories of sources of other bias and that they were not judging them consistently. We categorized different types of supporting explanations into 31 categories, and we found numerous other inconsistencies in reporting of sources of other bias in Cochrane reviews. Findings of this study are disconcerting because consistency in secondary research is very important to ensure comparability of studies.
Insufficient and unclear reporting of the ‘other bias’ domain was very common in the Cochrane reviews we analyzed. Among the most common support for judgment were comments that we categorized as ‘not described/unclear’, which is puzzling because ‘other bias’ domain is not specific like the other six domains of the RoB tool, and it is, therefore, difficult to fathom what it means that other bias was not described or that it was unclear. If the authors did not find sources of other bias, or if they thought that they could not assess other bias because of the brevity of report or language issues, they should have stated that. Likewise, for some trials, the only supporting explanation was that other bias was ‘Adequate’. Without any further explanations, readers cannot know what exactly the Cochrane authors found to be adequate in terms of other potential sources of bias. Many systematic reviews had a high number of included studies, and therefore some comments were repeated multiple times in the same systematic review.
The most commonly used specific category of other bias referred to baseline characteristics of participants. In RCTs, randomization should ensure allocation of participants into groups that differ only in intervention they received. Randomization should ensure that the characteristics of participants that may influence the outcome will be distributed equally across trial arms so that any difference in outcomes can be assumed to be a consequence of intervention [
4]. Baseline imbalances between the groups may indicate that there was something wrong with the randomization process, or that they might be due to chance [
5]. Severe baseline imbalances can occur because of deliberate actions of trialists if they aim to intentionally subvert the randomization process [
6] or due to unintentional errors.
Chance imbalances should not be considered a source of bias, but it may be difficult to distinguish whether baseline imbalances are caused by chance or intentional actions. If there are multiple studies included in a meta-analysis, it could be expected that chance imbalances will act in opposite directions. But the problem may occur if there is a pattern of imbalances across several trials that may favor one intervention over another, suggesting imbalance due to bias and not due to chance [
7]. Cochrane is now developing a second generation of the RoB tool, titled RoB 2.0, and one of the signaling questions in the RoB domain about randomization process asks “Were there baseline imbalances that suggest a problem with the randomization process” [
7]. The fact that so many Cochrane authors used comments about baseline imbalance as a domain of other bias, and not in the RoB domain about random sequence generation (selection bias) indicate that many Cochrane authors consider that this aspect should be emphasized separately from the selection bias domain.
The second most commonly used category of supporting explanations was related to funding of a trial, and comments about conflicts of interest were the fifth most common category. This is in direct contrast with the recommendations from the Cochrane Handbook, where it is acknowledged that information about vested interests should be collected and presented when relevant, but not in the RoB table; such information should be reported in the table called ‘Characteristics of included studies’ [
8]. RoB table should be used to describe specific methodological aspects that may have been influenced by the vested interest and directly lead to RoB [
8]. Therefore, it is obvious that the authors of the Cochrane Handbook assume that the influence of sponsors can be mediated via other domains of RoB tool such as selective reporting of favorable outcomes.
However, Lundh et al. have published a Cochrane review in 2017 about industry sponsorship and research outcomes, in which they included 75 primary studies, which shows that commercial funding leads to more favorable efficacy results and conclusions compared to non-profit funding [
9]. They concluded that industry sponsorship introduces bias that cannot be explained by standard domains of Cochrane’s RoB assessment [
9]. The debate about whether funding presents the source of bias or not is ongoing in the Cochrane, with some considering that commercial funding is a clear risk of bias, while others argue against such standpoint [
10,
11]. This debate apparently reflects the current situation in which many Cochrane authors continue to use funding and conflict of interest as a source of other bias despite the official warning against such use of information about sponsorship from the Cochrane Handbook, as we have demonstrated in this study.
The third most frequent category of supporting explanations for other bias was related to poor reporting, where Cochrane authors indicated that relevant information was missing or were inadequately reported. Poor reporting hinders transparency, as it allows authors to avoid attention to weak aspects of their studies. For this reason, reporting guidelines should be used [
12].
Comments about sample size were the fourth most common category either in a sense that the trial did or did not report sample size calculation, or that sample size was “small” without any further explanation of what the Cochrane authors considered to be a small sample. There were 21 trials for which Cochrane authors wrote that there were fewer than 50 participants in each arm. It is unclear where this cut-off is coming from, as there is no such guidance in the Cochrane Handbook in the chapter about the risk of bias. On the contrary, chapter 8.15.2. of the Cochrane Handbook specifically warns that “sample size or use of a sample size (or power) calculation” are examples of quality indicators that “should not be assessed within this domain” [
8].
The Cochrane Handbook also warns that authors should avoid double-counting, by not including potential sources of bias in the ‘other bias’ domain if they can be more appropriately covered by other domains in the tool [
8]. As can be seen by our study, Cochrane authors sometimes do double-counting because there were categories of comments supporting judgments that could have been addressed in the first six domains.
As we have shown, most Cochrane authors decided to use the other bias domain to describe potential additional biases that were not covered in the first six domains of the RoB tool. In the proposed RoB tool 2.0 there is no ‘other bias’ domain [
7]. The proposed RoB tool is much more complex, compared to the current version of the RoB tool, and many items that were specifically emphasized by Cochrane authors in the other bias domain, as shown in our study, are addressed in the RoB 2.0 tool. However, there are still potential biases from other sources that the RoB 2.0 may neglect by omitting the RoB domain for other bias. Relevant other bias that were identified in our study include, for example, problems with inclusion and exclusion criteria, data analyses, outcome domains and outcome measures that were used, usage of co-interventions that are not accounted for, deviations from the protocol, study design, issues related to specific types of trials such as cross-over trials and biases specific to other to certain topics. Therefore, we believe that there is a rationale for including ‘other bias’ domain in revised RoB tool too.
We have already conducted a similar analysis of Cochrane RoB domain related to other RoB domains, and we found that judgments and supports for judgments in those domains were very inconsistent in Cochrane reviews [
13‐
15]. This analysis related to sources of other bias in Cochrane reviews contributes to the perception that Cochrane RoB tool is inconsistently used among Cochrane authors. The authors do not necessarily follow guidance from the Cochrane Handbook. In the support for judgment, they mention issues that the Cochrane Handbook explicitly warns against. Various comments that serve as supports for judgments were inconsistently judged across Cochrane reviews and trials included in those reviews. Cochrane authors also use inconsistent terminology to describe the same concepts. Increasing complexity of the RoB tool, as proposed in the RoB tool 2.0 will likely only increase this problem of insufficient consistency in RoB appraisal and worsen this problem of insufficient comparability of judgments of RoB across Cochrane reviews.
Furthermore, our study indicated that Cochrane authors extensively use the available option to customize the RoB table. We found that there were as many as 102 (13%) out of 768 analyzed Cochrane reviews that did not use the other bias domain in the RoB table at all. Cochrane reviews are produced using the software Review Manager (RevMan). As soon as an author inserts a new study in the RevMan among included studies, an empty RoB table for the study automatically appears, with seven pre-determined domains. Therefore, Cochrane authors need to intentionally remove or add some domains if they want to customize the RoB table. Among 102 Cochrane reviews that did not have other bias domain, 33% of those reviews had comments about other potential sources of bias in the body of the manuscript. It is unclear why some Cochrane authors use only text for comments about other bias instead of using RoB table for this purpose. Additionally, we observed that in many Cochrane reviews without other bias domain there were other customizations of the RoB table, which had from one to six other, standard RoB domains included. Exactly half of those reviews without other bias domain in the RoB table had less than six standard domains in the RoB table.
Results of this study can contribute to better reporting of future systematic reviews and help authors of systematic reviews to avoid mistakes. Firstly, results of this manuscript will provide more comprehensive information for Cochrane authors regarding ‘other bias’ domain – we present many sources of other bias that Cochrane authors recognize, and that are not mentioned in the Cochrane Handbook. Secondly, we showed mistakes that Cochrane authors are doing when they mention in ‘other bias’ domain issues that actually belong to other six domains of Cochrane RoB tool. Thirdly, we are also pointing out mistakes that Cochrane authors are doing despite explicit instructions from the Handbook, i.e. authors use sample size and funding to comment about potential bias, even though the Handbook explicitly warns against this. Although our study was focused only on Cochrane reviews, our results are relevant also for non-Cochrane reviews that use Cochrane’s risk of bias tool. Therefore, our manuscript can help authors of Cochrane and non-Cochrane reviews to create better and more consistent reviews, to recognize additional potential sources of bias in trials they analyze, and to avoid mistakes that we have observed.
Limitation of our study is that we included in our analysis a limited number of analyzed Cochrane reviews, which were published in 2015 and 2016. We chose this convenience sample of Cochrane reviews because we were interested in the state of the ‘other bias’ domain in recent times; we did not aim to analyze the change of this domain over the very long time period. However, considering the number of Cochrane reviews analyzed, and the number of inconsistencies we observed, we have no reason to suspect that the results would be significantly different if a bigger cohort of published Cochrane reviews would have been used. It takes a long time to manually extract, check, analyze and categorize more than ten thousands of RoB domains, and therefore using the same methodology on a larger sample might not be feasible. It is possible that some unintentional errors in categorizations may have been made, and therefore, for transparency, we decided to present all categories and sub-categories of the supporting explanations we encountered in the Additional files
1,
2 and
3. Additionally, all systematic reviews are not the same and our findings cannot be generalized to all systematic reviews – we analyzed only Cochrane systematic reviews of RCTs because Cochrane RoB tool was developed for these types of studies. However, we believe that our findings can be very useful also for authors of non-Cochrane reviews who will use Cochrane RoB tool in their methodology.
Finally, it is worth emphasizing that it is possible that some trials from our cohort were included in more than one review, and that Cochrane authors could give them different judgments for ‘other bias’. It has been shown before that authors of different reviews can make different RoB judgments of the same trials [
16]. However, such analysis was not the aim of our study.