Citation bias favoring positive clinical trials of thrombolytics for acute ischemic stroke: a cross-sectional analysis

We analyzed citation rates from eight major trials of intravenously administered tPA and seven major trials of other thrombolytic agents for the treatment of acute ischemic stroke. Among the tPA trials, those showing a statistically significant benefit from tPA were cited more than three times as often as neutral trials and nearly 10 times as often as negative trials. Positive trials involving other thrombolytic agents were also cited far more frequently than neutral or negative trials. These results demonstrate the presence of substantial citation bias.

While the factors influencing individual author decisions about what studies to cite are likely quite varied, the net effect of these various influences with respect to tPA and other thrombolytics is to disproportionately cite positive trials relative to neutral and negative trials. Over time, this phenomenon has the potential to distort the medical literature in several important ways. First, if clinicians and policy-makers do not utilize a systematic approach when reviewing the medical literature on a given topic they are likely to be exposed to a biased subset of all the potentially relevant trials. Second, as the number of MEDLINE-indexed publications continues to grow, the sheer number of included manuscripts will make it more and more difficult to perform comprehensive medical literature searches based on keywords alone. As a result, readers are increasingly likely to utilize citation webs as part of their medical literature search strategies, potentially exacerbating any existing citation bias. For example, the ranking algorithm from Google Scholar puts great weight on the number of citations [26]. Additionally, citation counts are often used as markers of the quality and impact of published research, further emphasizing the results of highly cited, positive trials as compared to those with fewer citations [27, 28].

When possible, preferential citation of high-quality systematic reviews which utilize and report meticulous search methods may help to limit citation bias due to the selective citation of individual trials. In particular, meta-analyses using individual patient data allow for assessments of both efficacy and safety outcomes with applicability beyond the confines of sometimes homogeneous individual trial populations [29, 30].

Additionally, it is likely that journal impact factor has a causal influence in the citation rates of the included trials, and we considered adjusting our results based on impact factor. However, there is also a likely causal relationship between the direction of study results and the publishing journal [31]. Therefore, rather than functioning as a confounder, in this case impact factor is more likely to be a mediator between the direction of trial results and citation rate. As a result, adjusting for impact factor would be expected to result in an underestimation of the relationship between trial results and citation rate. For this reason, we did not adjust our analysis for impact factor of the publishing journals, though our Table 1 lists the identity of the publishing journal for each included trial.

Several limitations of this work should be considered when interpreting these results. First, classification of trials according to comparisons based on the primary outcome measure is not always straightforward. For example, the DIAS trial had multiple intervention groups and multiple primary endpoints; we classified it as positive because several of the analyses assessing these primary endpoints showed results favoring one or more of the intervention groups [20]. Similarly, the ATLANTIS B trial was stopped early, with no between-group difference in the primary efficacy outcome. However, rates of intracranial hemorrhage, symptomatic intracranial hemorrhage, and fatal intracranial hemorrhage were significantly higher in the intervention group; there was a strong trend towards increased overall 90-day mortality in the intervention group (p = 0.09); and in their conclusion the study authors describe their results as “negative” [18]. Based on these factors we classified the trial as negative.

An additional limitation is that we were unable to assess the context in which citations were made. For example, inclusion of a citation does not always mean that the cited study is presented favorably, and to some extent a high citation rate might be a marker of controversy rather than influence. Similarly, we did not assess in which journals studies were cited nor did we evaluate references to studies in the lay press; some journals have more influence on clinicians, researchers, and policy than others and summaries of studies in the lay press may also influence the impact of a study. Finally, while many of the included trials were similar with respect to sample size, publication date, and publication in a high-impact journal, it is likely that other study characteristics, including geographic location and author group, also influenced citation rates. Importantly, however, it is unlikely that the timing of tPA administration alone explains the observed differences in citation rate. For example, the IST-3 trial enrolled more patients in the 0–3-hour window than the NINDS trial [11, 16], and more patients in the 3–4.5-hour window than the ECASS III trial [12], yet the citation curve for IST-3 is consistent with the curves of the other neutral trials rather than the curves observed for these two positive trials. Likewise, the enrollment windows for the ECASS III trial (3 to 4.5 hours) [12] and the ATLANTIS B trial (3 to 5 hours) [18] were similar, yet the annual citation rate for ECASS III is over six times that of ATLANTIS B.