Comparative analysis of Cochrane and non-Cochrane reviews over three decades

The amount of scientific literature is growing exponentially [1]. With the introduction of evidence-based medicine in the late part of the twentieth century, synthesizing evidence became an area of focus [2]. The principles of evidence-based medicine dictate that the best clinical guidance should be based on the least biased and most up-to-date evidence available. This comes in the form of systematic reviews and meta-analyses, which are generally viewed as the study design that is least likely to be biased, assuming they are well-conducted and based on methodologically rigorous studies [3], and thereby providing the best evidence for informing decision-makers [4, 5]. In 1993, the Cochrane Collaboration (now “Cochrane”) was founded on the same ideals as evidence-based medicine was built upon, with an aim of providing systematic reviews and meta-analyses of high quality [6]. While there are examples of synthesizing evidence systematically several hundred years ago [7, 8], the development of the methodology and frequency of publication of systematic reviews has accelerated during the last 30 years where Cochrane has existed. Cochrane has played a considerable role in developing a thorough systematic review methodology [9, 10] and creating software to ease the production of systematic reviews, e.g., RevMan Web (Review Manager Web, version 4.12.0, The Cochrane Collaboration, 2022). Other non-Cochrane initiatives to streamline and improve systematic review methodology have also been developed, e.g., the Meta-analyses of Observational Studies in Epidemiology (MOOSE) guideline [11], the Preferred Reporting Items for Systematic reviews and Meta-Analyses (PRISMA) guideline [12], and the Joanna Briggs Institute guidelines [13]. However, with the development and distribution of systematic review methodology, it is unclear how the distribution of systematic review production through the years has changed.

We aimed to characterize changes in publication of non-Cochrane systematic reviews (henceforth referred to simply as “systematic reviews”) and Cochrane reviews throughout Cochrane’s existence. Our primary objective was to determine the yearly publication rates of systematic reviews and Cochrane reviews through a 30-year period from 1993 to 2022. Our secondary outcome was to further characterize changes for both types of reviews.

We extracted 264,127 records from PubMed. Two records were excluded as they had no registered date. Subsequently, 15,330 records were excluded for being published either before 1993 (n = 33) or in 2023 (n = 15,297), and 1344 were excluded for being protocols of systematic reviews. After a manual assessment of DOI duplicates, further 50 records were excluded. After DOI-matching to the CDSR, a total of 246,640 records were analyzed, of which 231,602 (93.9%) were systematic reviews and 15,038 (6.1%) were Cochrane reviews (Table 1).

Characteristics expressed as number (%) or mean ± standard deviation of included non-Cochrane systematic reviews and Cochrane reviews for periods of 10 years and the total 30-year period. Regarding countries, the top five most common countries of first author affiliations for the given periods were listed with the total number of systematic reviews published. Countries are abbreviated following the United Nations Conference on the Standardization of Geographical Names abbreviations [20]. Data were collected from PubMed and data on Cochrane reviews were linked via the records Digital Object Identifier. For each outcome, n represents how many records had data available for the outcome.

The number of published systematic reviews increased continuously throughout the 30-year period (Fig. 1) with a median yearly increase rate of 26% (Table 2). Meanwhile, Cochrane reviews reached their peak in productivity in 2013 and growth rates fluctuated over the whole 30-year period with a median yearly increase of 3%. Since 2015, Cochrane reviews have been published less frequently (Figs. 1, 2 and Table 2). For the period 1993–2002, 1377 Cochrane reviews were published, constituting 35% of all systematic reviews indexed in PubMed (Fig. 1). For the period 2013–2022, 7240 Cochrane reviews were published corresponding to 3.5% of all systematic reviews in PubMed for the period, and in 2022 alone, 344 Cochrane reviews were published, corresponding to a share of 0.8%.

Systematic reviews had a mean of 3.6 authors per review in 1993–2002, which increased to 5.9 in 2013–2022. Meanwhile, Cochrane reviews had 3.2 authors in 1993–2002, which increased to 5.3 in 2013–2022. Cochrane reviews continuously had fewer authors than systematic reviews for all 10-year periods. From 2013–2022, Chinese first authors published most systematic reviews (27,230, 15%). Meanwhile, for Cochrane reviews in the same period, Chinese first authors published 5% of these. From 1993–2002 and 2003–2012, Chinese first authors published 4 (0.2%) and 1079 (4%) systematic reviews, respectively. Generally, the native English-speaking countries the USA, the UK, Canada, and Australia combined produced a large proportion of the published systematic reviews for both systematic reviews (86,835, 42%) and Cochrane reviews (8999, 62%) (Fig. 3). The UK had produced most Cochrane reviews throughout the 30-year period (5213, 36%) and for all 10-year intervals, but the share of Cochrane reviews produced by UK first authors was decreasing (Fig. 3). In the first 10 years of the period, high-impact journals such as the BMJ, JAMA, and Lancet were among the journals publishing most systematic reviews (Table 3, Additional File 1). From 2013 to 2022, gold open access journals constituted the top eight journals publishing systematic reviews, while the remaining two journals in the top ten used a hybrid open access model. In total, 8851 unique journals published systematic reviews for the 30-year period.

In this observational study, we found that the number of published systematic reviews has increased considerably, whereas Cochrane’s relative share of systematic reviews has decreased throughout its existence. While systematic reviews increased, the publication of Cochrane reviews has decreased steadily since 2015. During the last decade, Chinese first authors produced a large portion of systematic reviews. Gold open access journals were the largest publishers of systematic reviews from 2013 to 2022, whereas high-impact journals were the largest publishers of systematic reviews from 1993 to 2002. The number of authors seemed to increase over time.

We found that publication of systematic reviews is growing rapidly. Compared with other general publications in life sciences, where authors found a growth rate of 5.1% [1], systematic reviews are growing faster. Our findings show that while the motivation for publishing systematic reviews was high, Cochrane reviews were being published less frequently. Cochrane reviews are known to take a long time to complete [21‐23] compared with systematic reviews [24‐26]. This may be partly due to the process of a Cochrane review having more mandatory steps involved, e.g., minimum requirements for databases searched, searching trial registries, and requirements regarding which analyses to conduct. This may also explain why they are known overall to be of higher quality [27‐32] and are valuable for developing healthcare policies [33]. The thought of high demands for resources and long publication times may intimidate potential authors. However, despite this, Cochrane’s author satisfaction was generally high [34]. Recently, Cochrane has begun rethinking their publication model [35] with several key projects aimed at improving the author experience and optimizing the publication process [36].

We found that balances had shifted relating to production and publication of systematic reviews over the past 10 years. Regarding production, especially systematic reviews written by Chinese first authors have expanded, increasing their publication of systematic reviews by a factor of 25 from 2003–2012 to 2013–2022. The same productivity increase among Chinese authors was not seen in Cochrane reviews. Despite the rapid increase in productivity, one Chinese study found similar methodological and reporting quality between Chinese systematic reviews and those from the USA [37]. Regarding publication, gold open access journals had grown to be the largest publishers of systematic reviews from 2013 to 2022. Open access has been gaining popularity and may be a step in the right direction for science [38]. However, gold open access journals operate with article processing charges that authors must pay to get their papers published. These charges can be steep and have been increasing rapidly over time [39]. There is a wish within the scientific community to transition to a diamond open-access model, where articles are published open access but without authors or readers having to pay exorbitant fees [40]. While this is undoubtedly desirable, there are still costs associated with academic publishing and determining how the diamond open-access model should be financed is not an easy matter. With the increasing amount of research production, there has been discussion of “research waste” [41], i.e., unnecessary and non-contributing research. It was argued already in 1994, at the beginning of our inclusion period, that the scientific community should aim towards producing fewer reviews and instead focus on increasing the quality of publications [42]. Since then, we have had 30 years of exponential increase in evidence synthesis. Relating to this concept, several suggestions towards reducing redundancy of reviews have been put forward [43]. Addressing research waste seems to be a topic gaining focus [43, 44]. We also found an increasing number of authors through time. This aligns with the findings of other studies [45, 46]. This may relate to an increasing amount of data in modern research thereby increasing complexity [47]. One study, however, found that adjusting for factors relating to increased complexity did not explain the increase in authorship when it came to trials and non-randomized studies [48]. Furthermore, systematic reviews differ from clinical trials in that data are available remotely through searches, and therefore, the resource demands may be lower. Interestingly, Cochrane reviews had fewer authors than systematic reviews. This may seem counterintuitive, as Cochrane reviews generally have higher demands and should therefore demand more resources. It is not clear what is causing the increasing number of authors in medicine.

Our study had several strengths. Firstly, we analyzed a very large dataset with the goal of having generalizable conclusions. We only searched PubMed, which is a biomedical database, where records are manually screened for indexation as systematic reviews. We thereby avoided a great deal of false positives that traditional searching would produce. This was demonstrated by validating the search and finding a practical precision of 98%. Generally, searches in large bibliographic databases have low precision [49]. One of the authors, a professional data analyst, conducted data cleaning to adjust for wrongly registered data. Furthermore, we collected data on publication dates from the CDSR, which is the primary source of Cochrane reviews. Therefore, we are confident that Cochrane reviews were correctly and sufficiently identified. However, our study also had some limitations. With large datasets, it is likely there were some miscategorized records. Based on our precision estimate and after data cleaning, we estimate this to be about 2% overall, but some areas may have had more than others. Despite the National Library of Medicine’s personal screening of records, we still found some protocols indexed as systematic reviews, indicating that the systematic review filter, while good, is not 100% perfect. Still, it is a more reliable method of retrieving systematic reviews than through searches and the most feasible method owing to the size of our sample. While searching PubMed ensured high external validity, there are undoubtedly more systematic reviews produced in the period indexed in other databases. However, correcting for this bias in our results would only exaggerate the conclusion that systematic reviews are increasing rapidly and constitute an increasingly larger portion of all systematic reviews compared with Cochrane reviews. On the other hand, a small portion of the increase in systematic reviews may be attributed to the post-indexation of older records and journals in PubMed. We do not believe this potential bias would impact the overall conclusions. Generally, we are confident that the assumed miscategorized data would not change the overall conclusions. This study cannot address the quality of the included systematic reviews or if there is an overlap of the research question of these systematic reviews, thus, whether unnecessary systematic reviews are being published.

It is unclear why Cochrane reviews were being published less frequently in the past 10 years. As Cochrane reviews are expected to be updated and despite them not always being updated frequently [50], we expected the number to increase given the combination of new Cochrane reviews being published and older ones being updated. It may be favorable to have more Cochrane reviews published, as they are generally of high quality owing to their thorough peer review process and rigorous methodology [27‐32]. Some of the decrease in publication of Cochrane reviews may be due to them generally taking a long time to complete [21‐23] and including many procedural and methodological demands [9, 10]. Future studies exploring why Cochrane reviews were being published less frequently are warranted. The continually increasing publication rates of systematic reviews may risk an information overload, where stakeholders cannot reasonably stay orientated and updated with new literature. It is, however, unclear to which degree the increase in publication of systematic reviews results from overlapping or redundant reviews being published, or if the new studies, in general, are of value to the research community. Further research into field overlap and the methodological quality of newer systematic reviews may clarify this.

In conclusion, the publication of systematic reviews has increased rapidly in the past 30 years, while fewer Cochrane reviews have been published since 2015. Especially Chinese first authors conducted many systematic reviews through the last 10 years. For the same period, gold open access journals were the largest publishers of systematic reviews. It may be favorable to have more Cochrane reviews published, as these are generally of higher quality than systematic reviews. Furthermore, further research regarding metrics related to research waste may clarify questions regarding the utility and value of systematic reviews throughout time.