A brief history of trial reporting and peer review
‘Better have them all removed now.’ That was the advice I received in the early 1990s when my pain free un-erupted wisdom teeth first came to the notice of a surgeon. He was emphatic that I would suffer complications in the future if I did not have all four teeth removed under a general anesthetic. This seemed drastic to me, but I was given the same advice by two health professionals and it was with trepidation that I questioned their advice. At the time, ‘Evidence-Based Medicine’ which proposed the use of scientific evidence to inform clinical decision making was still a novel idea [[
1]] and the Cochrane Collaboration [[
2]], aimed at facilitating up-to-date systematic reviews of randomized controlled trials, had recently been founded.
I decided to search for the evidence. My only source of information was a medical library where I could identify and photo-copy relevant looking articles or get copies via an ‘inter-library loan’. I did not find any useful information, but I decided against the procedure on the basis that the risk of a general anesthetic and a stay in hospital seemed to me to completely outweigh any benefit of having four perfectly healthy pain-free teeth removed.
A short time later, when I was a junior doctor, a subgroup analysis of the diabetic patients who took part in the original ‘4S study’ [[
3]], reported that simvastatin treatment improved morbidity and mortality in patients with diabetes [[
4]]. At the time, my peers and I took for granted that the editors of the journals where the studies were published must have chosen the best people qualified to peer review and the peer reviewers must have done a competent job. The reported findings were compelling enough to have a profound effect on the care received by patients with diabetes.
These experiences not only illustrate the barriers to information I faced as a patient, but the power of individual clinical trials to directly influence treatment decisions for individual patients and the blind faith I and my peers had in a system whereby publication in a peer reviewed journal gave the reported results the status of ‘the evidence’ and, therefore, the ‘Truth’.
While my faith in the publication process was naïve and misplaced, flaws in the way RCTs were conducted and reported were recognized and initiatives were underway to address these concerns. These culminated in the Consolidated Standards of Reporting Trials (CONSORT) statement [[
5]] which aims to specify in detail how RCTs should be reported to improve transparency and help peer reviewers and readers make informed judgments about clinical trials. Since then a number of reporting guidelines for other types of clinical studies have been developed [[
6]].
While reporting guidelines aimed to address how individual trials were reported, there were also concerns about how far only positive or favorable findings were published while those with less exciting, favorable or inclusive findings were not (publishing bias). In 2005, the International Committee of Medical Journal Editors (ICMJE) published a statement announcing that its member journals would adopt compulsory trial registration as journal policy [[
7]]. The aim was to register the existence of all clinical trials so that they became part of the public record.
Recently, in light of ongoing concerns about publication bias and the suppression of unfavorable results, the All Trials campaign [[
8]] was launched which calls for the registering of all clinical trials and availability of all data for treatments in current use.
Meanwhile, running parallel with this, the world of peer review, was undergoing a revolution. Most definitions of peer review include a description of a process of scrutiny by independent experts or peers in the same field [[
9],[
10]]. For peer-review journals this process involves sending submitted manuscripts to two or more people deemed to be knowledgeable enough in the field of the manuscript to judge its suitability for publication in that journal.
Flaws with the common single blind peer review system (where the reviewers know who the authors are, but the authors do not know who the reviewers are) were recognized [[
11]] and there were experiments with double blind peer review to attempt to address this as well as in open peer review where the identity of reviewers and authors is known to all. While closed peer review did not appear to improve the quality of peer review [[
12]], open peer review did appear to be feasible without undermining the quality of peer reviewer reports [[
13]] and was first adopted by the
British Medical Journal (
BMJ) in 1999 [[
14]].
The novel idea of an ‘Open Access’ journal, where all published research is freely available without subscription, began to emerge and although it was met by ferocious opposition from publishers [[
15]], BioMed Central [[
16]], the first completely online open access publisher was founded in 2000, followed, in 2006, by the launch of PLoS One [[
17]].
The number of peer reviewed journals has been increasing at a steady rate of 3.5% a year and almost all are now available online [[
18]]. With online publishing flourishing and with technical advances that allow comments to be made and shared in real time on a global stage, the process of traditional peer review, which can be slow and laborious, has been criticized [[
19]]. New models of peer review have emerged and include (Table
1): re-review opt out [[
20]], post-publication peer review [[
21]], decoupled peer review [[
22]-[
24]], portable peer review [[
25]], and collaborative peer review [[
26],[
27]].
Table 1
Models of peer review
Single blind | Reviewers know who the authors are, but authors do not know who the reviewers are. | The majority of biomedical journals | Varies from journal to journal. The journal editors select peer reviewers according to their own criteria. |
Double blind | Both the reviewers and authors remain anonymous | | As above |
Open peer review | Both reviewers and authors are known to each other | First introduced by the BMJ [[ 14]] | As above |
BMC series medical journals [[ 16]] |
Re-review opt out | Authors are able to ‘opt-out’ of re-review after revisions if reviewers deem the research to be sound. | | As above, but one referee will usually be selected from those nominated by the author. |
Collaborative peer review | Peer review includes a stage where the peer reviewers with or without the editor or authors take part in real time interactive discussion about the manuscript and agree a single set of revisions. | | A member of a ‘Board of Reviewing Editors’ oversees peer review and usually peer reviews themselves. |
| Members of the Editorial Board peer review and use a formal evaluation system |
Portable peer review | Manuscripts which are peer reviewed by one journal, but rejected on grounds of threshold or interest are transferred together with their peer review reports to other journals which have the scope and threshold to match the manuscript. This can occur within a publisher or between a consortium of publishers. | | Criteria for selecting peer reviewers will be that of the original journal |
Decoupled peer review | Manuscripts are submitted to a peer reviewing service which organizes peer review and provides advice on appropriate journals based on the peer review reports. | | Criteria can vary. For example, |
| Rubriq: Peer reviewers must have a terminal degree in the area of interest, be employed full time in an accredited research university at the level of professor, instructor, post doc fellow or faculty research associate, must be a published first author or corresponding author in a peer reviewed academic journal within the last four years, and have prior experience as a journal peer reviewer. There is a standardized scorecard. |
Peerage of science [[ 22]] |
Journals can also select manuscripts based on the peer review reports. |
Peerage of science: Peer reviewers select the manuscripts they wish to review. Peer reviewers need to be scientists to qualify to peer review. Peer review reports are reviewed by fellow reviewers. Only scientists who have published a peer reviewed scientific article in an established international journal as first or corresponding author will be validated as Peers. |
Post publication peer review | Manuscripts undergo initial checks and are published. Peer reviewers are then invited. Authors can revise their manuscripts. Revisions are published. If the manuscript ‘passes’ peer review, the article is indexed in databases such as Pub Med, Scopus etc | | F1000Research: Authors are asked to identify five potential referees who might be from the peer review panel. Author suggested referees should not have collaborated with the authors in the past five years, be from their own institution, or be too senior to be likely to undertake such refereeing (they should ideally have authored at least one article in the field as the lead author). |
The impetus behind these recent initiatives has been to reduce delays for authors and reduce burden for reviewers. Their focus is on the process of peer review in terms of how and when it is done, rather than the substance and quality of peer review itself or expertise of the peer reviewer.