Background
Methods
Study design
Vignettes’ conception
Methodological review
Intervention | Description |
---|---|
Training | Peer reviewers are asked to attend an online training program, with lessons on how to evaluate the methodology, the reporting of data and results, the ethical issues, and how to address them in a review. The course will also inform peer reviewers on what journals want from them from an editor’s perspective. Peer reviewers are then supervised for three articles specifically selected for the course. |
Adding an expert to the peer review process | An expert is asked to peer review the manuscript in addition to the usual peer review process. The expert should be a statistician or a methodologist. |
Use of reporting guidelines checklist | Peer reviewers are asked to complete a checklist based on guidelines (such as CONSORT or STARD, depending on the nature of their manuscript), in addition to their usual review. The checklist is then sent to the authors so they can revise their manuscript. |
Results-free peer review | Peer reviewers are blinded to the results of the study. The peer review process unfolds in 2 steps: 1. Peer reviewers receive the manuscript without the abstract, results or discussion. They write a first review and make a recommendation for publication. This first review is sent to the editor. 2. Peer reviewers then receive the full manuscript to comment on the results, discussion and abstract by answering two simple questions on the completeness of the reporting and on the validity of the interpretation. The review is sent to the editor and combined with the first one. |
Use of incentives | Reviewers are told they will receive an incentive (payment or discounted subscription to the journal) when they are asked to peer review the manuscript. |
Post-publication peer review | Manuscripts are posted online on an open access platform where researchers from all around the world with any background can peer review the study. Chosen researchers are also actively invited by the author and the editor to peer review the online publication. The peer review is entirely transparent: the reviewers’ names and affiliation, their report and the approval status they choose are published along with the article. Peer review reports are posted as soon as they are received and the peer review status of the article is updated with every published report. Once an article has passed peer review (i.e., it has received at least two “Approved” statuses from independent peer reviewers), it will be indexed in PubMed, PubMed Central, Scopus, and Embase. |
Vignettes’ content
Study type | |
---|---|
RCT with randomization of manuscripts | Each manuscript is randomized to be peer reviewed by a peer reviewer in the intervention group or a peer reviewer in the control group. |
RCT with randomization of peer reviewers | Peer reviewers are randomized to the intervention group or the control group. |
Cluster RCT with randomization of journals | Journals are randomized to the intervention group or the control group. All peer reviewers from a journal follow the same peer review process. |
Interrupted time series analysis | Data are collected at multiple time points before and after an intervention to detect whether the intervention had a significantly greater effect than any underlying secular trend. • Period 1: Peer reviewers follow the usual process of peer review. • Intervention: All peer reviewers follow the process of the intervention. • Period 2: Manuscripts are evaluated after the peer review process. |
Pairwise comparison | Each manuscript is sent to be reviewed by both a peer reviewer from the intervention group AND a peer reviewer from the control group. |
Stepped wedge cluster RCT with randomization of journals | The intervention is rolled out sequentially to the journals over a number of time periods. • During the first period, none of the journals follow the intervention. • During the second period, one journal is randomized to invite its peer reviewers to participate in the intervention peer review process. The other journals continue the usual process. • During the third period, an extra journal is randomized to participate in the intervention peer review process, and the other journals continue with the usual process. Therefore, two journals are undergoing the intervention during this period. This process of randomization is repeated at each period until the last journal finally joins the intervention group. |
Participants
Surveys
Sample size
Ranking of the study designs actually implemented
Outcomes
-
“If you read the results of this study, which study would you trust most?”
-
“Which protocol is logistically simpler to set up?”
Statistical analysis
Results
Participants
No. of participants (%) | |
---|---|
Age, years | |
< 40 | 65 (32) |
40–50 | 52 (25) |
51–60 | 45 (22) |
60 | 42 (21) |
Sex | |
Male | 117 (57) |
Female | 87 (43) |
Location | |
Europe | 114 (56) |
North America | 72 (35) |
South America | 0 (0) |
Asia | 4 (2) |
Africa | 2 (1) |
Oceania | 12 (6) |
Occupation* | |
Methodologist | 135 (66) |
Trialist | 99 (49) |
Editor | 102 (50) |
Other | 22 (11) |
Vignette-based surveys
Preferred study designs
Training peer reviewers (24 vignettes, 276 pairs) | Adding an expert to the peer review process (10 vignettes, 90 pairs*) | Use of reporting guidelines checklist (13 vignettes, 156 pairs*) | Results free peer review (24 vignettes, 276 pairs) | Using incentives (13 vignettes, 156 pairs*) | Post-publication peer review (10 vignettes, 90 pairs*) | |
---|---|---|---|---|---|---|
Estimate [95% CI] | Estimate [95% CI] | Estimate [95% CI] | Estimate [95% CI] | Estimate [95% CI] | Estimate [95% CI] | |
Study type | ||||||
RCT with randomization of manuscripts | 0.92 [-0.50 ; 2.41] |
2.03
[0.51 ; 3.49]
|
2.69
[1.39 ; 3.95]
|
2.53
[1.27 ; 3.76]
| 1.00 [-0.21 ; 2.16] |
2.55
[1.13 ; 4.09]
|
RCT with randomization of peer reviewers |
1.45
[0.14 ; 2.78]
| N/A | 1.99 [0.69 ; 3.37] | 2.24 [0.98 ; 3.50] |
2.25
[0.94 ; 3.49]
| N/A |
Cluster RCT with randomization of journals | 0.30 [-1.12 ; 1.63] | 0.76 [-0.90 ; 2.43] | 0.34 [-1.25 ; 1.93] | 0.63 [-0.56 ; 1.88] | -0.16 [-1.49 ; 1.18] | 1.73 [0.13 ; 3.51] |
Interrupted time series analysis | -0.10 [-1.48 ; 1.38] | -0.19 [-1.74; 1.39] | 0.10 [-1.21 ; 1.44] | 0.07 [-1.28 ; 1.40] | 0.73 [-0.51 ; 2.02] | 1.58 [0.13; 3.15] |
Pairwise comparison | 0.83 [-0.49 ; 2.18] | N/A | N/A | 1.61 [0.35 ; 2.86] | N/A | N/A |
Stepped wedge cluster RCT with randomization of journals*** | 0.00 [-] | 0.00 [-] | 0.00 [-] | 0.00 [-] | 0.00 [-] | 0.00 [-] |
Setting | ||||||
Single biomedical journal | -1.02 [-1.82 ; -0.27] | -2.62 [-4.23 ; -0.80] | -2.50 [-3.51 ; 1.50] | -0.20 [-1.03 ; 0.62] | -1.51 [-2.59 ; -0.43] | -3.13 [-4.09 ; -1.53] |
Several biomedical journals from a single publisher | -0.21 [-0.84 ; 0.44] | -1.10 [-2.32 ; 0.06] | -0.12 [-1.06 ; 0.80] |
0.18
[-0.52 ; 0.83]
| -0.01 [-0.82 ; 0.83] | -0.92 [-1.95 ; 0.22] |
Several biomedical journals from different publishers*** |
0.00
[-]
|
0.00
[-]
|
0.00
[-]
| 0.00 [-] |
0.00
[-]
|
0.00
[-]
|
Type of manuscript | ||||||
Actual manuscripts submitted to the journal(s) |
1.04
[0.37 ; 1.79]
| N/A | N/A |
0.57
[-0.15 ; 1.26]
| N/A | N/A |
One fabricated manuscript*** | 0.00 [-] | N/A | N/A | 0.00 [-] | N/A | N/A |
Trust and feasibility
Ranking of the study designs actually implemented
Studies identified | Ranking according to experts* | ||||||
---|---|---|---|---|---|---|---|
No. of studies | Study type | Setting | Type of manuscript | Preference | Trust | Feasibility | |
Training | 6 | - 5 randomized controlled trial of peer reviewers - 1 cross-sectional study | Single journal | - Real manuscripts - 1 RCT with fabricated manuscript | 8/24 (4 RCTs) 21/24 (1 RCT) | 11/24 (4 RCTs) 22/24 (1 RCT) | 9/24 (4 RCTs) 6/24 (1 RCT) |
Use of reporting guidelines checklist | 2 | Randomized controlled trial of manuscripts | Single journal | Real manuscripts | 5/13 | 5/13 | 2/13 |
Adding an expert | 2 | Randomized controlled trial of manuscripts | Single journal | Real manuscripts | 8/10 | 8/10 | 1/10 |
Discussion
Intervention | Comments on the best study design according to experts |
---|---|
Training intervention | The design recommended by the experts was an RCT with randomization of peer reviewers, set in several biomedical journals from different publishers, using actual manuscripts submitted to the journal. The choice of an RCT with randomization of peer reviewers has the advantage of being close to the real-life procedures of the peer review process, with the benefit of using randomization. The issue with the training intervention is its length in time. This raises issues related to poor adherence and missing outcome when peer reviewers randomized never assess a manuscript. The pairwise comparison was the second-ranked design. This design has the advantage of addressing the issue of manuscript variability, thus increasing statistical power, and avoiding the loss to follow-up problem, because no long-term follow up is needed. Such design has never been used to our knowledge. The cluster RCT and stepped wedge cluster RCT were not often chosen by the participants because of the risk of contamination, because peers can review for more than one journal at a time. |
Addition of an expert (methodologist or statistician) | The addition of an expert to the peer review process was preferably assessed with an RCT of manuscripts, set in several journals from different publishers, using the actual manuscripts submitted to the journal. The cluster RCT was the second preferred design for all three of the outcomes. This design has the advantage of including a large variety of reviewers and manuscripts, and it is logistically easy for the editors who do not have to change process for each manuscript. It is nevertheless a difficult design to put in place, as shown by its systematically low score in the feasibility rankings, and a very large number of clusters would be needed to compensate for the high variability between journals (publisher, editorial policies, subject area, quality of reviewers etc.). The interrupted time series set in a single journal was the preferred design in terms of feasibility. This study type is not randomized, which could potentially create bias. |
Use of reporting guidelines checklist | The favored designs to assess the use of reporting guidelines checklist was an RCT of manuscripts, set in several biomedical journals from several or a single publisher, using actual manuscripts. The choice to randomize manuscripts rather than peer reviewers is interesting in terms of logistics, because manuscripts receiving the intervention can be sent directly with the checklist. The preferred settings give a good external validity to the study. |
Results-free peer review | Our analysis suggests the factor influencing the most participant’s decision in their overall preference was the type of study. The favorite type of studies overall were the RCT of peer reviewers and the RCT of manuscripts. The choice of an RCT randomizing manuscripts for the results-free peer review seems appropriate because the intervention is held directly on the manuscript. The issue with the randomization of manuscripts in this situation is the possibility for peer reviewers to perform both with-results and results-free reviews. With the intervention having a potential learning effect, it would artificially increase the quality of reviews in the control group. This intervention has—to our knowledge—never been assessed, which is notable as it could help reduce the important bias towards positive results. |
Use of incentives | The use of incentives raised interesting comments from participants. Particularly, they highlighted that the existence of an incentive may encourage reviewers to accept invitations even if not fully qualified. In a similar way, reviewers in the incentive arm are likely to accept more reviews than the control arm, which raises some issues for the design. |
Post-publication peer review | It was one of the most innovative intervention we included in our study. Although this system has already been in place in several journals, such as F1000, it has, to our knowledge, never been assessed. This intervention is interesting because it changes the entire peer review process, not just the way peer reviews assess the manuscripts. The preferred type of study for this intervention was the randomization of manuscripts. Being randomized, this design would indeed lower the risk of bias of the study; however, it may be hard to implement such an intervention, because journals would have to manage two completely different peer review systems at the same time. |