Background
Methods
Study design
Information sources and search strategy
Eligibility criteria
Structured quality tool: scale or checklist including more than one item aimed at guiding the user to assess the overall quality of a peer review report. | |
Unstructured quality tool: scale or checklist including only one item inquiring the overall quality of a peer review report. | |
Items: elements of a scale or checklist representing a component of peer review report quality. Items in a scale could or could not have an attached numerical score. If there is no attached score, these items provide the evaluator with a guidance to assess the overall quality of a peer review report. | |
Overall quality score in a scale is measured as: • Sum of scores: score obtained by summing all scores for each item present in a scale. • Mean of scores: score obtained by dividing the sum of scores for each item with the total number of items included in the tool. • Single score: score obtained in those scales based on a single item. • Summary score: score obtained in those scales with more than one item deriving from a question inquiring the overall quality of peer review report. |
Study selection
Data extraction
General characteristics of tools
Scoring system instructions | ||
---|---|---|
Defined | Partially defined | Not defined |
5 (Exceptional) = The rare outstanding critique that is comprehensive, objective, and insightful. Evaluates purpose of the study, study design, scientific validity, and conclusions by numbering questions and constructive suggestions to be addressed by the author. Includes comments to the editor about whether this is something new and important and useful to our readers. 4 (Very good) = Excellent review indicating that the paper was carefully evaluated. Helpful comments to the author and editor with well-documented reasons for decision. 3 (Good) = Useful type of very satisfactory review. Analysis not as well organized, documented, or as complete as above but is reasonable, with adequate comments for the authors. 2 (Below average) = Very brief, superficial evaluation. Reasons for the decision not explained and comments to authors not helpful. 1 (Unacceptable) = Such a poor review that consideration should be given to not sending further papers to this reviewer. Reasons could include evidence of bias, unfair, faulty reasoning, or evaluation (totally disagrees with the opinion of other reviewers and editor) and comments to author either absent, inappropriate, or inadequate to explain how the paper was rated. (Landkroon 2006) [42] | 1 (Poor) = Does not follow reviewer guideline structure or preferred formatting in providing comments; unfavourable timeliness. 2 (Acceptable) = Comments are somewhat helpful; review meets timeline. 3 (Reliable) = Thorough and helpful comments; timely submission. 4 (Excellent) = Very strong and detailed comments; review was submitted early or on time; comments enhance the manuscript’s merit and relevance in the field. (Rajesh 2013) [32] | 1 = poor; 2 = fair; 3 = good; 4 = excellent (Friedam1995) [22] |
Quality components of the peer review report considered in the tools
Results
Study selection and general characteristics of reports
General characteristics of the tools
Characteristics of tools | N (%) |
---|---|
Type of tool: | |
Scale | 23 (96%) |
Checklist | 1 (4%) |
Number of items: | |
1 | 6 (25%) |
> 1 | 18 (75%) |
Weight of items a: | |
Same weight | 10 (42%) |
Different weight | 2 (8%) |
User defined weight | 1 (4%) |
Not applicable | 11 (46%)a |
Score System Instruction: | |
Defined | 5 (21%) |
Partially defined | 3 (12%) |
Not defined | 16 (67%) |
Tool development: | |
Reported | 1 (4%) |
Not reported | 23 (96%) |
Overall quality assessment b | |
Single score | 6 (22%) |
Summary score | 11 (41%) |
Mean score | 6 (22%) |
Sum score | 3 (11%) |
Not reported | 1 (4%) |
Journal or Company Name a | First Author, Year | Format | Quality defined b | Overall quality assessment | Items (n) | Items weights c | Scoring range d | Scoring system instruction e | Scale/ Checklist Development f | Validity g | Reliability h | Internal consistency | RCTs i |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Advances in Nursing Science; Issues in Mental Health Nursing; The Journal of Holistic Nursing | Shattell 2010 [33] | Scale | N | Summary Score | 6 | S | 1–10 | N | NR | NR | NR | NR | 0 |
American Journal of Roentgenology | Friedman 1995 [22] | Scale | N | Single Score | 1 | NA | 1–4 | N | NR | NR | NR | NR | 0 |
American Journal of Roentgenology | Kliewer 2005 [49] | Scale | N | Summary Score | 4 | NA | 1–4 | N | NR | NR | NR | NR | 0 |
American Journal of Roentgenology | Rajesh 2013 [32] | Scale | N | Single Score | 1 | NA | 1–4 | P | NR | NR | NR | NR | 0 |
American Journal of Roentgenology | Berquist 2017 [50] | Scale | N | Summary Score | 4 | NA | 0–4 | Y | NR | NR | NR | NR | 0 |
Annals of Emergency Medicine | Callaham 1998 [25] | Scale | N | Single Score | 1 | NA | 1–5 | N | NR | NR | Inter-Rater (ICC = 0.44, 0.24, 0.12) l | NR | 2 m |
Annals of Emergency Medicine | Scale | N | Summary Score | 6 | NA | 1–5 | N | NR | NR | Inter-Rater (ICC = 0.44, 0.24, 0.12) l | NR | 1 | |
Annals of Emergency Medicine; Annals of Internal Medicine; JAMA; Obstetrics & Gynecology and Ophthalmology | Justice 1998 [35] | Scale | N | Summary Score | 4 | S | 1–5 | N | NR | NR | NR | NR | 0 |
British Journal of General Practice | Moore 2014 [29] | Scale | N | Single Score | 1 | NA | A-E | Y | NR | NR | NR | 0 | |
British Medical Journal | Scale | N | Summary Score | 7 | S | 1–5 | N | Y | Face (N = 20) | Test-Retest (Kw = 1.00) | Internal Consistency (Cronbach’s alpha = 0.84) | 5 | |
Mean | Content (N = 20) Construct | Inter-Rater (Kw = 0.83) | |||||||||||
British Medical Journal | Van Rooyen 1999 (RQI 4) [27] | Scale | N | Mean n | 8 | S | 1–5 | N | NR | NR | Inter-Rater (Kw = 0.38–0.67) o | 2 | |
Chinese Journal of Tuberculosis and Respiratory Diseases | Yang 2009 [52] | Checklist | N | NA | 5 | NA | NA | N | NR | NR | NR | 0 | |
Journal of Clinical Investigation | Stossel 1985 [30] | Scale | N | Single Score | 1 | NA | Good- Fair- Poor | Y | NR | NR | NR | 0 | |
Journal of General Internal Medicine | Scale | N | Summary Score | 9 | S | 1–5 | N | NR | Construct | NR | 1 | ||
Journal of Vascular Interventional Radiology | Feurer 1994 [41] | Scale | N | Sum | 7 | D | 0–14 | N | NR | Content (N = 2) Preliminary Criterion (N = 2) (Kendall = 0.94) | Inter-Rater (ICC = 0.84) | 0 | |
NA | Review quality collector (RQC) 2012 [53] | Scale | N | Mean | 4 | User-defined weights | 0–100 | N | NR | NR | NR | 0 | |
Nursing Research | Henly 2009 [24] | Scale | N | Mean (CAS, GAS scale) | 15 | S | 1–5 | P | NR | NR | Inter-Rater (ICC = 0.79) p | 0 | |
Summary Score (OAS scale) | 1–5 | ||||||||||||
Summary Score (GRQ scale) | 0–100 | ||||||||||||
Nursing Research | Henly 2010 [36] | Scale | N | Mean (CAS, GAR, SARNR scale) | 26 | S | 1–5 | P | NR | NR | Inter-Rater (ICC = 0.75)p | 0 | |
Summary Score (GRQ scale) | 0–100 | ||||||||||||
Obstetrics & Gynecology, Dutch Journal of Medicine | Landkroon 2006 [42] | Scale | N | Summary Score | 5 | NA | 1–5 | Y | NR | NR | Test-Retest (ICC =0.66–0.88) Inter-Rater (ICC = 0.62) | 0 | |
Pakistan Journal of Medical Sciences | Jawaid 2006 [34] | Scale | N | NR q | 5 | S | 1–5 | N | NR | NR | NR | 0 | |
Peerage of science | Peerage Essay Quality (PEQ) 2011 [37] | Scale | N | Mean | 3 | S | 1–5 | N | NR | NR | NR | 0 | |
Publons Academy | Review Rating and Feedback Form 2016 [38] | Scale | N | Sum | 4 | S | 0–3 (Full score: 0–12) | N | NR | NR | NR | 0 | |
The Journal of Bone and Joint Surgery | Thompson 2016 [31] | Scale | N | Single Score | 1 | NA | 80–100 | Y | NR | NR | Inter-Rater (ICC = -4.5 to 0.99) r | 0 | |
The National Medical Journal of India | Das Sinha 1999 [54] | Scale | N | Sum | 5 | D | 0–100 | N | NR | NR | NR | 0 |
Quality components of the peer review reports considered in the tools with more than one item
N | Domains | Subdomains | Explanations and Examples |
---|---|---|---|
1 | Relevance of the study | Explanation: Items inquiring if the reviewer has discussed in the peer review report the importance of the research question and usefulness of the study. Example: ‘Did the reviewer give appropriate attention to the importance of the question?’ [28] | |
2 | Originality of the study | Explanation: Items inquiring if the reviewer has commented in the peer review report on the originality of the manuscript. | |
3 | Interpretation of the study results | Explanation: Items inquiring if the reviewer has commented in the peer review report on how authors interpreted and discussed the results of the study. Example: ‘The reviewer commented accurately and productively on the quality of the author’s interpretation of the data, including acknowledgment of the data’s limitations.’ [26] | |
4 | Strengths and weaknesses of the study | General | Explanation: Items inquiring if the reviewer has identified and commented in the peer review report on the general strong and weak points of the study. Example: ‘How well it identified the study’s strengths and weaknesses?’ [35] |
Methods | Explanation: Items inquiring if the reviewer has identified and commented in the peer review report on the strong and weak points specifically related to study’s methods Example: ‘Did the reviewer clearly identify strengths and weaknesses in the study’s methods?’ [28] | ||
Statistical methods | Explanation: Items inquiring if the reviewer has identified and commented in the peer review report on the strong and weak points specifically related to study’s statistical methods Example: ‘Confidence intervals/p-values/overall fit’ [36] | ||
5 | Presentation and organization of the manuscript | Explanation: Items inquiring if the reviewer has made comments in the peer review report on the data presentation such as tables and figures and on the organization of the manuscript such as writing communication. Example: ‘Are there any constructive suggestions on improvement of a. writing; b. data presentation and c. interpretation’ [54] | |
6 | Structure of reviewer’s comments | Explanation: Items inquiring if the reviewer has made in the peer review report organized and structured comments. Example: ‘Concise well-organized comments to the editor’ [50] | |
7 | Characteristics of reviewer’s comments | Clarity | Explanation: Items inquiring if the reviewer has provided in the peer review report clear and easily to read comments. Example: ‘How clear was this review? The review was easily read and interpreted by the editor and authors.’ [38] |
Constructiveness | Explanation: Items inquiring if the reviewer has provided in the peer review report helpful, relevant and realistic comments. | ||
Detail/Thoroughness | Explanation: Items inquiring if the reviewer has provided in the peer review report detailed and thorough comments supplying appropriate evidence. Example: ‘Detail of commentary’ [33] | ||
Fairness | Explanation: Items inquiring if the reviewer has provided in the peer review report balanced and objective comments. | ||
Knowledgeability | Explanation: Items inquiring if the reviewer has showed in the peer review report to know and understand correctly the content of the manuscript. Example: ‘Knowledge of the manuscript’s content area.’ [28] | ||
Tone | Explanation: Items inquiring if the reviewer has used a courteous tone in the peer review report. Example: ‘Overall tone of the reviewers was also assessed as harsh or courteous.’ [34] | ||
8 | Timeliness of the review report | Explanation: Items inquiring if the reviewer has completed the peer review report on time. Example: ‘Punctuality of the review’ [49] | |
9 | Usefulness of the review report | Decision making | Explanation: Items inquiring if the reviewer has provided a peer review report useful to make a decision about the acceptance, revision or rejection of a manuscript Example: ‘The reviewer provided the editor with the proper context and perspective to make a decision about acceptance or revision of the manuscript.’ [26] |
Manuscript improvement | Explanation: Items inquiring if the reviewer has provided useful suggestions in the peer review report to improve the manuscript. Example: ‘This aspect is solely interested in how well the review aids the authors for improving their work and/or writing. Whether the review makes a good judgment regarding acceptance of the submission plays no role here whatsoever.’ [53] |