Background
Methods
Sample selection
Journal | Number of RCTs | Survey completed | % survey completed |
---|---|---|---|
General | |||
Annals of Internal Medicine | 18 | 6 | 33% |
BMJ | 31 | 13 | 42% |
JAMA | 76 | 20 | 26% |
Lancet | 85 | 28 | 33% |
New England Journal of Medicine | 134 | 45 | 34% |
PLoS Medicine | 16 | 9 | 56% |
Specialty | |||
Anesthesia and Analgesia | 34 | 9 | 26% |
Annals of Rheumatic Diseases | 42 | 12 | 29% |
British Journal of Anaesthesia | 39 | 19 | 49% |
Circulation | 22 | 9 | 41% |
Diabetes Care | 84 | 12 | 14% |
Journal of the American College of Cardiology | 27 | 3 | 11% |
Journal of Clinical Oncology | 70 | 19 | 27% |
Journal of Infectious Diseases | 23 | 6 | 26% |
Journal of Pediatrics | 31 | 7 | 23% |
Lancet Oncology | 75 | 20 | 27% |
Pediatrics | 59 | 14 | 24% |
Stroke | 27 | 7 | 26% |
Total | 893 | 258 | 29% |
Eligibility criteria
Online survey
Data extraction and analysis
Results
Characteristics of primary reports of randomised trials
Characteristic | n = 258 |
---|---|
Journal type | |
General medical | 118 (46%) |
Specialty | 140 (54%) |
Most common specialties | |
Oncology | 44 (17%) |
Cardiology | 42 (16%) |
Paediatrics | 26 (10%) |
Anaesthesia | 22 (8%) |
Public health | 18 (7%) |
Most common countries where trial was conducted | |
USA | 40 (22%) |
UK | 27 (15%) |
Netherlands | 16 (9%) |
Australia | 4 (2%) |
Not reported | 38 (20%) |
Trial designa | |
Parallel | 225 (86.5%) |
Cluster | 16 (6%) |
Factorial | 11 (4%) |
Crossover | 8 (3%) |
Split-body | 1 (0.5%) |
Intervention | |
Drug | 127 (49%) |
Surgery/procedure | 59 (23%) |
Behavioural/education | 40 (16%) |
Biological/vaccine | 12 (5%) |
Device | 9 (3%) |
Dietary supplement | 6 (2%) |
Other | 5 (2%) |
Study centres | |
Single | 46 (18%) |
Multiple | 191 (74%) |
Unclear | 21 (8%) |
Number of intervention groups | |
2 | 202 (78%) |
3 | 35 (13.5%) |
4 | 20 (8%) |
> 4 | 1 (0.5) |
Median sample size (IQR)b | 325 (138 to 1010) |
10 to 90 percentile | (60 to 2485) |
Primary outcome clearly defined | |
Defined | 238 (92%) |
Not defined | 20 (8%) |
Statistical significance of primary outcome (where defined)c | |
Significant | 116 (49%) |
Non-significant | 122 (51%) |
Trial registration | |
Reported | 235 (91%) |
Not reported | 23 (9%) |
Trial protocol accessible | |
Reported | 141 (55%) |
Not reported | 117 (45%) |
Source of funding | |
Solely industry | 52 (20%) |
Part industry | 27 (10%) |
Non-industry | 159 (62%) |
No funding | 7 (3%) |
Not reported | 13 (5%) |
Response to online survey by authors of primary reports of randomised trials
Survey responses n = 258 | |
---|---|
1. How satisfied have you been with the overall handling of your manuscript by the journal? (1 = very unsatisfactory to 10 = very satisfactory) | |
Mean (SD) | 8.6 (1.5) |
Median | 9.0 |
Min – max | 1–10 |
2. How would you rate the overall quality of the peer review of your manuscript? (1 = very low to 10 = very high) | |
Mean (SD) | 8.5 (1.5) |
Median | 9.0 |
Min – max | 1–10 |
3. In getting your manuscript published, did the editor or peer reviewers ask you to change any aspects of your study so that it was different from what was planned in your trial protocol? | |
Yes | 36 (14%) |
No | 222 (86%) |
4. Did the editor or peer reviewers ask you to change or clarify the trial’s primary outcomes measure(s)? | |
Yes | 8 (3%) |
No | 250 (97%) |
5. Did the editors or peer reviewers ask you to change or clarify the statistical analysis of your primary outcome measure(s)? | |
Yes | 69 (27%) |
No | 189 (73%) |
6. Were you asked to include any additional analyses that had not been included in the original manuscript? | |
Yes | 94 (36%) |
No | 164 (64%) |
7. Were you asked to modify your overall conclusions? | |
Yes | 60 (23%) |
No | 198 (77%) |
8. Did you register your trial in a trial registry? | |
Yes | 249 (96.5%) |
No | 9 (3.5%) |
9. Did you publish the protocol of this trial in a journal? | |
Yes | 136 (53%) |
No | 122 (47%) |
Sub-questions | 4a. Change or clarify the trial’s primary outcome measures? (yes, n = 8) | 5a. Change or clarify the statistical analysis? (yes, n = 69) | 6a. Include any additional analyses? (yes, n = 94) | 7a. Modify the overall conclusion? (yes, n = 60) |
---|---|---|---|---|
Who requested the change? n (%)a | ||||
Editor | 3 (38%) | 22 (32%) | 29 (31%) | 35 (58%) |
Reviewer | 8 (100%) | 39 (57%) | 77 (82%) | 31 (52%) |
Statistician | 0 (0%) | 32 (46%) | 21 (22%) | 6 (10%) |
Do not know | 0 (0%) | 2 (3%) | 0 (0%) | 2 (3%) |
What changes or clarifications were requested? n (%)a | ||||
Change of the primary outcome measure | 2 (25%) | n/a | n/a | n/a |
Clarification of the primary outcome measure | 2 (25%) | n/a | n/a | n/a |
More cautious conclusion | n/a | n/a | n/a | 53 (88%) |
Stronger conclusion | n/a | n/a | n/a | 2 (3%) |
Other | 4 (50%) | n/a | n/a | 5 (8%) |
Did you fulfil the request? n (%) | ||||
Yes | 5 (62.5%) | 66 (96%) | 83 (88%) | 54 (90%) |
No | 3 (37.5%) | 3 (4%) | 11 (12%) | 6 (10%) |
What was your main motivation to fulfil the request? n (%)a | ||||
Improvement of the reporting of the trial | 4 (50%) | n/a | n/a | 32 (53%) |
Improvement of the statistical methods/analysis | n/a | 38 (55%) | 30 (32%) | n/a |
Avoiding rejection of the paper | 2 (25%) | 30 (44%) | 49 (52%) | 29 (48%) |
Other | 3 (38%) | 16 (23%) | 31 (33%) | 4 (7%) |
How did you judge the request? (1 = not problematic to 10 = very problematic) | ||||
Mean (SD) | 6.8 (3.1) | 3.6 (2.4) | 3.7 (2.6) | 3.5 (2.2) |
Median | 7.5 | 3.0 | 3.0 | 3.0 |
Range | 2–10 | 1–10 | 1–10 | 1–10 |
Does the published article indicate that the analyses have not been pre-specified in the protocol? n (%) | ||||
Yes | n/a | n/a | 22 (23%) | n/a |
No | 72 (77%) |
4. Did the editor or peer reviewers ask you to change or clarify the trial`s primary outcomes measure(s)? | Yes | No | Fisher’s exact test | P value |
(n = 8) | (n = 250) | |||
Type of journal | ||||
General | 3 (38%) | 115 (46%) | 0.73 | 0.46 |
Specialty | 5 (62%) | 135 (54%) | ||
Type of intervention | ||||
Drug | 2 (25%) | 125 (50%) | 0.28 | 0.15 |
Non-drug | 6 (75%) | 125 (50%) | ||
Statistical significance of primary outcome (where defined)a | ||||
Significant | 6 (86%) | 110 (48%) | 0.06 | 0.05 |
Non-significant | 1 (14%) | 121 (52%) | ||
Funding sourceb | ||||
Sole/part industry | 1 (14%) | 78 (34%) | 0.43 | 0.26 |
Non-industry | 6 (86%) | 153 (66%) | ||
5. Did the editors or peer reviewers ask you to change or clarify the statistical analysis of your primary outcome measure(s)? | Yes (n = 69) | No (n = 189) | Chi2 test | P value |
Type of journal | ||||
General | 31 (45%) | 87 (46%) | 0.02 | 0.88 |
Specialty | 38 (55%) | 102 (54%) | ||
Type of intervention | ||||
Drug | 28 (41%) | 99 (52%) | 2.82 | 0.09 |
Non-drug | 41 (59%) | 90 (48%) | ||
Statistical significance of primary outcome (where defined)a | ||||
Significant | 27 (40%) | 89 (52%) | 2.66 | 0.10 |
Non-significant | 40 (60%) | 82 (48%) | ||
Funding sourceb | ||||
Sole/part industry | 17 (26%) | 62 (36%) | 1.20 | 0.16 |
Non-industry | 48 (74%) | 111 (64%) |
Textual analysis of author comments to the online survey questions
Changes of methods to impute missing data |
“….they asked us to include the multiple imputation data for the primary outcome, which was a sensitivity analysis, in the abstract and up front in the results. We ensured that all of the data were presented and it made almost no difference to the results so we were OK with it.” (case 478) “Requested that we change report of primary outcome to use multiple imputation of missing data.” (case 454) |
Change of statistical model/methods |
“We were asked to use a different type of regression model and different independent variables than we had stated in our trial protocol.” (case 556) |
“They recommended a different statistical approach. In the end we felt it was a better approach, and it did not change the overall intent or purpose of the study, but it was different than what we originally had planned in our protocol.” (case 450) |
Analysis populations and adjustment issues |
“To include all patients after randomisation in ITT and to delete the per-protocol analysis from the main text. Also, to present the baseline adjusted analysis as primary analysis and the analysis adjusted for unbalanced baseline characteristic as additional analysis.” (case 509) |
Clarifications or presentation issues |
“The editors requested clarification on a planned non-inferiority analysis.” (case 651) |
“Clarify statistical methods used.” (case 563) |
“Change mean and SD into median and 95% CI in case of skewed distributions.” (case 462) |