Are CONSORT checklists submitted by authors adequately reflecting what information is actually reported in published papers?

We looked at 12 randomly selected randomized trials published in either Trials, BMJ Open or PLoS One between 1 January 2016 and 30 June 2017 (see Additional file 1: Point 1, search strategy and study selection). We chose those journals because they require authors to submit a CONSORT checklist with their manuscript and make this checklist accessible for their readers as supplementary material. For each paper selected, we retrieved the initial CONSORT checklist and manuscript submitted by the authors. First, we independently determined whether the CONSORT checklist originally submitted by the authors was the appropriate extension for the study design. Then, for papers using the appropriate checklist, we compared it with what was actually reported in the published paper to identify any inconsistencies (see Additional file 1: Point 2, analysis of inconsistencies). We focused on six core CONSORT items of the ‘Methods’ and ‘Results’ sections: (6a) outcomes; (8a) randomization or sequence generation; (9) allocation concealment mechanism; (11a) blinding; (13a) flow of participants and (13b) losses and exclusions. We selected those items because they are essential for systematic reviewers to evaluate the risk of bias. For each item, the CONSORT explanation and elaboration document [9] was used to determine what information should be reported.

We graded the consistency between what authors said and what they reported for each item as follows:

From the 12 randomly selected randomized trials, the standard CONSORT checklist was appropriate for six papers (four of which were standard parallel trials covered by CONSORT and two of which were crossover trials, for which there is not an extension yet). The other six required CONSORT extensions (for cluster trials, three; for pragmatic trials, two; and for non-pharmacological interventions, one) but authors did not use them in any case, despite the extensions being available at the time of submission. The aforementioned extensions were published between 2008 and 2012 [7, 8, 10], yet the six papers requiring their uptake were all submitted later than May 2015.

For the six papers for which an appropriate CONSORT checklist was submitted, only one paper had complete consistency between the checklist and the published paper. The most concerning problems centred on items 8 and 9 (see Fig. 1). For example, a possible inconsistency identified regarding CONSORT item 9 (Allocation concealment mechanism – mechanism used to implement the random allocation sequence, describing any steps taken to conceal the sequence until interventions were assigned) is the following: authors claim through the checklist that they have reported the item; however, we found that the paper stated “The sequence of the test conditions was randomized for each participant by LB and KDK. A card was made for each possible sequence and a card was picked blindly for each participant.” This statement does not make it clear how the authors implemented the random allocation sequence nor how they kept the assignment concealed. Picking a card does not guarantee that allocation used in the analysis has preceded treatment, nor does it allow readers to reproduce the mechanism used to implement the random allocation sequence.

A full summary of the evaluations for all six items across all six papers is shown in Additional file 2. The level of reporting inconsistencies found for every item among the six papers considered in the analysis is provided in Fig. 1. Illustrative examples of inconsistencies found for each item are shown in Additional file 3.

Among the numerous potential reasons for the presence of reporting inconsistencies, we underline two explanations. Firstly, it is possible that authors are not attentive to the requirements of CONSORT or, despite their efforts to be compliant with the requirements, they are struggling to interpret certain items or the level of detail that is required. Examples include:

The misinterpretation of CONSORT is a major concern, as it means that essential information regarding study conduct is miscommunicated. This is particularly relevant for item 11a, as according to PRISMA item 19 [11], when assessing the risk of bias of a study it is necessary to know whether or not patients, health care providers, data collectors and outcome assessors are blinded.

Secondly, the issues described in this study might also lie with the reviewers and editors. It is possible that they are falsely reassured with regard to the reporting quality of the manuscripts, merely by the presence of a completed checklist. Moreover, the fact that the reporting inconsistencies persist throughout the editorial process might mean that editors and reviewers are not using reporting guidelines as a method to review manuscripts [12] although the CONSORT explanation and elaboration document suggests that: ‘Readers, peer reviewers and editors can also use CONSORT to help them critically appraise and interpret reports of randomized controlled trials.’

In an effort to take full advantage of requiring the submission of checklists, journals should consider clarifying their stance on whether the full checklists, or at least the checklists’ core items should be examined by editors or reviewers, or even by trained editorial assistants [13].

As the page numbers reported by authors in the checklist are not updated after the peer review process and the typesetting process, they do not correspond to the page where the information is placed in the published paper. Having to update the page numbers in the checklist from original submission to published paper could act as a checkpoint for editors or reviewers to remind them to verify whether authors are appropriately reporting the key information in the latest version of the manuscript. An alternative solution could be to ask authors to address the section and the paragraph where the information corresponding to each item is reported. This would reduce the risk of overburdening the authors and could potentially help deter the misconception that these checklists are merely bureaucratic. Furthermore, making available the updated checklist could help systematic reviewers easily and quickly find the relevant information to assess the risk of bias of the studies included in the reviews [11].

Poor-quality reporting in health research critically affects the credibility, reproducibility and generalizability of the methods and findings of randomized trials. For these reasons, further exploration of methods that will obligate authors to consistently and accurately fulfil and submit CONSORT checklists is required. Moreover, journals should consider making clear whether the checklists should be examined by editors, peer reviewers or a trained editorial assistant.