Background
Methods
Results
Researchers’ inaccurate statements about outcome reporting
Theme | Trialists’ response | Issue |
---|---|---|
Timing of pre-specification | ||
Stating or implying that pre-specification after trial commencement is acceptable | “The prespecified analysis of PATHWAY-2 precisely followed a detailed statistical analysis plan (SAP) that was published in BMJ Open before data lock and unblinding of data (4), and was provided in full to The Lancet, dated and signed before any unblinding or analysis. All primary and secondary endpoints reported in The Lancet were listed at ClinicalTrials.gov before data lock and unblinding” (Trial 57, Lancet, 02/04/16). | The authors suggest that pre-specification should happen before “data lock and unblinding”. However, CONSORT item 6b requires that trial reports declare and explain “any changes to trial outcomes after the trial commenced, with reasons” in the paper reporting the results of the trial. |
Failure to report changes to pre-specified outcomes in paper | ||
Failure to recognise that post-commencement changes are acceptable but should be declared in the paper reporting the results of the trial | “Length of stay in survivors and days to death (the fifth so-called new endpoint) are components of length of hospital stay, but they were presented separately to prevent bias from higher mortality in either group that resulted in a difference in length of stay between groups” (Trial 27, Lancet, 30/01/16). | This change from protocol was not mentioned or explained in the paper. CONSORT item 6b requires that trial reports declare and explain “any changes to trial outcomes after the trial commenced, with reasons” in the paper reporting the results of the trial. |
Stating that pre-specified outcomes missing from the trial report, or declarations of changes, will be reported elsewhere but failing to declare this in the trial report | “With regards to cost outcomes, both the primary and four of the eight so-called missing secondary outcomes (therapy costs, quality of life, institutionalisation, and cost-effectiveness ratios) will be presented in a later publication as stated in the headline paper” (Trial 27, Lancet, 30/01/16). | Changes from pre-commencement outcomes should be declared in the paper reporting the results, as above. Note that while the authors state the additional outcomes “will be presented in a later publication as stated in the headline paper”, there is no such disclosure in the paper; we asked the authors to identify it in our follow-up letter; this was not published and we received no reply. |
Registries | ||
Incorrect statements about registries | “Trial registries often do not request or have space for sufficient detail about secondary outcomes” (Trial 10, Lancet, 16/04/16) | There are no restrictions on posting secondary outcomes to registers. |
Multiple sets of discrepant pre-specified outcomes | ||
Making reference to protocols that are publicly inaccessible, or were published after trial commencement, which allegedly contain outcomes that are discrepant with registry entries but consistent with the published report | “The trial was registered at ClinicalTrials.gov where we indicated that the primary outcome was... The protocol was also sent to The Lancet shortly after the study started and a summary of the protocol was published. The submitted protocol clearly indicated that the primary outcome was… all outcomes at 2 years prespecified in the submitted protocol were reported but were not included in the published protocol” (Trial 10, Annals, 16/04/16). | The argument appears to be that there is a publicly inaccessible pre-commencement protocol that contains pre-specified outcomes different from those in the contemporaneous pre-commencement registry entry. There is no methodological justification for discrepant outcomes between registry entry and protocol for the same trial at the same time point: the two should be the same, and changes after trial commencement should be discussed in the results paper. Registries were devised as a publicly accessible location for trial information specifically to prevent selective outcome reporting. Having multiple discrepant sets of pre-specified outcomes, with the option to choose between multiple discrepant documents, undermines the purpose of pre-specifying outcomes. |
Making reference to multiple discrepant sets of pre-specified outcomes | “All primary and secondary endpoints reported in The Lancet were listed at ClinicalTrials.gov before data lock and unblinding. The protocol… also posted on the public domain, EudraCT, before patient recruitment, correctly identified the primary objective… The primary outcome measure was correctly stated on EudraCT” and so on (Trial 57, Lancet, 02/04/16). | This trial had multiple different sets of conflicting “prespecified” outcomes in different locations at similar dates. For example, different outcomes are registered on ClinicalTrials.gov in February and July 2015, and both sets of outcomes in turn are inconsistent with those in the protocol of June 2015. |
Issues with time points | ||
Incorrect statements around issue of multiple time points | “We do not see how these multiple measurement time points should be counted as separate outcomes, as the procedure of the COMPare team seems to propose. We think this leads to misuse of overall statistics on their website and exaggerated conclusions about the magnitude of outcome switching in RCTs” (Trial 70, BMJ, 04/02/16). | The trial report states “Secondary outcome measure were symptoms of depression and anxiety measured with the CES-D and HADS-A at baseline and at 3, 6, 9, 12, 18 and 24 months”; and all of these time points are then separately reported in Table 5 of the trial report as a mean with a standard deviation. These are all outcomes, according to the CONSORT guidelines. None of these time points was pre-specified before trial commencement; therefore, 21 non-pre-specified secondary outcomes were reported. |
Researchers’ response styles
Researchers’ response styles | |
Diversion
| |
1. Stating that trials are hard work to conduct | |
2. Stating that other issues are more important | |
3. Response based on issues not raised by COMPare | |
4. Ad hominem | |
Challenging legitimacy of discussion
| |
1. Expressing a preference for conventional peer review over open post-publication critical appraisal | |
2. Disagreement with the general approach of COMPare/CONSORT | |
3. Asserting that there should be the opportunity to post comments on COMPare’s own raw data sheets online | |
4. Stating that they applaud the overall goal, followed by a caveat | |
Trust | |
1. Statement that discrepancies were not motivated by desire to manipulate findings | |
2. Stating that outcome misreporting doesn’t matter if the main results of the study are unlikely to be affected | |
Incorrect statements about outcome reporting in their own paper
| |
1. Denying that specific misreported outcomes were indeed misreported | |
2. General denial of COMPare’s findings | |
Technical/Rhetorical
| |
1. Appealing to the existence of a novel category of outcomes whose results need not be correctly reported | |
2. Stating that space constraints prevent all pre-specified outcomes from being reported | |
3. Stating that it is not necessary to pre-specify some outcomes as they are “necessarily implied” by other outcomes | |
4. Inaccurate statements about COMPare’s methods |
Theme | Trialists’ response | Issue |
---|---|---|
Diversion | ||
Stating that trials are hard work to conduct | “Our 13 authors and 44 collaborators dedicated almost a decade to bringing to fruition the first prospective comparison of drug treatments for resistant hypertension”. “The obstacles to performing all clinical trials these days are immense” (Trial 57, Lancet, 02/04/16). | |
Stating that other issues are more important | “We also believe that larger issues are at stake in keeping control over the procedure of a pragmatic trial that merit more discussion on its influence than outcome counting, e.g. the development and implementation of interventions, training professionals to comply with strict protocols, setting up a trial in multiple centres using the same procedures, keeping contact with participants to avoid drop-out (often impossible to avoid due to illness or death), blinding of outcome assessors, medical ethics, phishing incidents [2], etcetera” (Trial 70, BMJ, 04/01/16). | |
Response based on issues not raised by COMPare | “The only deviation we can see from the ISCRTN entry is the fact that we exceeded our initial trial sample size (691 in the published report versus 600 in the trial registry). We don’t think this is a hanging offence, and we did this to ensure we maintained our level of pre-specified statistical power when follow up was a little lower than we anticipated (such things do happen). We note that trials commonly fail to achieve their pre-specified sample size ...” (Trial 47, BMJ, 21/12/15). | All examples given here discuss issues that COMPare did not raise. None of them justifies undeclared discrepancies between pre-specified and reported outcomes. For trial 47, for example, the only publicly accessible pre-commencement outcomes were in the ISCRTN registry entry. This contains 11 pre-specified secondary outcomes, three of which are not reported in the BMJ paper, with no declaration of their omission. |
Ad hominem comments | “In the last few months, the COMPare team has monitored five top journals to analyse trials on outcome switching. Based on their interpretation of the CONSORT guidelines, comments on outcome switching have been produced. However, until now, their work has not gained or secured widespread support - neither by funders (their project is paid out-of-pocket) nor by the editors of the five top journals who do not seem keen to publish their comments...” (Trial 70, BMJ, 01/04/16). “With their approach of criticising and not being open to discussion... COMPare places themselves outside the research community. Although it can be debated to what extent it is possible to develop and criticise an aspect of science from the outside by persons not directly involved [4], we believe the research community should be critical, but with the aim to support and improve science” (Trial 70, BMJ, 01/04/16). | |
Challenging legitimacy of discussion | ||
Expressing a preference for conventional peer review over open post-publication critical appraisal | “In retrospect, we believe that expert and constructive peer reviews are sufficient to raise science to a higher level” (Trial 70, BMJ, 04/01/16). | |
Disagreement with the general approach of COMPare | “The COMPare team might well catch some true outcome switching and ‘fishing’; however, in their net they are also catching researchers who have not switched outcomes or selectively reported, but have simply made minor errors of omission in their registry entries” (Trial 10, Lancet, 23/07/16). | |
Asserting that there should be the opportunity to post comments on COMPare’s own raw data sheets online | “We hope the COMPare project team will take into account our comments, post our response on their website ...” (Trial 17, Lancet, 14/05/16). | We set out to correct the record of misreported trials in the journal where they were misreported. Although we shared our raw underlying data sheets in an online repository, we felt that the appropriate place for a critical discussion about the correct reporting of the pre-specified outcomes was the journal where the trial results were reported. Consigning the discussion to our online data repository, rather than journal correspondence, would significantly reduce the visibility of a constructive discussion around correct outcome reporting. |
Stating that they applaud the overall goal, followed by a caveat | “While we support the principles of COMPare ...” (Trial 25, Annals, 11/12/15). | |
“Trust the trialist” | ||
Statement that discrepancies were not motivated by desire to manipulate findings | “In response to Dale and colleagues, it should be noted that the PATHWAY programme was devised by eight academic investigators with no vested interests other than a wish to answer previously intractable questions arising from centuries of cumulative experience of hypertension practice and trials” (Trial 57, Lancet, 02/04/16). | It is unlikely that all outcome misreporting reflects a deliberate attempt by trialists to misrepresent a study’s findings; however, a culture of permissiveness around correct outcome reporting does permit misrepresentation. |
Stating that outcome misreporting doesn’t matter if the main results of the study are unlikely to be affected | “If Dale and colleagues’ inference is that spironolactone’s overwhelming superiority over licensed antihypertensive drugs is due to selection of multiple results” (Trial 57, Lancet, 02/04/16). | It is unlikely that all outcome misreporting changes or exaggerates the overall finding from a trial. However, the evidence from the current systematic review shows that this tends to be the case, and a culture of permissiveness around correct outcome reporting facilitates such misrepresentation. |
Incorrect statements about outcome reporting in their own paper | ||
Denying that specific misreported outcomes were indeed misreported | “We have clarified in the Methods section that physician diagnosed pneumonia was not a primary outcome” (Trial 27, Lancet, 30/01/16). | COMPare searched the paper repeatedly and found no such disclosure; in fact, the paper in question explicitly describes physician-diagnosed pneumonia as the “co-primary outcome”. |
General denial of COMPare’s findings | “We whole heartedly agree with the scrutiny of endpoints in high-profile clinical trials such as ours that Dale and colleagues have performed. It is reassuring that this analysis indicates that our Article is correctly reported and as such is consistent with the scientific and clinical intent of the trial as described in the protocol” (Trial 56, Lancet, 11/06/16). | This trial was not correctly reported, as explained in the COMPare letter to which this comment was a reply: two pre-specified outcomes were unreported, and four additional outcomes were reported without disclosing that they were novel. |
Technical/Rhetorical | ||
Appealing to the existence of a novel category of outcomes whose results need not be correctly reported | “None of these are key secondary endpoints” (Trial 56, Lancet, 11/06/16). | The outcomes pre-specified in the registry entry were not reported for this trial. The phrase “key secondary outcomes” is one used by the WHO, in their list of 20 items that should be in all registry entries, to denote all the secondary outcomes pre-specified in the registry, which should all be reported. |
Stating that space constraints prevent all pre-specified outcomes from being reported | “As indicated by Aaron Dale and colleagues, two of three pre-specified primary outcomes were not fully described in the results section of our Article for word limitation reasons” (Trial 29, Lancet, 11/06/16). | While the authors appeal to length limits, this paper reported an additional outcome (“distribution of clinical stages of cancer”), stratified by clinical stage, percentage of reported breast cancer–positive patients and relative sensitivity. This resulted in their reporting 16 additional outcomes that were not pre-specified (none of which was declared as non-pre-specified). Reporting non-pre-specified outcomes was common throughout the project. |
Stating that it is not necessary to pre-specify some outcomes as they are “necessarily implied” by other outcomes | “the adjudication of the pre-specified endpoints of any myocardial infarction, target vessel myocardial infarction, revascularisation, or target vessel revascularisation, necessarily implies the assessment of the non-target vessel myocardial infarction and the non-target vessel revascularisation” (Trial 17, Lancet, 14/05/16). | This is an additional outcome that was not pre-specified. Clear pre-specification is required by registers, regulators, and CONSORT in order to avoid selective reporting. Unnecessary flexibility leaves trialists the option to selectively report outcomes, with no public record of the original intentions of the trial. |
Inaccurate statements about COMPare’s methods | “... we suggest that a trial’s published protocol should also be reviewed by COMPare in tandem with its Registry entry as part of their process” (Trial 25, Annals, 11/12/15). | The COMPare method used both: protocols were used preferentially; if these were unavailable, or published after trial commencement, then the trial registry entries were used instead. |
Researchers’ correction of errors
Theme | Trialists’ response | Issue |
---|---|---|
Acknowledgement of error by trialists | ||
Clear acknowledgement of CONSORT breach and then a clarification | “One prespecified secondary outcome from the protocol (assisted vaginal delivery) was omitted from the analysis plan in error, and, therefore, not reported” (and further corrections for same trial) (Trial 46, Lancet, 14/04/16). | We regard a clear correction as best practice. In our cohort of 58 submitted letters, it was uncommon. |
“We did not report the results of the Steatotest as we had incomplete data for this because of sample haemolysis… For the purpose of transparency, we include the median values at baseline…” “We presented data on three parameters that had not been predefined as secondary endpoints…” (Trial 56, Lancet, 11/06/16). | Note that these trialists also introduced a spurious distinction regarding unreported outcomes (“none of these are key secondary endpoints”). | |
“We accept that our reporting of the change in the primary depression outcome in the BMJ paper could have been better ... we accept that, by rule, we have failed to be entirely transparent and we meet their criteria for such a rating” (Trial 47, BMJ, 14/01/16). | ||
Correcting the wrong error | “We have reviewed these discrepancies and concede that we failed to fully update the trial registry” (Trial 8, Lancet, 13/02/16). | The error was not failure to update the registry entry but rather failure to report pre-specified outcomes or document discrepancies. |