Reporting of clinical trials: a review of research funders' guidelines

Randomised controlled trials (RCTs) are planned experiments, involving the random assignment of patients to interventions, and are seen as the gold standard of methodological designs to evaluate the effectiveness of a treatment in medical research in humans [1].

However, empirical research consistently suggests that published work is more likely to be statistically significant (P < 0.05) than unpublished research, and significant results have a higher probability of being published than non-significant outcomes [2]. Evidence also suggests that published research without statistically significant results takes much longer to achieve publication than its statistically significant counterparts, further biasing evidence over time [3‐5]. Such "study publication bias" is well recognised as a potential threat to the validity of any inference made as a result of a review of the literature [6].

Outcome reporting bias (ORB) has been defined as the selection on the basis of the results of a subset of the original variables recorded for inclusion in publication of trials [7]. Recent work [8‐14] provides direct empirical evidence for the existence of outcome reporting bias. Chan et al [11] compared publications with protocols for 102 RCTs and found that statistically significant outcomes had a higher odds of being fully reported compared to non-significant outcomes for data both on efficacy (pooled odds ratio 2.4, 95% confidence interval 1.4 to 4.0) and safety (4.7, 1.8 to 12). Further, 62% of trials had at least one primary outcome that was changed, introduced, or omitted. A second study of 48 trials funded by the Canadian Institutes of Health Research found similar results [10] as have more recent studies [12, 13]. Other researchers have also concluded the problem to be major, and deserving of substantially more attention than it currently receives [15].

The registration of RCTs before initiation, on clinical trial registries [16], is one step forward in trying to prevent reporting biases as there is documentation that the trials exist, so that reviewers are able to search these registries in order to locate unpublished trials to include in a systematic review.

Organisations that fund clinical trials often issue research guidelines to aid trialists in the conduct of their study. A review of these guidelines has been conducted to discover how much guidance research funders give concerning the publication of results. That review is the focus of this report.

Particular international guidelines [ICH Harmonised Tripartite Guideline for Good Clinical Practice (E6) [17], The EU Clinical Trials Directive [18], The Declaration of Helsinki [19] and the CONSORT Statement [20] were frequently referred to by funders of clinical trials and are thus briefly described first in terms of their coverage of publication issues.

The International Conference on Harmonisation Good Clinical Practice (ICH GCP) guideline came into effect in 1997 and "is an international ethical and scientific quality standard for designing, conducting, recording, and reporting trials that involve the participation of human subjects [17]." It was introduced to provide a unified standard for the United States, EU, and Japan to facilitate acceptance of clinical data by the respective regulatory authorities [21]. The ICH GCP guidelines do not mention trial registration but do state that "All important deviations related to study inclusion or exclusion criteria, conduct of the trial, patient management or patient assessment should be described. Adherence to the study protocol is essential. If modification of the protocol becomes necessary a clear description of the rationale for the modification should be provided in a protocol amendment." This relates to the issue of ORB because if outcomes are changed from those stated in the protocol for valid reasons, this should be documented before the write up of any trial reports so that it is obvious that results have not been selectively reported based on their results.

Regarding the primary outcome of a trial, the ICH GCP guidelines state "The primary measurements and endpoints used to determine efficacy should be clearly specified. There should generally only be one primary variable. The primary variable should be specified in the protocol, along with the rationale for its selection. Redefinition of the primary variable after unblinding will almost always be unacceptable, since the biases this introduces are difficult to assess." The guidelines also state that "The results of a clinical trial should be analysed in accordance with the plan prospectively stated in the protocol and all deviations from the plan should be indicated in the study report. In the statistical section of the clinical study report the statistical methodology should be clearly described including when in the clinical trial process methodology decisions were made." This indicates that the primary outcome should usually not change between protocol and publication of the trial report; if there is a change, it should be documented and explained in the trial report.

The Declaration of Helsinki [19] was first adopted in 1964 and has since been revised five times by the World Medical Association, most recently in 2000, although clarifications have been added since (2002 and 2004). Publication is discussed, stating that "Both authors and publishers have ethical obligations. In publication of the results of research, the investigators are obliged to preserve the accuracy of the results. Negative as well as positive results should be published or otherwise publicly available. Sources of funding, institutional affiliations and any possible conflicts of interest should be declared in the publication. Reports of experimentation not in accordance with the principles laid down in this Declaration should not be accepted for publication."

The CONSORT (Consolidated Standards of Reporting Trials) Statement is intended to improve the reporting of a randomised controlled trial [20]. It comprises a checklist of 22 essential items to be reported and a flow diagram. CONSORT includes a recommendation that the methods section should include information on 'clearly defined primary and secondary outcome measures and, when applicable, any methods used to enhance the quality of measurements'. The results section of CONSORT states 'For each primary and secondary outcome, a summary of results for each group, and the estimated effect size and its precision (e.g. 95% confidence interval)' should be reported.

The EU Clinical Trials Directive 2001 (updated in 2005) [18] was transposed into UK law through The Medicines for Human Use (Clinical Trials) Regulations 2004 [22]. The aims of this Directive was to "protect the rights, safety and well being of trial participants, to simplify and harmonise the administrative provisions governing clinical trials and to establish a transparent procedure that would harmonise trial conduct in the EU and ensure the credibility of results." The Directive refers to the Declaration of Helsinki and discusses the monitoring of trials and protocol adherence, stating that "after the commencement of the clinical trial, the sponsor may make amendments to the protocol. If those amendments are substantial and are likely to have an impact on the safety of the trial subjects or to change the interpretation of the scientific documents in support of the conduct of the trial, or if they are otherwise significant, the sponsor shall notify the competent authorities of the Member State or Member States concerned of the reasons for, and content of, these amendments and shall inform the ethics committee or committees concerned." This relates to the issue of ORB due to the reasons stated above regarding protocol deviations/amendments for the ICH GCP guidelines.

A survey of major research organisations and UK charities (see Additional file 1, Appendix 1), using the same list as The DAMOCLES project [23], was undertaken to examine whether existing guidelines for research conduct refer to issues related to reporting biases. UK based charities listed on the Association for Medical Research Charities website were contacted (see Additional file 1, Appendix 1). National and international organisations were selected if they funded research in the United Kingdom, since it was felt important to obtain information from different funding sources and from different international geographical locations. Research organisations were eligible if they fund clinical trials.

This review provides a narrative summary of the guidelines that are available from major organisations across the world and UK charities that fund RCTs. We found that funders of clinical trials have not yet incorporated recent concerns about ORB into their recommendations for researchers. There is a need to provide detailed guidance for those conducting and reporting clinical trials to help prevent the selective reporting of outcomes.

In this review we focussed on guidelines for good research practice for clinical trials. The main limitation of this review is that the search of organisations was not exhaustive. We concentrated mainly on organisations from Europe, USA, Australia and South Africa, and only UK charities have been investigated. However, the ICH guidelines, discussed below, are international.

The majority of guidelines provided no references to the problem for several of the questions in the questionnaire (see Additional file 1, Appendix 3); these were mainly questions specifically regarding outcomes. This is likely to be due to the fact that awareness of the problem of ORB has been limited over the past decade. As we have found, the importance of this issue has not filtered through to these guidelines.

Many of the guidelines state that a trial should be registered before it begins. The purpose of registration is to inform other researchers of trials that are being conducted; the aims include reducing publication bias and the duplication of research [6].

Several guidelines that we obtained discuss the importance of adherence to the protocol and state that amendments to the protocol should be reported. This issue is related to ORB as outcomes that are specified as primary or secondary in the protocol should be written up as such in the final report. Insertion of new outcomes and deletion of old outcomes should not occur unless an official amendment to the protocol was made, and such changes should be reported in journal articles. The ICH E3 [25] guideline states that "if the protocol did not identify the primary variables, the study report should explain how these critical variables were selected (e.g., by reference to publications, guidelines or previous actions by regulatory authorities) and when they were identified (i.e., before or after the study was completed and unblinded)." The Medicines and Healthcare products Regulatory Agency (MHRA) state that "new questions based on the observed data may well emerge during the analysis. This additional work should be strictly distinguished in the report from the work which was planned in the protocol." No such reports were found in the cohort of 102 trials studied by Chan et al [11] despite over half having changed their primary outcomes in some way.

Publication is often mentioned in funders' guidelines, stating that work should be published in a peer reviewed journal or at least stating that it should be disseminated. The Association of the British Pharmaceutical Industry states that work should be published and refers to guidelines on good publication practice [26], which aim to ensure that clinical trials sponsored by pharmaceutical companies are published in a responsible and ethical manner.

Many RCTs are carefully monitored by trial steering committees and this requirement is stated in several of the guidelines/terms and conditions. Monitoring tends to focus on whether trialists are following correct procedures. Therefore, issues such as protocol adherence and trial registration should be noticed and highlighted if they are not done correctly. Once a trial has been completed, such oversight may diminish. Ensuring that trial results are published may also be a concern of the data monitoring committee [23].

These four issues – registration of a trial, protocol adherence/amendment, publication and monitoring against guidelines – are important with regard to ORB, as the more rigorous the guidelines are in these areas, adherence makes ORB less likely.

Statements found in the guidelines generally refer to publication of negative and positive findings which could be interpreted as referring to study publication bias (i.e. non-publication of whole studies) or ORB (selective non-publication of outcomes). The empirical evidence of publication bias was first seen in the early 1990's [2] and much work has now been done in the area of publication bias since [6]. By contrast, studies providing the empirical evidence for ORB were published more recently between 2002 and 2008 [2]. Some of the more detailed and most often referred to guidelines, such as ICH [17, 27, 25, 28] and MRC Good Research Practice guidelines [29], were written before 2000, when ORB first began to be recognised [7]. However, the Department of Health's Research Governance Framework [30] was updated in 2005 and provides no guidelines aimed specifically at preventing ORB.

Organisations tended to have their own guidelines whereas the charities mostly had terms and conditions but referred to some of the organisations' guidelines. This may reflect some of the charities being small and that some had only recently begun to fund RCTs.

The CONSORT Statement [20] addresses the write up of the final report and does not link this explicitly to the trial protocol. It does not specifically refer to ORB as it only states that outcome measures should be specified in the methods section and fully reported in the results section. Although this recommendation attempts to prevent partial reporting of outcomes (e.g. p-value only), and in addition may help identify outcomes that have not been reported, these recommendations alone will not prevent the selective reporting of outcomes. The CONSORT statement does not state that all outcomes that were included in the protocol should be stated in the methods section and reported in the results section of the final report. It also does not state that outcomes should not be upgraded or downgraded between the protocol and final report, or that new outcomes should not be added or old outcomes deleted based on their results. Therefore even when trialists follow the CONSORT Statement, it would be impossible to determine whether all outcomes in the protocol are reported in the final report. However, these concerns will be addressed in the revision of CONSORT that is under way. (DG Altman, personal communication)

Since this work has been completed, the members of the WHO Registry Platform Working Group on the Reporting of Findings of Clinical Trials have proposed that "the findings of all clinical trials must be made publicly available [31]."

In the USA legislation is in place so that legally researchers are required to make their findings publicly available within a specific timeframe [32]. The legislation states that "The expansion of the registry data bank to include results of clinical trials; beginning not later than 90 days after the date of the enactment of the Food and Drug Administration Amendments Act of 2007, for those clinical trials that form the primary basis of an efficacy claim or are conducted after the drug involved is approved or after the device involved is cleared or approved, the Secretary shall ensure that the registry data bank includes links to results information." It also includes specifics on the reporting of outcomes: "The primary and secondary outcome measures as submitted and a table of values for each of the primary and secondary outcome measures for each arm of the clinical trial, including the results of scientifically appropriate tests of the statistical significance of such outcome measures." It is hoped that these initiatives will reduce the problem of ORB.

This review forms part of the Outcome Reporting Bias in Trials (ORBIT) project [33]. It has been funded to investigate how prevalent outcome reporting bias is in trials within systematic reviews and whether this has a large impact on the conclusions of the reviews, as well as trying to understand why trialists do not report all outcomes and investigating the prevalence of ORB in individual patient data reviews on the Cochrane Collaboration Individual Patient Data Methods Working Group register. The identification of this gap in the guidelines is part of a suite of initiatives to reduce the problem of ORB.

Our review indicates that there is a need to provide more detailed guidance for those conducting and reporting clinical trials to help prevent the selective reporting of outcomes. Current guidelines need to be updated, as many include statements regarding the publication of 'negative' studies to prevent publication bias but do not go as far as mentioning ORB. In the MRC guidelines the statement 'all outcomes stated in the protocol should be fully analysed' needs to be extended to specify that prespecified primary and secondary outcomes should be fully reported and that outcomes should not be selected for inclusion in the final report based on their results. Recommendations for text to include in research funders' guidelines are suggested (see Additional file 4).

The next step is to conduct a systematic review of empirical studies that have compared protocols to publications, to assess their adherence to specified guidelines. It would then be important to conduct a survey of trialists to see which guidelines they follow and their adherence to them, to assess their understanding of the issues discussed in this paper.