Methods
As part of ongoing research into biased reporting of trial outcomes, we assessed the role of editorial and peer review processes in publication bias. We evaluated whether submitted manuscripts with negative outcomes were less likely to be published than studies with positive outcomes, and determined the influence of sponsorship in this context. We retrospectively reviewed manuscripts submitted to eight journals from January 2010 through April 2012. One general medical journal (British Medical Journal) and seven specialty journals (Annals of the Rheumatic Diseases, British Journal of Ophthalmology, Diabetologia, Gut, Heart, Journal of Hepatology, and Thorax) were included. Journals were selected based on impact factor (journals indexed with the highest impact factors within subject categories, according to Institute for Scientific Information Journal Citation Report 2011), and the number of drug RCTs published in 2010–2011. Manuscripts reporting results of RCTs were included, if at least one study arm assessed the efficacy or safety of a drug intervention and a statistical test was used to evaluate treatment effects (n = 472). Two authors (MvL, HJO) extracted information on trial results and financial or material support from manuscripts.
During the start of this evaluation, we encountered several problems related to (1) the classification of outcomes, (2) inclusion of post-hoc and subgroup analyses and follow-up studies of RCTs in the study sample, and (3) assessment of the role of the funding source. These problems hampered unequivocal judgment on sponsorship and the direction of trial outcomes. We composed a list of recommendations addressing these problems, to facilitate a standardized assessment of manuscripts included in our study and future studies on publication bias. To assess internal validity, reliability and usability of the recommendations were tested through evaluation of manuscripts submitted to journals included in our study. Minor adjustments were made during this process, and a comprehensive list of recommendations was created.
Discussion
In this study, several problems related to the classification of outcomes and sponsorship were identified. We have formulated recommendations addressing these problems to facilitate a uniform assessment of manuscripts included in studies on publication bias.
In this article, we have focused on the assessment of publication bias related to funding source. However, the relationship between favourable outcomes and sponsorship is much more complex. Numerous systematic reviews have found that industry sponsorship is associated with results that are favourable to the product of the company funding the trial [
4,
20,
21]. Besides publication bias, there are other ways by which outcomes may be influenced. Industry bias may occur through the choice of comparators, dosing and timing of comparisons, coding of events and selective data analysis, interpretation of data and selective outcome reporting [
6,
21,
22]. Although we have concentrated on publication bias, our recommendations are generally applicable to studies comparing industry versus non-industry trials in relation to favourable outcomes. Furthermore, we only considered drug RCTs, but the recommendations are equally relevant to device studies, as these are also often sponsored by companies with a financial interest in the study outcomes.
In our recommendations, reported results are classified as positive or negative. Trials in which primary endpoints are reached or study hypotheses are confirmed are considered to be positive. Possible alternatives to categorize trials would include ‘significant’ vs ‘non-significant’, or ‘favorable’ vs ‘unfavorable’ findings. However, use of these terms would impede the classification of non-inferiority or equivalence trials and studies with safety parameters as the primary endpoint. We have not included a third ‘neutral’ category, as we aimed to keep the classification as straightforward as possible. Moreover, previous studies on publication bias found relatively low numbers of studies with neutral or unclear results [
23‐
25], so we included these in the ‘negative’ category.
For the classification of outcomes, we have concentrated on results reported for the primary endpoint. However, in several studies on industry bias, trials are categorized as positive if the conclusion is favourable (i.e. the authors recommend the test drug), not the reported results [
13]. In this article, we recommend considering conclusions only when trials cannot be classified based on reported results. Previous studies reporting on concordance between study results and conclusions found that industry-sponsored trials were less concordant than non-industry studies [
6]. This lack of concordance, and the finding that industry-sponsored studies are more likely to have favourable conclusions [
6], may be explained by the use of distorted presentation or ‘spin’ in papers [
26]. As results may be more objective measures of treatment effects, we chose to concentrate on the assessment of results. Future studies on publication bias should clearly state whether trial results or conclusions are being examined.
Trials were classified as non-industry, industry-supported or industry-sponsored. One of the most fundamental differences between industry-supported and industry-sponsored studies relates to the overall responsibility for the conduct of the trial as defined by ICH-GCP guidelines. Industry-supported trials are done under the responsibility of non-profit organizations in contrast to industry-sponsored studies. This distinction is usually not made in previous literature but is very relevant in terms of quality assessment of reported research. We have not included a category of ‘sponsorship not stated’. Many journals nowadays ask for information on received funding and the role of the funding source, and require that manuscripts on RCTs conform to ICMJE requirements and CONSORT guidelines. Therefore, most trials recently submitted should report on funding and be registered in a trial registry, making information on sponsorship traceable. However, the non-industry category may include trials that received industry support, if authors failed to disclose funding. In a recent Cochrane review on sponsorship and research outcomes, studies were also coded as non-industry if it was not reported who sponsored the study. The review authors stated that some of these trials were likely industry-sponsored, but no changes in results were seen when studies without sponsorship statements were excluded [
6]. In addition, industry-sponsored studies may be misclassified as industry-supported, as apparently independently conducted industry-supported trials may have unreported sponsor involvement [
27].
For future studies on publication bias, it is important that the quality of reporting on design and sponsorship of trials is improved. Journals should actively enforce the CONSORT Statement and ICMJE requirements, and could require authors to complete a form for disclosure of sponsorship and support received for the reported trial [
28].
We emphasize that our recommendations are only validated for use by three investigators. In order to prove whether the recommendations are acceptable to researchers in general, it is essential that the external validity of this proposed classification system is determined in a study undertaken by different researchers on a different sample of manuscripts. To establish external validity, a Delphi consensus technique could be conducted among a larger group of researchers. Future research should focus on whether the classification system is adopted in practice by investigators undertaking studies on trial outcomes and sponsorship.
In conclusion, the recommendations proposed in this article represent a first step towards a uniform method of classifying trial outcomes and sponsorship. This is essential to draw valid conclusions on the role of the funding source in publication bias and will ensure consistency across future studies.
Funding
This work was supported by an unrestricted educational grant was received from MSD for this research. MSD (Merck, Sharp & Dohme) B.V. is a Dutch subsidiary of Merck & Co., Inc located in Oss, The Netherlands. The funder had no role in the study’s design, data collection and analysis, preparation of the manuscript or the decision to publish. MSD has not commented on or reviewed the manuscript.
Competing interests
All authors have completed the ICMJE uniform disclosure form at
http://www.icmje.org/coi_disclosure.pdf (available on request from the corresponding author). HJO is a paid employee from Teva Pharmaceuticals next to his professorship at the university; JO is the immediate past-president of the World Association of Medical Editors (WAME). MvL reports no competing interests.
Authors’ contributions
The ICMJE criteria for authorship are read and met by MvL, JO and HJO. HJO originated the idea for this manuscript with support from MvL and JO. MvL wrote the first draft of the paper. The majority of the research underpinning the paper was undertaken by MvL. JO has reviewed and revised the article together with HJO and MvL. All authors agree with the manuscript’s results and conclusions and approved the final version submitted for publication. HJO is the guarantor of this article.