Opportunities for selective reporting of harms in randomized clinical trials: Selection criteria for non-systematic adverse events

The U.S. Food and Drug Administration (FDA) and other regulators may approve drugs and biologic agents for marketing when the results of randomized clinical trials indicate that their potential benefits outweigh their potential harms, which are often called “adverse events” (AEs). In clinical trials, AEs can be collected non-systematically when participants report them spontaneously to investigators or in response to open-ended questions such as “have you noticed any symptoms since your last examination?” (Table 1) [1‐4]. Specific AEs can also be collected systematically by assessing their presence or absence using the same method for all participants in a trial (e.g., clinical examination, questionnaires, and medical instruments [1, 2]). Previous studies have described non-systematic AEs as “passively collected” and systematic AEs as “proactively collected” [1, 2]. AEs are categorized as “serious” when they lead to or prolong hospitalization, cause death, or disrupt normal life functions [4, 5].

Physicians, patients, and policy makers rely on systematic reviews and clinical practice guidelines to make medical decisions. Syntheses of clinical trial findings should include all available evidence; however, they are often based on AE information reported in public sources, such as journal articles (Table 1) [6]. Many systematic reviews that plan to synthesize AEs ultimately do not address AEs [7]. By contrast, regulators have access to non-public sources of trial information. Although previously non-public sources are becoming available for some trials [8‐10], including through data sharing services such as Vivli [11], Yale Open Data Access (YODA) [12, 13], and clinicalstudydat​arequest.​com, clinical study reports (CSRs) and individual patient datasets (IPD) remain unavailable for many trials. Consequently, the results of many trials are available only in public sources [14, 15], which are often incomplete [15‐24]. Reporting bias, which is the selective reporting of research results, occurs when reporting is influenced by the nature of the results (e.g., the direction, magnitude, or statistical significance of the results). Reporting bias may lead to overestimating potential benefits of an intervention [15‐20] and underestimating potential AEs [22, 25‐32].

There is little evidence about the methods trialists use to select AEs for reporting, and there is little evidence about whether those methods contribute to reporting bias. Previous guidance for reporting adverse events has discouraged vague statements about AEs [36, 37], and the Consolidated Standards of Reporting Trials (CONSORT) Extension for Harms discourages authors from reporting AEs above frequency thresholds (e.g., > 10%) [38]. Nonetheless, there is little evidence about how different trials and different sources for a single trial report AEs. The objectives of this study were to (1) compare selection criteria for reporting non-systematic AEs (i.e., the methods authors use to decide which AEs to include in a particular source as illustrated in Table 2, (2) examine how different selection criteria could affect AE reporting, and (3) assess how different selection criteria could impact meta-analyses of AEs.

This analysis is a sub-study of the Multiple Data Sources (MUDS) study. MUDS was a methodologic study, the overall objective of which was to examine multiple data sources about trials and to determine whether the information source could affect the conclusions of systematic reviews and meta-analyses. The published protocol and amendments [24, 39] describe the study methods. Briefly, we searched for public and non-public sources (see definitions in Table 1) and requested additional non-public sources (e.g., CSRs that they had not been disclosed previously) from the companies that manufacture gabapentin and quetiapine as described elsewhere [22, 39, 40]. Electronic searches included the Cochrane Central Register of Controlled Trials (CENTRAL), PubMed, Embase, LILACS, and CINAHL through 2 March 2015 for gabapentin trials and through 26 January 2015 for quetiapine trials. For trial registrations, we searched the International Clinical Trials Registry Platform Search Portal and ClinicalTrials.gov through 10 October 2014.

In this study, we examined how reports of clinical trials use selection criteria for reporting non-systematic AEs. We found that public sources and CSR-synopses applied selection criteria while CSRs reported all AEs; consequently, most AEs and serious AEs were not reported in public sources. In public sources, every combined selection criterion we found included a numerical threshold; however, numerical thresholds were not consistent across trials or across sources for individual trials. Some selection criteria also included requirements related to the participant group and the difference between groups.

When we applied various selection criteria to eligible trials, we observed meaningful differences in the number of AEs that would be reported. Trialists might have used selection criteria to identify what they consider the most important AEs; however, we are unaware of any consensus about which AEs are most important for these drugs and conditions. We found no evidence that patients or clinicians were involved in deciding which AEs would be reported in public sources. Instead, it appears that public sources included the most common AEs. Trialists could have performed analyses similar to ours in which they applied different selection criteria and then decided post hoc which selection criteria would allow them to report, or not report, particular AEs. In a previous study, we found that trialists did not “group” AEs for reporting [22], which would be another method to consolidate AEs for reporting that has been advocated elsewhere [37]; however, grouping AEs can also disguise important AEs by combining them with less important AEs [43].

Evidence syntheses (e.g., systematic reviews, clinical practice guidelines) could help identify rare AEs if all observed AEs were available for all trials [44]; however, rare AEs cannot be identified when clinical trials report only those AEs occurring above numerical thresholds. Just as selectively reporting potential benefits based on quantitative results leads to biased meta-analyses [45‐48], reporting AEs based on trial results will necessarily lead to biased overall estimates because only “positive” signals will be available. Selection criteria make it impossible for systematic reviewers and meta-analysts to accurately assess the AEs caused by medical interventions.

We found that no public source reported all AEs for any trial. Only serious AEs and AEs occurring in more than 5% of participants are required to be reported in www.​ClinicalTrials.​gov according to the Food and Drug Administration Amendments Act of 2007 (FDAAA) [1, 49]. Doctors and patients often rely on post-marketing surveillance studies to identify rare AEs, including serious AEs, yet these studies lack comparison groups that are present in clinical trials. It might be possible to detect rare AEs in clinical trials and to calculate between-group differences, if investigators would not use numerical thresholds to determine which AEs to report. Investigators must ensure that for common AEs, such as headache, data are collected consistently across intervention groups so that proportions in each group can be compared. This is difficult when AEs are collected non-systematically. Although systematically collecting AEs would result in more trustworthy and usable information about effects between groups [22, 23], it is not always possible, or even desirable, to anticipate which AEs patients will experience.

Authors have previously discouraged the use of selection criteria [36‐38]; however, trials would have to report hundreds or thousands of AEs if selection criteria were not used. Making CSRs and equivalent reports for non-industry trials public, and grouping AEs for analysis and reporting, would partially address the problem of reporting many AEs in journal articles and other public sources. A complete solution to the problems we have identified is not obvious.

Multiple sources of public AE information from trials (e.g., journal articles, FDA reviews), which may be written by different authors for different purposes, lead to inconsistent information for patients and physicians [50]. For example, FDA reviewers have access to CSRs that report all AEs that occurred in each trial. FDA reviewers consider pre-clinical data (e.g., pharmacokinetics, animal trials) and apply clinical and statistical judgment when deciding what to report in medical and statistical reviews about new drugs and biologic agents. The FDA and manufacturers also decide what to include in prescribing information (drug “labels”) written for patients and doctors. By comparison, other stakeholders obtain their information about interventions from a variety of sources, and those sources may use different selection criteria for reporting. For example, individual authors and journal editors can decide what to report in journal articles, which vary tremendously. Different selection criteria across reports of clinical trials, including multiple reports of the same trial, lead to inconsistent and confusing information for stakeholders; consistent standards for reporting AEs, and open access to trial information (e.g., CSRs) could help.

Because current methods for selecting AEs lead to biased reporting that will necessarily produce incorrect effect estimates in systematic reviews and clinical practice guidelines, regulators, lawmakers (e.g., the U.S. Congress) and journals could require that trialists report all AEs on www.​ClinicalTrials.​gov or other registries. When it is not feasible to report all AEs in a given source (e.g., a conference abstract), the source could direct readers to additional information in a registry such as www.​ClinicalTrials.​gov. FDA and other regulators could make all AEs publicly available. If all AEs are not made available by regulators or by trialists, systematic reviewers and meta-analysts should interpret results with extreme caution and explain the limitations of using only publicly available data.

There is a pressing need to make clinical trial data available to the public, especially CSRs and IPD; however, sharing CSRs and IPD will not solve all problems identified in our research. First, reports describing hundreds of AEs might overwhelm physicians and patients by including “too much” information [51]. Many AEs reported in CSRs and IPD are not intervention-related; selection criteria might have been used to help decision-makers identify AEs that are caused by medical products. Sharing lengthy and confusing reports and datasets could increase the appearance of transparency while actually disguising important information. Second, reanalyzing clinical trial data is time consuming and therefore expensive, and reanalysis should not be necessary to identify AEs. Most decision-makers want trustworthy summaries of clinical trials.

To avoid reporting bias, and to avoid overwhelming patients and physicians with information, trials could pre-specify which AEs will be collected and reported in summaries such as journal articles [36]. Core outcome sets are the minimum sets of outcomes to include in trials [52, 53], and they normally focus on the assessment of potential benefits for people seeing treatment for a particular health problem. Because different types of interventions for the same health problem might be associated with different AEs, and because a single intervention might be used to treat several health problems (e.g., quetiapine is used to treat bipolar depression and schizophrenia), core outcome sets focused on the AEs associated with a particular intervention or group of interventions could also improve the comparability of clinical trials. Notably, core outcome sets for AEs would help trialists identify and report those AEs that are most important to patients, not just the AEs that occurred most often [22, 23, 51]. As described elsewhere, systematic assessment of important AEs would produce much more useful information compared with non-systematic assessment [22, 23].

It is a limitation that we included only two drug-indications, and that we had non-public information for a minority of trials in each case study. The use of selection criteria and the extent to which they are pre-specified and consistent might differ across companies and investigators. Nonetheless, the results of this methodological investigation highlight fundamental problems with methods currently used to report AEs that occur in clinical trials.

Stakeholders require complete AE information to make informed healthcare decisions. Our findings show that systematic reviews and clinical practice guidelines are unlikely to provide accurate effect estimates for AEs because the use of selection criteria results in limited and necessarily biased information about AEs. It may be both difficult and undesirable to include all AEs in journal articles and other public reports, so standards are needed to determine which AEs to include in reports of clinical trials and how to report AEs completely in other public sources such as trial registers [54].