Clinical trial sample selection
We reviewed original research articles, including brief reports and communications but not research letters or correspondences, to identify the 50 most recent primary publications of clinical trials in each journal, beginning with articles published in print journal issues in December 2015 and continuing in reverse chronology as far back as January 2010. We used the table of contents of each print journal issue to identify articles for possible inclusion. Clinical trials were systematically identified by screening the article’s abstract and, if necessary, the Methods section, for statements meeting the World Health Organization’s (WHO) definition of a clinical trial, also used by ICMJE, namely any study that “prospectively assigns people or a group of people to an intervention, with or without concurrent comparison or control groups, to study the cause-and-effect relationship between a health-related intervention and a health outcome” [
25].
We limited our sample to publications reporting the findings of a trial’s primary outcome(s), which is most pertinent to the information contained within its registration, and excluded trials reporting secondary analyses of previously published results, secondary outcomes only, or interim analyses of primary outcomes. We further excluded publications describing phase I trials, as these studies typically do not assess effectiveness and have minimal impact, if any, on clinical practice. We additionally excluded trials beginning prior to July 2005, since trials preceding the ICMJE policy were unlikely to be prospectively registered.
Data collection
From trial publications, we extracted information on journal, intervention, allocation, manuscript submission date, enrollment start date, primary outcome(s) with associated findings, and registration number(s) corresponding to the trial(s) reported. To account for the possibility of duplicate registrations, we searched the WHO International Clinical Trials Registry Platform (ICTRP), which aggregates registrations across registries endorsed by WHO, and in turn endorsed by ICMJE, using the reported registration numbers and additionally reviewed registrations for alternate identifiers to determine the earliest registration for each trial. For publications not reporting registration information, we searched the WHO ICTRP platform using search terms pertaining to intervention, first author, senior author, and sponsor to identify unpublished registrations, cross-referencing potential matches against sample size and enrollment criteria. We contacted corresponding authors of unmatched trials for registration information before concluding that the published trial was unregistered.
Using the earliest registration for each trial, we collected registration date, primary outcome submission date, primary completion date, start date, primary outcome(s) at initial registration, enrollment, phase, location, and funding source. We supplemented information on the latter four elements from trial publications when missing from the registry. Among trials we determined to have been first registered at
ClinicalTrials.gov, we additionally collected date of original primary outcome submission, which the registry uniquely lists separately from the trial registration date. We categorized intervention, funding source, location, enrollment, and allocation as outlined in Table
1 for use in pre-specified stratified analyses. We considered interventions involving drugs, devices, or biologicals as regulated by the Food and Drug Administration (FDA).
Table 1
Characteristics of clinical trials published in the 10 highest-impact US medical specialty society journals between 1 January 2010 and 31 December 2015 (N = 486)
Interventiona |
Drug | 287 | 59.1 |
Device | 46 | 9.5 |
Vaccine or biological | 86 | 17.7 |
Other | 102 | 21.0 |
Phaseb |
Phase II | 190 | 39.1 |
Phase III | 110 | 22.6 |
Phase IV | 46 | 9.5 |
Not listed | 153 | 31.5 |
Randomization |
Yes | 372 | 76.5 |
No | 114 | 23.5 |
Fundingc, d |
Industry | 216 | 44.4 |
Non-industry | 270 | 55.6 |
Enrollment |
≥ 100 | 280 | 57.6 |
< 100 | 206 | 42.4 |
Location(s) |
US only | 166 | 34.2 |
US and international | 84 | 17.3 |
International only | 236 | 48.6 |
Trial registrye |
ClinicalTrials.gov | 383 | 87.2 |
EU-CTR | 76 | 17.3 |
ISRCTN | 24 | 5.5 |
Other registriesf | 47 | 10.7 |
Data abstractions were performed in tandem by ADG and JAA. Consistency and accuracy were verified through a 10% random sample validation of each investigator’s collections. A third author (JDW) repeated all searches for trials that were determined to be unregistered, supplementing with additional searches of the National Institutes of Health (NIH) funding database using grant funding identifiers when listed in publications [
26]. All disagreements were resolved by consensus with input from the senior investigator (JSR).
Main outcome measures
We first determined whether each trial was registered by ensuring that a corresponding registration record could be located. For registered trials, we next ascertained timeliness of registration by determining whether the trial was registered within 30 days of its enrollment start date. Trials registered greater than 30 days after enrollment initiation were categorized as “retrospective.” Although ICMJE policy mandates registration at or before the time of first patient enrollment, we allowed a 30-day grace period between registration and enrollment initiation in order to account for potential flexibility on the part of journal editors with regard to registration timeliness. Further, because the FDA Amendments Act (FDAAA) of 2007 specifically mandates registration within 21 days of first patient enrollment as a legal mandate for studies of therapeutics and devices regulated by FDA, a 30-day window grace period was chosen to accommodate inconsistencies between legal and journal requirements. Month-based representations of dates were recorded as the last day of the corresponding month (i.e., September 2012 was transcribed as 30 September 2012) to conservatively classify registrations as “retrospective.” We elected to use enrollment start dates reported in registries as opposed to those reported in publications in our determinations of registration timeliness, as we believed that enrollment start date may not be consistently reported in publications, while, in registries, it is a mandatory registration element and, hence, less easily excluded or otherwise misrepresented.
Among trials registered retrospectively, we established whether registration might have occurred after initial ascertainment of the primary outcome, and hence potentially permitted unaccounted protocol modifications after interim analyses, by comparing the trial’s registration date against the date on which the primary outcome would have been collected for the trial’s first enrolled participant(s). For example, a trial with a primary outcome assessing serum creatinine levels at 6 weeks that registered in November 2012 but that began enrolling patients in February 2012 would have been retrospectively registered after observation of the primary outcome among participants enrolled at the trial’s initiation. In this case, investigators could have in theory observed primary outcome data for participants that enrolled at trial initiation and modified the trial based on data collected from the trial’s first participants. Without information on trial-specific recruitment, however, we cannot comment on how robust or useful this initial primary outcome data may have been in an interim analysis informing post hoc modifications, though we can comment on the theoretical potential for an interim analysis to have occurred in this scenario. In instances where multiple time frames were designated as primary, we based our calculations on the shortest primary outcome time frame specified in the registry. If no time frame was listed in association with the registered primary outcome, we used the time frame described in the trial’s publication. We noted cases where the nature of the primary outcome (e.g., median survival) did not permit this determination.
We next compared primary outcomes at initial registration against those specified in publications, excluding any primary outcomes pertaining specifically to safety or tolerability. We classified primary outcomes as discordant or non-discordant. Primary outcome comparisons were performed in a systematic manner across three sequential domains: number of primary outcome(s), definition(s) of primary outcomes, and specified time frame(s) for primary outcome(s) ascertainment. Specifically, we began by first comparing the number of primary outcomes in the initial registration record against the number of outcomes explicitly stated as primary in the publication. If the number of primary outcomes differed, the primary outcomes were classified as discordant. If the number of primary outcomes was the same, we compared the definition of the primary outcome (e.g., all-cause mortality vs. recurrent VTE). If the primary outcomes differed in how they were defined, they were classified as discordant. Because primary outcomes may be registered in varying levels of detail, it may not be possible to know with certainty whether registered and published primary outcomes are truly concordant. Accordingly, we classified registered and published primary outcome definitions as discordant on the basis of explicit known differences in what was specified, rather than focusing on differences in level of specification. For example, if a trial registration specified a primary outcome definition of “anxiety level” and the publication reported a primary outcome definition of “Hamilton Anxiety Rating Scale,” we classified these outcome definitions as non-discordant rather than discordant. In not doing so, we sought to avoid classifying primary outcomes as discordant based simply on differences in the level of detail provided between registrations and published reports. In cases where the number of registered and published primary outcomes was the same but greater than one, registered and published primary outcome definitions were paired and compared on the basis of clinical judgment with input from the senior investigator. If primary outcome definitions did not differ, we next compared the time frame, if available, at which the primary endpoints were ascertained. If the explicitly stated time frames differed (i.e., 3-month mortality vs. 30-day mortality), these outcomes were classified as discordant. We noted cases where registered outcomes were too poorly specified (e.g., vague study of “efficacy of intervention”) to permit comparison. Trials without a designated primary outcome specified in the publication were excluded from outcome comparison analyses.
Finally, we categorized each trial on the basis of its primary outcome findings whenever formal hypothesis testing had been conducted or inferences could be made regarding the statistical significance of reported findings (i.e., inferential studies). By examining trial publications, we determined whether the trial’s findings indicated, based on the reported primary outcome(s), that a study intervention was statistically significantly better (i.e., positive), statistically significantly worse (i.e., negative), or not statistically significantly different (i.e., not significant) than a designated comparison (placebo group, active group, or predefined threshold) and classified the overall trial accordingly. For trials that assessed a non-inferiority hypothesis, we considered establishment of non-inferiority to represent a “positive” result and failure to establish non-inferiority a “not significant” result. In instances where more than one primary outcome was reported, we categorized trials with at least one significant primary outcome as “positive” or “negative” on the basis of the statistically significant outcome; trials with mixed findings (some positive and some negative primary outcomes) were classified by prioritizing the results of clinical outcomes over surrogate markers. Trials with mixed all clinical or all surrogate primary outcome results were arbitrated based on the relative importance of the significant outcomes in question. For trials that did not specifically designate a primary outcome, any outcomes reported in the trial’s abstract were considered primary and the overall study was categorized using the scheme described previously. Trials that presented analyses in a solely descriptive manner or that lacked a designated comparison against which to judge the statistical significance of reported results were noted as “non-inferential” and excluded from analyses of association. Trials categorized as “positive” were judged to report overall favorable findings, whereas those categorized as “negative” or “not significant” were judged to report overall unfavorable findings.
Statistical analysis
We used descriptive statistics to characterize the proportion of trials registered, the proportion registered retrospectively, as well as the proportion registered after initial primary outcome ascertainment, overall, and stratified by specialty journal and trial characteristics. We additionally determined the proportion of trials with non-discordant registered and published primary outcome measures and the proportion reporting favorable findings, overall, and stratified by journal, registration timeliness, intervention, funding source, location, allocation, and enrollment. We used chi-squared testing to assess differences in registration and registration timeliness by journal and by each of the aforementioned trial characteristics. We also used chi-squared testing to assess differences in primary outcome concordance and study findings by journal, trial characteristics, and timeliness of registration. All statistical hypothesis testing was performed using chi-squared analyses with a two-sided type I error level of 0.006 to account for multiple comparisons. Accordingly, all reported p values correspond to chi-squared testing. We additionally report relative risks (RR) with 95% confidence intervals (CI) for all chi-squared comparisons involving two binary variables. Statistical analyses were performed using JMP Pro Version 11.2.0 (SAS Institute, Inc.).