Benefits and harms of the human papillomavirus (HPV) vaccines: comparison of trial data from clinical study reports with corresponding trial register entries and journal publications

Since 1995, the pharmaceutical industry has written structured clinical study reports of randomized clinical trials following international guidelines to document their products’ benefits and harms when applying for marketing approval [1]. Clinical study reports are usually confidential documents, but can be requested or downloaded from the European Medicines Agency (EMA) [2], ClinicalStudyDat​aRequest.​com (CSDR), GlaxoSmithKline’s trial register website and, in the future, possibly from the US Food and Drug Administration (FDA) [3]. Publicly available trial data mainly come from biomedical journal publications and trial register entries such as those on ClinicalTrials.​gov. The intention of ClinicalTrials.​gov is that all studies publish all results and that those who do not publish results within 12 months of trial completion are fined. According to fdaaa.​trialstracker.​net, 32% of studies on ClinicalTrials.​gov have no results posted and no fines have been issued. Clinical study reports usually have all prespecified data included or include amendments. There can be important differences in results from published [4] and unpublished [5] versions of corresponding study documents. Clinical study reports include highly detailed information on all aspects of a trial [6] and are on average about 2000 pages long [7], but it can be difficult to obtain complete and unredacted clinical study reports [8].

We carried out a systematic review of the human papillomavirus (HPV) vaccines’ clinical study reports [9] based on an index we constructed of 206 HPV vaccine studies [10]. As of July 2017, 62% (92/149) of the completed studies were not published in journal publications and 48% (71/147) of the completed studies on ClinicalTrials.​gov had no study results posted [10]. Systematic reviewers often only use journal publications and trial registers for their reviews, which may increase the risk of using a data set influenced by selective outcome reporting.

To our knowledge, no study has looked at differences of pooled estimates—such as meta-analyses—of corresponding study documents of the same intervention. Our primary aim in this study was to compare meta-analyses of HPV vaccine data from clinical study reports with data from corresponding trial register entries and journal publications. Our secondary aim was to compare the reporting of study design aspects of the corresponding study documents.

We compared corresponding HPV vaccine study documents of clinical study reports, trial register entries and journal publications to investigate the degree of reporting bias for prespecified outcomes and the reporting of trial design aspects; see our protocol on PROSPERO [11] (registered as ‘Protocol amendment no. 3’ for our systematic review of the HPV vaccines [9]).

Clinical study reports were obtained from EMA and GlaxoSmithKline [9]. We identified the clinical study reports’ corresponding trial register entries on ClinicalTrials.​gov and corresponding primary journal publications from our published index of the HPV vaccine studies. The search strings used to identify the studies are available in the index publication [10]. We assessed all identified journal publications for a study (including supplementary documents and errata) for eligible information and chose the primary publication that corresponded to the clinical study report for our comparison. We did not check for eligible information in additional trial registers (such as the EU Clinical Trials Register) or letters to the editors.

Data extraction and comparison of the study documents were carried out by two researchers (LJ extracted the data; TJ checked the extractions; and PCG arbitrated). For each study document, the following data were compared: study ID, number of pages, date of document, time from study completion to publication in a journal, result availability, protocol availability (including pre-specification of outcomes and inclusion of a statistical analysis plan), reporting of PICO criteria (participants, interventions, comparisons and outcomes) and reporting of six major design-related biases defined by the Cochrane Handbook (version 2011) for the Cochrane risk of bias tool [12] (random sequence generation, allocation concealment, blinding of outcome assessors, blinding of personnel, blinding of participants and loss to follow-up). We collected these data, as they are important to evaluate a study’s internal and external validity. We did not include the Cochrane risk of bias tool domain ‘selective outcome reporting’, since we compared this domain quantitatively between corresponding documents.

For each study document, we extracted and compared data on the outcomes we assessed in our systematic review [9]. As our review contained 166 meta-analyses, we only compared the 20 most clinically relevant outcomes (or statistically significant outcomes with a p value ≤ 0.05; noted in parentheses). Benefit outcomes included all-cause mortality, HPV-related cancer mortality, HPV-related cancer incidence, HPV-related carcinoma in situ, HPV-related moderate intraepithelial neoplasia, HPV-related moderate intraepithelial neoplasia or worse and HPV-related treatment procedures. Harm outcomes include fatal harms, serious harms (including those judged as ‘definitely associated’ with postural orthostatic tachycardia syndrome [POTS] and complex regional pain syndrome [CRPS; see our systematic review protocol amendment [13] for these two post hoc exploratory analyses] and the nervous system disorders that were Medical Dictionary for Regulatory Activities [MedDRA] classified in this system organ class), new-onset diseases (including back pain, vaginal infection and the vascular disorders that were MedDRA classified in this system organ class) and general harms (including fatigue, headache and myalgia). Histological outcomes were assessed irrespective of involved HPV types. The most aggregated data account (participants with events over the total number of participants) was used for the meta-analyses, and the most detailed harm account of MedDRA preferred terms was used for event comparisons. For example, if harms were registered separately per harm, we would count the separate harms and summarize them as a total number of harms. For all GlaxoSmithKline clinical study reports and for serious harms for Merck clinical study reports, we pooled MedDRA preferred terms in their respective system organ classes. A participant could potentially be included more than once in a separate analysis (e.g. if a participant experienced both serious ‘headache’ and serious ‘dizziness’, the participant would be counted twice in the MedDRA system organ class analysis of serious nervous system disorders); we therefore consider the MedDRA system organ class analyses exploratory.

Merck Sharp & Dohme did not provide a formal definition for its new-onset disease category—new medical history—but described the category as ‘all new reported diagnoses’ in the clinical study report of trial V501-019. Although ‘new medical history’ was not explicitly mentioned in the trial register entries and journal publications, we included eligible new reported diagnoses not reported as serious or general harms in this category.

For our meta-analyses, we used the intention to treat principle. Risk ratios (RRs) were calculated with the random effects inverse variance method. Random effects estimates were compared to fixed effect estimates, as the former method may weigh small trials unduly if there is considerable heterogeneity between trials [12].

We included study documents from 22 randomized clinical trials and 2 follow-up studies and obtained 24 clinical study reports, 24 corresponding trial register entries and 23 corresponding primary journal publications (for the remaining journal publication—HPV-003, of 61 participants—the manufacturer confirmed that no journal publication had been published [10]). See Additional file 1 for our study’s PRISMA statement.

There were on average 50 and 121 times more pages in the clinical study reports than in their corresponding trial register entries and journal publications. This was likely a main reason why the clinical study reports were superior at reporting trial design aspects. If our systematic review of clinical study reports [9] had relied on trial register entries or journal publications, it would have had no data for a quarter of our prespecified outcomes (11/40). Although the inclusion of clinical study reports led to significantly more eligible and available data, no changes in the direction of available results occurred when comparing the risk ratios of corresponding meta-analyses as ratios of relative risks. This may have several explanations. First, GlaxoSmithKline might be more transparent than other pharmaceutical companies [36], so corresponding study documents from GlaxoSmithKline could be more consistent compared to corresponding study documents from other companies [37‐40]. Second, we used the random effects model, but more risk ratios had narrower confidence intervals with a fixed effect model. Third, there were low event numbers for several outcomes; differences in low event numbers may be overestimated when using risk ratios [12]. Finally, the studies were designed with a lack of placebo controls and incomplete reporting of harms [8] and the trial register entries and journal publications only included very few of the assessed data points (from 3% to 44%) compared to the clinical study reports. This may have skewed some of our comparison results towards being false-negative and led to an underestimation of harms caused by the HPV vaccines. Major study design features such as the use of active comparators and the reporting format of harms are not affected by the number of pages in a study document, but the vast increase in the amount of detail in clinical study reports allows for a more complete understanding that might impact conclusions. We have expanded on the issues of the lack of placebo controls and incomplete harms reporting elsewhere [8].

There were no significant differences in the meta-analysis estimates of the assessed outcomes from corresponding study documents. The clinical study reports were the superior study documents in terms of the quantity and the quality of the data they contained and should be used as primary data sources in systematic reviews; trial register entries and journal publications should be used concomitantly with clinical study reports, as some data may only be available in trial register entries or journal publications. A systematic review of the HPV vaccines would have had considerably less information and data included if it relied on trial register entries and journal publications instead of clinical study reports. A full data set would be expected to be available from case report forms and individual participant data, but there are regulatory barriers that need to be lifted before independent researchers can access such data [8]. Corresponding study documents ought to use consistent terminology and provide all aggregate and individual benefits and harms data. To test our results’ generalizability, we recommend that other researchers replicate and expand on our method of comparison for other interventions.