A comparison of interventional clinical trials in rare versus non-rare diseases: an analysis of ClinicalTrials.gov

On July 25, 2013 we downloaded the 2012 AACT database from the CTTI website [14]. The database comprised of 133,128 clinical studies registered with ClinicalTrials.gov to September 27, 2012. The key feature of the design, structure and purpose of the AACT database are detailed in Tasneem et al. and the corresponding website [11],[14].

When submitting trials to ClinicalTrials.gov data submitters are requested to provide the diseases and clinical conditions under study as Medical Subject Heading (MeSH) terms. Further, ClinicalTrials.gov uses an algorithm provided by the National library of Medicine (NLM) to annotate additional MeSH terms to a given study [11].

The same condition may exist in ClinicalTrials.gov under various synonyms. To help categorise conditions we downloaded the 2013 MeSH thesaurus and the 2013 supplementary concept records [15],[16]. Each record within these sources contained a MeSH Id, a preferred descriptor name and the names of related concepts which provided synonyms and lexical variations of the preferred descriptor. Table 1 shows an example for Ewing Sarcoma of the information extracted from these sources. In this case ‘Sarcoma, Ewing’ is the preferred MeSH descriptor and the remaining entries are potential variations all mapped to the same MeSH Id.

For the 133,128 clinical studies in the AACT database a total of 211,063 user submitted conditions had been provided. Of these 144,871 (68.6%) could be annotated with a MeSH Id. A further 268,796 MeSH terms had been annotated to these trials by the NLM algorithm; these were also matched to their relevant MeSH Id.

To subgroup studies into rare and non-rare diseases it remained to identify MeSH Id’s related to rare diseases. To identify these we consolidated information from three sources. Firstly, ClinicalTrials.gov identifies trials involving rare diseases within the ‘By Topic’ display option. This listed a total of 1,430 rare diseases based on the US definition. Secondly, we downloaded documentation entitled ‘Rare Disease and Cross References’ from OrphanData.org containing 6,767 diseases classified as rare within the EU [17]. As a final resource we extracted rare conditions from the alphabetical lists provided by the Office of Rare Diseases Research [18]. This provided an additional 6,526 rare disease condition names. Upon removing duplicates a total of 11,959 rare disease names, synonyms and lexical variations remained. Using a combination of lookups, pattern matching and manual search a total of 6,389 (54%) rare disease names, synonyms and lexical variations were matched to their relevant MeSH Id. In total 4,516 unique MeSH Id’s were used and identified as rare. Whilst 54% may seem a low percentage, many of the 11,959 diseases listed are extremely rare and only recently identified. For quality assessment purposes we performed a Targeted Search of ClinicalTrials.gov using each of the 11,959 disease names as conditions. The total number of studies returned from each search was recorded. For example a targeted search using ‘Krabbe disease’ as the condition name returned 19 studies that may relate to this disease. In total 194,226 studies were returned via this search method, of which, 177,333 (91%) where contained within the 6,389 of conditions we identified as having MeSH Id’s.

Final datasets for comparison (Figure 1) were created using the annotated MeSH Id’s following a number of restrictions. Briefly, of the 133,128 studies in the database only interventional studies were eligible for inclusion leaving 108,113 studies. Attention was restricted to trials first registered on or after January 1, 2006 leaving 86,812 studies.

Our goal was to characterise trials on patients with rare diseases thus any trials accepting healthy volunteers were excluded leaving 66,574 studies. Finally, the conditions identified as rare are only rare in specific countries. We excluded any trial with participating centres registered outside the US, EU or Canada leaving 43,986 studies, of these 27,966 trials had a complete complement of MeSH Id’s allowing for categorisation.

A trial was identified as a non-rare disease trial if none of its annotated MeSH Id’s had been identified as rare. A total of 21,329 trials had no rare MeSH Id’s; 6,637 trials had at least 1 rare disease MeSH Id. Trials were finally classified as rare based on the user submitted MeSH Id’s only. We classified a trial as a rare disease trial if all user submitted MeSH Id’s were rare; this left 2,759 rare disease trials.

Prior to analysis certain missing characteristics were inferred. When a trial relates to a single group with the interventional model described as single assignment the trial was designated as non-randomised with open blinding [12].

Descriptive statistics were used to characterise the identified datasets. For categorical data, frequencies and percentages are stated; for continuous data, medians and interquartile ranges (IQR) are stated. We did not have any pre-specified hypotheses in this study and have not presented results from statistical tests. Between group differences in percentages are presented for categorical characteristics. The 95% confidence intervals for the between group difference was calculated using the Wilson procedure without continuity correction. R 2.13 was used for all statistical analysis [19].

This analysis of existing publicly available data relates to clinical trials and their characteristics rather than human participants. Ethical approval was not required.

Basic characteristic of trials in the identified datasets are shown in Table 2. The meaning of characteristics is generally self-evident, although exact definitions for certain characteristics are given in the text. Further definitions of characteristics can be found in the ClinicalTrials.gov draft Protocol Data Element Definitions and e-appendix 1 of Califf et al. [12],[20].

The proportion of trials which are in rare diseases has decreased over time based on the year enrolment to protocol began (Table 2). The overall recruitment status also suggests a higher proportion of rare disease trials are active but not yet recruiting or have terminated early with fewer currently recruiting.

Study duration is defined as the date from which enrolment begins until the final date on which data was, or is anticipated to be, collected. The median study duration, in years, is longer in rare disease trials (median, 3.2[IQR 2.0-4.9] vs. 2.3[1.3, 3.8]).

Participant eligibility characteristics based on gender and age are more inclusive in rare disease trials with a higher proportion of rare disease trials that included both genders (93.6% vs. 86.0%), paediatric patients (20.6% vs. 11.1%) and elderly patients (86.4% vs. 83.2%). It should be noted that this is only enrolment criteria and not data on those who actually participated.

In ClinicalTrials.gov a trial can report either actual or expected numbers recruited but not both. The anticipated number of patients to be recruited in rare trials is less than that of non-rare disease trials (median, 41[IQR 24–74] vs. 76[40–182]). Similarly, the actual number of patients recruited is less for rare disease trials than in non-rare trials (29[12–60] vs. 62[27–175]). For rare disease trials the median actual enrolment is 70.1% (29/41) of the median anticipated enrolment compared to 81.6% (62/76) for non-rare trials and the proportion of trials in rare diseases decreases as enrolment number (anticipated or actual) increases.

The proportion of rare disease trials with an industrial lead sponsor, defined as ‘the organization or person who oversees the clinical study and is responsible for analysing the study data’. ClinicalTrials.gov [20], is slightly greater (34.5% vs. 30.2%) than non-rare disease trials. The majority of trials of either type are undertaken within a single administrative region (US, Canada or the EU). However, a higher proportion of rare disease trials are undertaken across these administrative regions (8.1% vs. 4.4%) compared to non-rare disease trials. Further, exploring individual countries as opposed to these administrative regions, a higher proportion of rare disease trials reported being multi-national (11.5% vs. 6.9%).

Design characteristics of rare diseases trials differ to non-rare diseases trials (Table 3). Centres (labelled as facilities in ClinicalTrials.gov) identify where the protocol is being conducted. A lower proportion of rare disease trials conducted the study protocol at a single centre compared to non-rare trials (61.3% vs. 72.7%).

Rare disease trials are more likely to be early phase trials (those reporting a phase 1 and/or 2 element), (72.5% vs. 38.5%). A lower proportion of rare disease trials employed the N/A phase description (11.3% vs. 30.2%) which is advised to be used for trials that do not involve drug or biologic products although this is not enforced in ClinicalTrials.gov [20]. The primary purpose of a rare disease trial is generally treatment (91.2% vs. 79.3%) with the intervention used tending to be a drug (79.9% vs. 55.8%).

The appointment of a Data monitoring committee (DMC) is more common in rare disease trials (53.2% vs. 40.9%). They are also more likely to have a single group assignment (63.0% vs. 29.6%), with no randomisation (64.5% vs. 28.4%) and open label (78.7% vs. 52.2%). Rare disease trials are also more likely to explore both safety and efficacy endpoints in the same trial (63.2% vs. 45.9%).

Numerous limitations to this study need to be noted. Firstly our categorisation of rare and non-rare trials relied on conditions being submitted as MeSH terms. Whilst this is recommended by ClinicalTrials.gov, it is not enforced. Approximately 30% of studies were instantly excluded because we could not map conditions to MeSH Id’s. Of the 11,959 rare disease terms identified, we could only match 54% to MeSH Id’s. Better application of text matching routines and the use of clinical experts could improve this percentage. A potential consequence is that a number of false negative results may be present in the non-rare disease trial dataset; their correct categorisation could help improve the robustness of this characterisation study. However, taking a random sample of 50 trials from the non-rare dataset we did not find any false-negative results. Thus, we believe that this is unlikely to introduce any bias into the finding. The use of standard vocabulary for conditions submitted to ClinicalTrials.gov could facilitate future research.

Using both US and EU rare disease sources may also limit this study as definitions of rare disease differ in these regions. However, there is likely a large crossover of conditions regardless of regions. Taking a random sample of 50 trials from the rare disease data set found only 4 (8%) trials that were not listed as rare in both the US and EU. These 4 trials listed the conditions of epithelial ovarian cancer, compartment syndrome, photosensitive epilepsy and focal segmental glomerulosclerosis. They are all listed as rare in the ClinicalTrials.gov registry but prevalence for these conditions was not confirmed in the EU orpha.net database [3].

Califf et al. describe a number of limitations to the ClinicalTrials.gov dataset [12]. Firstly, ClinicalTrials.gov does not register all trials. Registries such as ClinicalTrialsRegister.eu can also be used. Confirmation of temporal trends in the ClinicalTrials.gov dataset may need to be corroborated across registries. This is particularly true for specialist datasets; if large research institute register future studies in a different database this could have large implications on the identification of trends.

Secondly, data elements can be missing or unavailable due to changes in the ClinicalTrials.gov registry definitions, statutory requirements or simply not entered by data submitters. Further to missing data elements, data provided can also be inconsistent or inaccurate. For example a number of trials reported with single arm assignment had also reported a crossover, factorial or parallel intervention model. Furthermore, we did not explore the details behind the labels provided for each trial and we recognise that there may be inaccuracies in definitions such as ‘randomized’ or ‘blinded’. Causes for this may be the expertise of data submitters or a lack of suitable constraints on the data being submitted to ClinicalTrials.gov. Missing data, inconsistencies, and inaccuracies can impact the quality of this, and any other, characterisations of trials. However, we believe that these issues are likely to introduce random error rather than bias since the issues are likely to be distributed similarly across rare and non-rare disease trials.

It is also worth pointing out the strengths of this analysis in that we have provided one of the largest and most complete comparative surveys of rare disease clinical trials to date.

This study contrasts rare and non-rare diseases clinical trials across a broader range of characteristics than previously seen in the literature. Trials in rare and non-rare conditions differ in methodological approach including quantifiable differences in the reported use of randomisation, blinding and data monitoring committees. Rare disease trials are smaller, longer and more frequently terminated than those in non-rare conditions.

Data used for this study are available from the following website http://​www.​ctti-clinicaltrials.​org/​what-we-do/​analysis-dissemination/​state-clinical-trials/​aact-database or can be obtained by contacting the corresponding author.