Background
The low prevalence of rare diseases, the phenotype heterogeneity and the long latency period, may prevent and/or make the possibility of performing randomized clinical trials (RCTs) and large studies extremely difficult [
1,
2]. Therefore, with these diseases, knowledge of treatment efficacy or any other type of clinical knowledge must be based only on observational studies, rare disease registries and case reports [
2], where real world data and evidence play an important role in health care decisions [
3]. However, RCTs are assessed, in evidence-based medicine as the best corroboration of the efficacy of new treatments, while case reports show a lower grade level of evidence [
1].
Previous FDA drug approvals with breakthrough status suggest that sometimes non-controlled studies can provide the same quality of evidence to demonstrate a positive risk–benefit ratio as individual RCTs [
2,
4]. Accordingly, randomized phase II controlled-trials were not superior to single-arm phase II trials in predicting phase III study success [
5].
In rare diseases, research based on registries and case studies is likely the best option, due to lack of patients, and case reports are often the primary evidence of the effectiveness of a new therapy or treatment [
6]. Due to prior considerations, an increasing interest in case report analyses, and combining their results in systematic reviews, exists [
7,
8].
Case report databases are developed as the Consensus-based Clinical Case Reporting (CARE) guidelines [
9] attempt to homogenize and upgrade the quality of the information published in case reports; however, there are still questions about how to aggregate them in ways that would be most meaningful [
8].
A previous systematic review of clinical studies, evaluating the effectiveness of enzyme replacement therapy (ERT) in adults (≥ 18 years) with mucopolysaccharidosis Type I (MPS-I), rated the strength of evidence (SOE) for ERT on each outcome with the Grading of Recommendations Assessment, Development and Evaluation (GRADE) criteria [
10]. Another study showed a good rate of agreement between SOE and specific outcomes in a case report meta-analysis and clinical study meta-analyses [
11]. This agreement has not been confirmed in other patient populations. Furthermore, the mentioned case report meta-analysis and clinical study meta-analysis were developed by the same research group.
Our proposal performed a meta-analysis of case reports of MPS-II patients treated with ERT, and compared the degree of evidence assigned to each outcome, vs. what was assigned in a previous clinical study meta-analysis, published by an independent research group. In a population suffering from MPS-II, we sought to confirm the impressive rate of agreement seen between case reports and clinical study meta-analyses in patients with MPS-I [
11].
Discussion
Personalized medicine based on molecular diagnosis has fragmented complex diseases, such as cancer, into multiple molecular subtypes, each one representing a rare disease [
18,
19]. This has extended the classification of rare disease to other illnesses that were not previously considered as such. Thus, the recent importance of research methods derived from rare diseases [
20], the development and improvement of rare disease registries [
21] and the recovered interest in case reports for aggregating results in systematic reviews [
7].
There are few publications aggregating case report results in a quantitative manner [
22]. There is only one study comparing the results of a case report meta-analysis and a meta-analysis including RCTs [
11]; it showed that both meta-analyses reach similar conclusions in adult MPS-I.
We selected the Bradley meta-analysis [
12] as the gold standard, because it analyzed the overall MPS-II population without an age restriction. Furthermore, it had been recently published and developed by an independent research group. Along with the Bradley study, the efficacy and safety of ERT in patients with MPS-II had been analyzed in three previous meta-analyses: that by da Silva et al. in 2016 [
23], which only selected one phase II/III trial [
24], that by Alegra et al. in 2013 [
25], which combined 2 RCTs [
24,
26] and 1 open label study with the same patients of all ages [
27], 1 open-label study of adults [
28], and 1 cohort study of children [
29], plus the one by Pérez-López et al. in 2018, which analyzed adult MPS-II patients (> 16 years) [
30].
In agreement with our previous study [
11], we proposed a single method to aggregate results from different case reports. We considered the number of cases, showing a certain characteristic among the total number of cases analyzed. This method has the flexibility of combining outcomes independent of the measurement of the variable; it also allows different ways for controlling the multiplicity, depending on the relationship among the outcomes analyzed [
17]. Additionally, we could consider the information provided from aggregated results of case reports as a single observational study and combine the results with case series, clinical trials and rare disease registries in a meta-analysis; or we could simply add the cases of all studies, as if it was a single study [
22]. This allows incorporating all available evidence (single case observations, clinical studies and rare disease registries) to evaluate a particular research question. As an example, previous studies in infectious disease have used this strategy to develop classification tree models to predict disease outcomes [
22].
However, the publication bias and heterogeneity of the included studies represents two critical aspects that were not considered in previous case report aggregations [
13,
31]. Regarding publication bias, funnel plot tests cannot, be implemented when aggregating case reports [
13]. Previous publications criticize the use of the safe-false N in meta-analyses of clinical studies [
32]. This index evaluates if a significant result of a meta-analysis can become significant without considering whether the differences evaluated are clinically meaningful. This limitation can be avoided in a case report aggregation by testing if the percentage of responders is higher than the that of responders in historical controls (clinically meaningful difference). In accordance, previous clinical trials have demonstrated the utility of rare disease registries as historical controls [
21].
We have proposed to analyze heterogeneity based on different sensitivity analysis to evaluate the robustness of the meta-analysis results. We would consider that an equivalent approach may be easily developed in further publications [
33]. Accordingly, we have proven the robustness of our results through a specific strategy: considering all selected case reports, excluding those published after the clinical study meta-analysis bibliographic search, and excluding studies analyzed in clinical study meta-analyses, including excluding congress communications. In all scenarios, our results show good agreement with the SOE score of clinical study meta-analyses.
Some authors have underlined the utility of N-of-1 trials to compare the effect of different treatments in only one patient [
34]. These designs can randomize repeated cycles of treatment challenges (e.g., A-B-A-B) in a single participant, in which A is the test drug and B is the comparison drug. These studies achieve the usual methodological safeguards of classical clinical trials (controlled, randomized, and blinded). However, these designs are not applicable in situations where the disease is not clinically stable or the carry-over effects of treatment cannot be avoided [
35]. Therefore, in some diseases, most of the available evidence comes from case reports [
2]. Methods to aggregate results of different N-of-1 trials in a meta-analysis assumed randomized allocation of treatment exposure about study periods [
36,
37]. They cannot be applied to aggregate results of case report narratives or rare diseases registries.
As we have mentioned, rare disease registries may be valuable sources of information not only on disease course but also on treatment outcomes. A global registry, Hunter Outcome Survey (HOS), has been collecting information on patients with MPS II for over 10 years [
38]. Our results seem to agree with those from the registry. Based on data from the HOS registry [
39], ERT with idursulfase has a positive effect on uGAGs, and liver volume, 2 outcomes showed as of acceptable evidence in our analysis by the strong method, and also on 6MWT, which was also categorized in our analysis as acceptable evidence by the weak method. In addition, data from the HOS registry showed that 59% of patients younger than 12 years and 67% of those 12 years or older were positive for antibodies by week 13 of treatment [
40], in agreement with antibody development being classified as acceptable evidence by the strong method in our analysis.
Regarding IRRs, which our analysis categorized as unacceptable evidence, we only considered as relevant the IRRs that caused ERT dose changes. Data from the HOS registry showed that although 32% of patients suffer from IRRs, most patients (85%) experience them during the first 3 months of treatment and most IRRs are mild or moderate in severity and can be managed without interrupting treatment [
38].
An important point to consider is the futility boundary selected in our analysis (null hypothesis). The objective of Bradley et al. [
12] and their meta-analyses was to identify benefits and harms of ERT, with the study defined as a pilot. There was not a criterion for clinical meaningfully effect. Therefore, the objective of the meta-analysis was similar to phase II designs, in which it is intended to explore the benefits and harms of a specific treatment. In accordance with previous publications evaluating treatment activity in phase II trials [
41], a percentage of patients equal or lower than 5% showing a response has been considered as the null hypothesis. Accordingly, we observed that the best agreement with clinical study meta-analysis results were observed with the preplanned limit of no effect of 5%, which agrees with previous recommendations in designs with the same purpose [
41]. Case report meta-analyses with other purposes, e.g., identifying effects higher than an active comparator, may require the null hypothesis to be based on historical control estimations.
We have demonstrated that standardization and a good definition of outcomes evaluated in case reports are strongly related with the validity of the results obtained based on their aggregation [
8]. Thus, excluding results from poorly defined outcomes is a useful criterion to control the quality of single cases in a case report meta-analysis, as required in any meta-analysis [
13].
Different authors have underlined the impact of clinical report results in clinical practice and research [
1], while clinical cases have traditionally been of great importance in determining patient treatment in the context of rare diseases [
7]. Our study suggests that the combination of these single cases can lead to robust results. Previous experience in a personalized medical context suggests that understanding the biologic mechanism of disease is more critical for treatment success in pivotal studies than a simple demonstration of superiority in a randomized-controlled study [
39,
42].
Clinical reports have a high risk of publication bias [
6] and it is expected that only positive results will be published. As a conservative assumption, we considered all outcomes not reported in a case study to not have improved. However, this assumption cannot prevent bias related to unpublished cases. Based on this issue, an alternative explanation of study results is that case reports confirming clinical study conclusions have a higher probability of being accepted and published in a journal. However, this does not explain that the level of agreement of case reports and clinical study meta-analyses results was higher, selecting only standardized and well-defined outcomes.
Another important limitation is that we cannot estimate the effect size of an outcome. Nevertheless, we observed that most clinical cases do not report enough information to aggregate study results in a mean, median, or a proportion with a confidence interval. This highlights the importance of initiatives to homogenize and upgrade the quality of the information published in case reports [
9]. Additionally, we did not analyze the effect of ERT, taking into account the different treatment doses used in case reports, since we intend to compare our results with those of Bradley et al. [
12], who they did not report this subgroup analysis. Either way, most cases evaluated in both studies were treated with a standard dose (0.5 mg/kg/weekly).
This analysis was confirmed in a MPS-II population treated with ERT, with results explored in a specific group of MPS-I patients. New studies must assess if results can be generalized for other diseases and patient profiles.
A meta-analysis of clinical reports cannot replace evidence provided by clinical trials. Subject recruitment in rare diseases and personalized medicine represent a critical task in clinical research [
2,
43,
44]. In a therapeutic context, in which most studies become clinical reports, excluding them from systematic review increases the risk of bias and reduces efficiency, as all available evidence is not considered [
45]. There is evidence that case reports translate useful data collection in cases of rare phenomena, and contribute to the progress and dissemination of novel scientific discoveries three or more years earlier than clinical studies [
11]. In this period, daily clinical practice or the design of confirmatory clinical trials require evidence from published clinical reports [
8].
Competing interests
We wish to draw the attention of the Editor to the following facts which may be considered as potential conflicts of interest and to significant financial contributions to this works:
Miguel Sampayo-Cordero has received consulting and advisor fees from: Nestle Health Science, Laboratorios Leti, Roche and Allergan. In addition, Miguel Sampayo has received research funding fees from Nestle Health Science.
Bernat Miguel-Huguet declares no conflict of interest.
Almudena Pardo Mateos has been contracted in the past by Shire and by Genzyme.
Andrea Malfettone declares no conflict of interest.
José Pérez-García has received consulting and advisor fees from: Roche and Eli Lilly.
Antonio Llombart-Cussac has received consulting and advisor fees from Roche, GlaxoSmithKline, Novartis, Celgene, Eisai, and AstraZeneca and has stock options, patents and intellectual property from MedSIR.
Javier Cortés has received consulting and advisor fees from: Roche, Celgene, Cellestia, AstraZeneca, Biothera Pharmaceutical, Merus, Seattle Genetics, Daiichi Sankyo and Erytech. In addition, Javier Cortés has received honorarias from: Roche, Novartis, Celgene, Eisai, Pfizer and Samsung. Add more, Javier Cortés has received research funding fees to the institution from Roche. Finally, Javier Cortés has stock options, patents and intellectual property from MedSIR.
Marc Moltó-Abad has received research support from Shire; has received fees from Sanofi/Genzyme, Shire and Alexion for participation in their respective registries. In addition, this work used laronidase, a product manufactured by Shire, to evaluate the efficacy of ERT in MPS II patients.
Cecilia Muñoz-Delgado as received consulting fees, fees as a speaker, and research support from Shire and Genzyme.
Marta Pérez-Quintana as received consulting fees, fees as a speaker, and research support from Shire.
Jordi Pérez-López has received consulting fees, fees as a speaker, and research support from Shire.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.