Clinical significance in pediatric oncology randomized controlled treatment trials: a systematic review

The academic literature was systematically searched using a comprehensive search strategy to identify all RCTs in pediatric oncology (Additional file 1: Appendix A). We searched MEDLINE, EMBASE, and the Cochrane Childhood Cancer Group Specialized Register through CENTRAL from inception to July 2016. Our search strategy was developed by initially using the Canadian Agency for Drugs and Technologies in Health search filter to identify RCTs [22] and subsequently adapted to identify RCTs in pediatric oncology using the Cochrane Childhood Cancer Search Filter, which has been validated by Leclercq et al. [23]. We also assessed the reference lists of the studies that fulfilled our inclusion criteria to identify additional studies. Our search was restricted to studies in English and was inclusive of the published literature.

Studies were deemed eligible if the study adhered to a RCT study design (i.e., did not included a non-randomized component or a historical control), where the study population consisted of pediatric patients diagnosed with cancer and the primary outcome of the trial was a relevant cancer treatment outcome (e.g., a treatment regimen assessing overall survival, event-free survival, etc.), and the trial was a phase III trial that did not stop early due to futility. This did not include studies wherein the primary outcome was treatment complications or side effects, pharmacokinetic trials, toxicity trials, non-clinical interventions, or drug safety profile trials. Only studies that reported a sample size calculation where a delta value was reported or could be calculated for the randomized question were included. Trials that were long-term follow-up studies were excluded and only studies that reported the most recent trial results were reported. Only phase III trials that were not stopped early due to futility were eligible for inclusion. RCTs where the study population consisted of both pediatric and adult patients were deemed eligible if adults were less than or equal to 25 years of age. We chose this age range to reflect norms in pediatric oncology treatment research wherein trials most often include participants up to 21 years of age, though many have also included participants up to age 25 years because of the potential benefit for these slightly older patients. Restricting the age limit to 21 years would result in the exclusion of a number of trials where the majority of participants were aged less than 21 years.

Two investigators (HH and KN, non-independently) screened the title and abstracts based on the specified inclusion criteria. The full text was retrieved and reviewed if the title and abstract was insufficient to determine fulfillment of inclusion criteria. Subsequently, one investigator (HH) conducted a full-text review to assess all of the studies that passed through the first round of title and abstract for inclusion eligibility. The principal investigator (AFH) was available to resolve any discrepancies or disagreements encountered during study selection.

A standardized data extraction template was developed to collect attributes relevant to clinical and statistical significance in addition to general characteristics. The data extraction template was initially piloted on a sample of 15 included studies to ensure that pertinent information was captured and subsequently finalized based on the results of the pilot. Data was collected by one investigator (HH) based on each of the randomized questions within all RCTs, thereby capturing each outcome and the corresponding reported sample size calculation.

Clinical significance was determined based on the guidelines proposed by Man-Son-Hing et al. [10] which considers the relationship between the MCID of the treatment effect and the CI and designated to one of the following four different levels (Fig. 1): (1) Definite – the MCID is smaller than the lower limit of the CI of the treatment effect, (2) Probable – the MCID is greater than the lower limit of the CI of the treatment effect, but smaller than the treatment effect, (3) Possible – the MCID is less than the upper limit of the CI of the treatment effect, but greater than the treatment effect, and (4) Definitely Not – the MCID is greater than the upper limit of the CI of the treatment effect. The CI should be based on the α value specified in the sample size calculation.

We restricted this analysis to randomized questions in superiority trials because these guidelines are intended to be applied to superiority trials. For each study, the delta value was assumed to be the MCID irrespective of whether explicitly stated by the authors. This assumption was applied in an earlier study by Chan et al. [13], and is a pragmatic approach in the context of pediatric oncology. This is based on the premise that the delta value must be reflective of the MCID to the extent that it will result in strong evidence to change standard of care, while also feasible to achieve in a rare disease population. A traditional approach to surveying clinicians and patients to determine a MCID is not realistic in the scope of rare diseases and thus the delta value must follow a definition, which is pragmatic yet clinically relevant and evidence-based. This analysis was restricted to randomized questions to account for studies with multiple primary outcomes. Additionally, only randomized questions were included where the treatment effect and its CI were reported or could be calculated.

The CI of the treatment effect for each randomized question was determined with the methodology outlined by Hackshaw [26] and Altman and Anderson [27] for dichotomous and time-to-event primary outcomes respectively when the CI of the treatment effect was not reported. The CI of the treatment effect for time-to-event outcomes could be calculated only if the CI associated with the experimental and control estimates were reported or a Kaplan-Meier curve with patients at risk was reported. The time point specified in the sample size calculation was used and if it could not be inferred from a Kaplan-Meier curve, or was not reported, the time point reported was used. If the aforementioned was not provided, the CI for the treatment effect could not be calculated and the randomized question was excluded from this analysis. The treatment effect CI was calculated based on the design α value if reported in the sample size calculation and if not reported an alpha of 0.05 was assumed. In the event the primary outcome of the sample size calculation included both an absolute and relative difference, the absolute difference was used. The level of concordance between statistical and clinical significance was assessed through descriptive statistics. Descriptive statistics were calculated to assess the reporting of methodological attributes associated with clinical significance. SAS (Statistical Analysis Software) version 9.4 (SAS Institute, Cary, NC, USA) was used to perform all analyses.

The strengths of this study stem from the inclusion of all pediatric oncology RCTs, from database inception to July 2016, that evaluated a range of cancer treatments for various cancer types. To our knowledge, this is the first study to assess clinical significance in the rare disease context of pediatric cancer RCTs. A limitation is that the search was restricted to studies in English and was inclusive of the published literature, and was thus, prone to language and publication bias. The assessment of clinical significance was based on the delta value in the published report and not the trial protocol and, therefore, it is possible that information recorded as not reported was reported in the trial protocol. However, the parameters used in this review comprise of CONSORT-mandated items and thus should be reported in the published report. A limitation associated with assessing clinical significance as per Man-Son-Hing et al., [10] is that the weight for each level of clinical significance will vary depending on the research question. However, the definitions applied to classify the level of clinical significance are arbitrary and not meant to be adhered to strictly. For example, in the context of a disease with no available effective treatments, a treatment found to be statistically significant but that only shows possible clinical significance would likely warrant implementation in the clinical setting. This is of particular relevance to pediatric cancer, where some cancer subtypes still have dismal survival and high relapse rates. Conversely, demonstrating superiority to a well-established treatment would require adherence to a strict definition of clinical importance because changes to recommended treatment should not proceed unless definite clinical significance is demonstrated. In interpreting the study findings, it is also important to note that only 13 of the 95 randomized questions were from non-inferiority trials, representing a small minority. Thus, it is critical that the assessments of these non-inferiority trials in this research not be over-interpreted or generalized to the superiority trials.

Based on our analysis, the majority (77.9%) of randomized questions did not describe the clinical importance of their findings in relation to the delta value of the interventions in question. The minority (4.2%) of randomized questions explicitly identified the delta value as the MCID with justification. These findings are in line with the limited number of previous studies investigating clinical significance reporting, wherein under-reporting was found [13, 19, 20, 28, 29]. Chan et al. [13] found that, among a random sample of 27 RCTs in major medical journals, 20 articles included sample size calculations, 90% of which reported a delta value but only 11% stated that the delta value was chosen to reflect the MCID of the intervention. Study results were interpreted from the perspective of clinical importance in 20 of 27 (74%) articles, with only one article discussing clinical importance in relation to a reported sample size MCID. In a review of 57 dementia drug RCTs, Molnar et al. [29] found that 46% discussed the clinical significance of their results, and no studies used formally derived MCIDs. These results are in line with our review findings.

In our study, the failure to incorporate a MCID into study design and/or state whether or not the delta value in the sample size calculation was based on the MCID could, in part, be attributed to poor reporting in combination with the difficulty of achieving a reasonable sample size, even when recruiting patients from multiple institutions and over a long duration of time [30‐32]. The formidable challenges of conducting pediatric oncology trials include parent and physician reluctance to involve children in trials, difficulties obtaining consent, permission and assent for study participation, and the dedicated time and attention required to educate children and families, not to mention the remarkably complex logistics of involving multiple sites in multiple countries and the stringent safety monitoring required [33, 34]. Therefore, sample size calculations are perhaps often based on study feasibility and a larger delta value is chosen to reduce the sample size required for the study. This has the potential to place patients at risk of participating in a trial that might lead to erroneous conclusions based on flawed study design and risks the mismanagement of precious time and resources [35]. Often times, studies report that the intervention groups do not differ, when in actuality they lacked sufficient power to support this claim as well as detect a clinically meaningful treatment effect [11, 36‐39]. This is concerning in our review wherein about two thirds of the randomized questions in superiority trials were not statistically significant, yet 25.5% were found to be have probable or possible clinical significance. Arguably, it is unreasonable to expect for RCTs to be purely powered based on the MCID with disregard for feasibility issues; however, trials would be improved if they were powered with consideration of the MCID as well as feasibility, as recommended in the CONSORT Statement [11]. This point is relevant in rare disease trials where a MCID purely reflective of clinician and patient preferences is not realistic. Rather, the MCID in a rare disease context is perhaps best determined by weighing the evidence in the literature and/or pilot studies, clinical expertise, and patient preferences in combination with feasibility. Careful evaluation of the aforementioned would help ensure that evidence-based decisions to change clinical practice are supported.

As Cook et al. [24] state, improved standards in both RCT sample size calculations and reporting of these calculations could assist health care professionals, patients, researchers and funders to judge the strength of the available evidence and ensure responsible use of scarce resources. Without explicit discussion of the treatment effect in relation to the MCID, we allow for subjective interpretation of trial results based solely on whether or not the results were statistically significant. Statistical significance determined by a p value only provides information on whether a significant difference exists and does not provide information on the direction and size of the effect [14]. For instance, if a decision-maker relies solely on statistical significance, they are only able to infer that an experimental treatment is significantly different, or not, from the control, based on the power of the study. However, if a decision-maker relies on clinical significance, which also provides information on statistical significance, they can infer the direction and size of this difference in relation to a MCID (by comparing where the treatment effect and its CI fall in relation to the MCID). The latter approach provides greater utility because a decision-maker can ascertain how harmful or beneficial an experimental treatment is in comparison to the control and assess the value of a trial with greater confidence, whether deemed statistically significant or not.

In addition to designing a study based on a delta value reflective of a MCID, it is necessary to provide justification of the MCID, experimental value and control value, which our study revealed were only reported by a minority of studies. Reporting of this justification will allow the reader to apply the appropriate weight to the authors’ conclusions as values based on systematic reviews or meta-analyses will have a higher weight than values based on the research team’s expertise. It is also essential for the treatment effect to be interpreted with the precision of the CI in mind [10, 14]. For instance, a treatment effect may be within the MCID, but due to inadequate power from an inaccurate sample size calculation, the precision of this estimate may be weak and thus the findings should be graded as low evidence. CIs should be reported for the experimental and control values as well as the treatment effect as recommended by the CONSORT Statement [11].

Given our study results and the implications discussed, in Table 4 we propose recommendations, adapted and informed by Cook et al. [24], Moher et al. [11], and Koynova et al. [25] to promote the incorporation of clinical significance into RCT design and interpretation. Our results also raise the question of whether clinical significance is under-utilized and poorly reported in other rare disease contexts wherein obtaining an adequate study sample challenges feasibility. Moreover, research and knowledge translation efforts are required to raise awareness and understanding of the importance of clinical significance.