EditorialHard-Wired Bias: How Even Double-Blind, Randomized Controlled Trials Can Be Skewed From the Start
Section snippets
Crossover
Crossover from placebo or control to the investigational agent is typically done in pharmacokinetic studies, or for interventions aimed at assessing a subjective end point (permitting intrauser comparisons). However, in modern clinical trials, crossover is increasingly used in studies testing the basic efficacy of a novel compound. For instance, many RCTs of cancer treatments allow patients assigned to placebo to crossover to treatment upon progression of their disease.6 Such crossover affects
Selection Bias
Bias in selecting study subjects is a frequent concern in clinical trials. Selection bias can be caused by the inclusion and exclusion criteria of a study, which prevents the generalizability of results to a broader patient population. For instance, in this issue of the Mayo Clinic Proceedings, Zimmerman and colleagues demonstrate that the inclusion criteria for randomized controlled trials of antidepressants have become increasingly narrow over time.17 The authors compared trials from
The Unequal Playing Field
Another way modern trials have hard-wired bias is by promoting an unequal comparison. For instance, a head-to-head trial of the tyrosine kinase inhibitors axitinib and sorafenib for the treatment of metastatic kidney cancer appears to be fair; however, a closer examination reveals problems.24 Specifically, although the starting dose of both drugs is appropriate, the dose reductions for toxicity favor the axitinib arm. For similar adverse effects, sorafenib has steeper dose reductions, and for
Control Arms
The control for a clinical trial is ideally selected on the basis of the clinical question posed. Controls should reflect the best therapy currently being used in the target population, and for studies evaluating subjective end points, the control should be as close as possible to the investigational arm. If this condition is not met, a trial essentially uses a straw-man comparator. Many times, the use of a sham control has unmasked bias in studies supporting the use of a medical procedure. For
Censoring
Informative censoring28 in clinical trials can distort our perception of the benefits of a treatment. All survival analyses are based on the premise that censoring is uninformative—the patients censored are no different from those who are followed. However, this assumption should be questioned. In many trials for cancer treatment, censoring occurs because patients who cannot tolerate the study medication, or have excess toxicity, withdraw from the study. These patients are likely to be
Conclusion
RCTs remain the best way to draw sound conclusions about the efficacy and impact of drugs, devices, screening, and diagnostic tests, but unfortunately randomization does not ensure a fair trial. In this respect, randomization can be viewed as necessary but not sufficient for sound scientific decision making. The purpose of our analysis is not to disparage the growth of RCTs—which we believe is inevitable and unquestionably valuable—but to highlight persistent challenges. Many types of bias can
Acknowledgments
The views and opinions of the authors do not reflect those of the National Cancer Institute.
References (29)
- et al.
Observational studies often make clinical practice recommendations: an empirical evaluation of authors’ attitudes
J Clin Epidemiol
(2013) - et al.
The misguided ethics of crossover trials
Contemp Clin Trials
(2014) - et al.
Efficacy of everolimus in advanced renal cell carcinoma: a double-blind, randomised, placebo-controlled phase III trial
Lancet
(2008) - et al.
Have treatment studies of depression become even less generalizable? A review of the inclusion and exclusion criteria used in placebo-controlled antidepressant efficacy trials published during the past 20 years
Mayo Clin Proc
(2015) - et al.
Comparative effectiveness of axitinib versus sorafenib in advanced renal cell carcinoma (AXIS): a randomised phase 3 trial
Lancet
(2011) Evidence-based medicine. A new approach to teaching the practice of medicine
JAMA
(1992)Why most published research findings are false
PLoS Med
(2005)- et al.
Reanalyses of randomized clinical trial data
JAMA
(2014) Double-crossed: why crossover in clinical trials may be distorting medical science
J Natl Compr Canc Netw
(2013)
Sipuleucel-T (Provenge®) for castration-resistant prostate cancer
BJU Int
Correcting overall survival for the impact of crossover via a rank-preserving structural failure time (RPSFT) model in the RECORD-1 trial of everolimus in metastatic renal-cell carcinoma
J Biopharm Stat
Cited by (29)
Dose modification rules and availability of growth factor support: A cross-sectional study of head-to-head cancer trials used for US FDA approval from 2009 to 2021
2022, European Journal of CancerCitation Excerpt :Imbalances may occur that threaten the reliability of these studies. These have been called ‘hard-wired biases’, as they often cannot be adjusted for post hoc and can only be acknowledged [2]. Examples include trials that compare new drugs to older, antiquated therapies, which are no longer in favour, a problem increasingly referred to as an inappropriate or straw man control arm [3–5].
Biases in study design, implementation, and data analysis that distort the appraisal of clinical benefit and ESMO-Magnitude of Clinical Benefit Scale (ESMO-MCBS) scoring
2021, ESMO OpenCitation Excerpt :While some degree of attrition may be expected post-progression, the acceptable thresholds should be judged based on previous experiences from real-world studies. Failure to provide optimal post-progression treatment can exaggerate the impact of a PFS gain on OS even when both arms receive the same suboptimal therapies.41,58,60 This underscores the importance of documenting post-progression subsequent treatments until death as part of routine follow-up data.58
A scoping review and survey provides the rationale, perceptions, and preferences for the integration of randomized and nonrandomized studies in evidence syntheses and GRADE assessments
2018, Journal of Clinical EpidemiologyCitation Excerpt :NRS try to deal with confounding using multivariable analyses (standard regression and propensity), and matching. However, many experts believe that, even after optimal adjusted analysis, residual confounding remains a major issue reducing certainty of evidence, and thus consider RS far superior to observational studies in establishing causation [8,9]. Others authorities describe NRS as “real world research”, and that newer NRS analysis approaches employing causal inference techniques, such as instrumental variables, marginal structural models, propensity scores, among others, can to a certain extent address concerns of selection bias and confounding [4,10,11], and that in particular circumstances, NRS may provide higher certainty than RS or help with judging the confidence in RS.
When is crossover desirable in cancer drug trials and when is it problematic?
2018, Annals of OncologyDoes statin use cause memory decline in the elderly?
2016, Trends in Cardiovascular MedicineCitation Excerpt :In contrast, a large proportion of trials that measured memory domains had a high degree of bias, mostly attributable to dropouts. Further, a recent review of intrinsic bias in randomized controlled trials proposed that a study design may be “hard-wired” for biased outcomes, based on specific study design characteristics [40]. In examining the published randomized controlled trials reporting memory outcomes in the elderly, limitations additional to the biases discussed above may include underrepresentation of the elderly, exclusion of participants at risk of cognitive decline, and less rigorous methods of ascertainment of dementia or cognitive impairment, or of measuring cognitive or memory decline.
See also page 1180
Potential Competing Interests: Dr Prasad and Dr Berger are US Federal employees.