Editorial

Hard-Wired Bias: How Even Double-Blind, Randomized Controlled Trials Can Be Skewed From the Start

Different drug modification rules or growth factor support guidance may affect the results in oncology randomised controlled trials. We aimed to estimate the prevalence of unequal rules for dose modification rules or the use of myeloid growth factors in head-to-head registration Food and Drug Administration trials.

This cross-sectional analysis included all head-to-head registration randomised controlled trials leading to a US Food and Drug Administration approval between 2009 and 2021. Trials examined anti-cancer drugs in the advanced or metastatic setting where a comparison could be made between arms regarding either dose modification rules or myeloid growth factors recommendations. Sixty-two registration trials met inclusion criteria. Information abstracted for each trial included tumour type, setting, phase, and type of sponsor. We assessed, according to pre-specified rules, imbalance in drug modification rules, myeloid growth factors recommendations or both.

We find 40 of 62 (65%) selected trials have unequal rules for dose medication, granulocyte colony-stimulating factor (G-CSF) use or both. Six trials (10%) had rules favouring the control arm, while 55% of selected trials (34/62) favoured the experimental arm. Among these, 50% (17/34) had unequal drug modification rules, 41% (14/34) had unequal G-CSF rules and 9% contained both (3/34).

We find that 55% of trials testing anti-cancer drugs against each other used protocol rules that favoured the experimental arm. This leaves open the question of whether new molecules are truly superior to older molecules or if instead different outcomes are due to more aggressive dosing or growth factor support. Trials should utilise equal rules for dose medication and G-CSF support.

The European Society for Medical Oncology-Magnitude of Clinical Benefit Scale (ESMO-MCBS) is a validated, widely used tool developed to score the clinical benefit from cancer medicines reported in clinical trials. ESMO-MCBS scores assume valid research methodologies and quality trial implementation. Studies incorporating flawed design, implementation, or data analysis may generate outcomes that exaggerate true benefit and are not generalisable. Failure to either indicate or penalise studies with bias undermines the intention and diminishes the integrity of ESMO-MCBS scores. This review aimed to evaluate the adequacy of the ESMO-MCBS to address bias generated by flawed design, implementation, or data analysis and identify shortcomings in need of amendment.

As part of a refinement of the ESMO-MCBS, we reviewed trial design, implementation, and data analysis issues that could bias the results. For each issue of concern, we reviewed the ESMO-MCBS v1.1 approach against standards derived from Helsinki guidelines for ethical human research and guidelines from the International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use, the Food and Drugs Administration, the European Medicines Agency, and European Network for Health Technology Assessment.

Six design, two implementation, and two data analysis and interpretation issues were evaluated and in three, the ESMO-MCBS provided adequate protections. Seven shortcomings in the ability of the ESMO-MCBS to identify and address bias were identified. These related to (i) evaluation of the control arm, (ii) crossover issues, (iii) criteria for non-inferiority, (iv) substandard post-progression treatment, (v) post hoc subgroup findings based on biomarkers, (vi) informative censoring, and (vii) publication bias against quality-of-life data.

Interpretation of the ESMO-MCBS scores requires critical appraisal of trials to understand caveats in trial design, implementation, and data analysis that may have biased results and conclusions. These will be addressed in future iterations of the ESMO-MCBS.

To review the literature and obtain preferences and perceptions from experts regarding the role of randomized studies (RSs) and nonrandomized studies (NRSs) in systematic reviews of intervention effects.

Scoping review and survey of experts. Using levels of certainty developed by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) working group, experts expressed their preferences about the use of RS and NRS in health syntheses.

Of 189 respondents, 123 had the expertise required to answer the questionnaire; 116 provided their extent of agreement with approaches to use NRS with RS. Most respondents would include NRS when RS was unfeasible (83.6%) or unethical (71.5%) and a majority to maximize the body of evidence (66.3%), compare results in NRS and RS (53.5%) and to identify subgroups (51.7%). Sizable minorities would include NRS and RS to address the effect of randomization (29.5%) or because the question being addressed was a public-health intervention (36.5%). In summary of findings tables, most respondents would include both bodies of evidence–in two rows in the same table—when RS provided moderate, low, or very-low certainty evidence; even when RS provided high certainty evidence, a sizable minority (25%) would still present results from both bodies of evidence. Very few (3.6%) would, under realistic circumstances, pool RS and NRS results.

Most experts would include both RS and NRS in the same review under a wide variety of circumstances, but almost all would present results of two bodies of evidence separately.

Statin therapy has strong evidence supporting health benefits and mortality reduction in cardiovascular disease, stroke, diabetes, renal disease, and genetic lipid disorders. Further, reports that statin therapy might be protective against Alzheimer’s disease have subsequently been refuted in randomized trials. Low-level evidence based on case reports suggests that statins may adversely affect memory, a significant consumer concern. In this review, the published evidence on statins and memory in the elderly in randomized controlled trials and prospective observational cohort studies was examined in detail.

Overall, there was moderate-strength evidence that statin therapy did not increase the risk of dementia in the elderly and low-strength evidence for no increased risk for Alzheimer’s disease. Further, there was moderate-strength evidence that statin therapy in the elderly did not increase the risk for mild cognitive impairment or worsen global cognitive performance in the cognitively intact or impaired. There was moderate-strength evidence for no deterioration of memory function in the elderly.

On balance, there was a moderate level of evidence of neither harm nor benefit on memory; however, the published literature contains a number of deficiencies that are detailed in this review, not limited to selection biases and deficiencies of detailed testing.