When can a clinical trial be called ‘randomized’?

doi:10.1016/S0264-410X(02)00475-9

Vaccine

Volume 21, Issues 5–6, 17 January 2003, Pages 468-472

https://doi.org/10.1016/S0264-410X(02)00475-9 Get rights and content

Abstract

It is widely recognized that, in the context of the evaluation of medical interventions, randomized clinical trials constitute the gold standard. This is because randomization tends to balance both measured and unmeasured baseline characteristics, allows for masking, and provides a basis for inference. It is understandable, then, that investigators would wish to utilize this methodology whenever it is feasible to do so. Unfortunately, some studies are labeled as randomized when in fact they are not. These studies then receive more credibility, and influence medical practice, more than they deserve to. After reviewing the benefits of randomization, paying particular attention to the specific aspects of randomization that confer each benefit, we will explore the issue of what constitutes a randomized study.

Introduction

In the context of the evaluation of medical interventions, randomized clinical trials (RCTs) constitute the gold standard [1]. This is because randomization tends to balance both measured and unmeasured baseline characteristics [2], allows for the possibility of masking [3], and provides a basis for inference [4]. In vaccine studies, e.g. the vaccinated group should be comparable to the control group, so that like is compared to like [3] and any observed differences can be attributed to the vaccine itself [5]. It is understandable, then, that the RCT reigns supreme in the evidence-based hierarchy of study designs [1]. Given the credibility RCTs enjoy [2], it is also understandable that investigators who want to maximize the credibility of their studies would opt for RCTs. For this reason, the RCT has gained widespread popularity. For example, a 9 January 2002 Science citation index expanded search on the word “randomized” found 20 articles reporting on RCTs published in Vaccine in the year 2000. Yet randomization is not without controversy; it has even been described as the devil in some contexts [6]. The combination of credibility reserved for RCTs [2] and desire on the part of some researchers to avoid randomizing [6] may lead to the temptation to label as an RCT a study that in fact was not randomized.

In Section 2, we explore the issue of precisely what constitutes an RCT and when a study can claim to be randomized. In Section 3, we review the benefits of randomization, paying particular attention to the specific aspects of randomization that confer each benefit. In Section 4, we mention some studies that ostensibly were randomized but in fact provided details that contradict randomization. In Section 5, we critique the description of randomization offered in RCT reports published in Vaccine in 2000. In Section 6, we offer suggestions for improving the reporting of randomization details in RCTs.

Section snippets

What is a randomized study?

Randomization is an attribute not of the allocation sequence but rather of the method by which this allocation sequence was generated. There are many ways to randomize [7], so randomization refers to a broad collection of allocation methods. For example, the allocation proportions need not be 1:1, and may even change during the trial. Such was the case in a randomized study of nurse telehealth care for depression [8], for which the initial 60:40 randomization to two groups was later changed to

Salient features of randomization

In this section, we consider the specific aspects of randomization that confer the benefits of baseline balance [2], masking [3], and a basis for inference (the calculation of valid P-values) [4]. Regarding baseline balance, randomization has been called “the best method [for] removing selection bias between two groups of patients” [13]. Yet Chalmers [3] stated “that treatment allocation based on strict alternation abolishes selection bias as effectively as treatment allocation based on strict

Nonrandomized studies that claimed to be randomized

As we have seen in Section 3, nonrandomized studies allow for bias and do not support the calculation of valid P-values [18], [19]. Given that RCTs generally enjoy more credibility than nonrandomized studies [2], it is clear that a study with biased results due to a bad randomization (or no randomization) is most damaging to the medical community when the problems with (or lack of) randomization are not detected [13]. As such, referring to a nonrandomized study as randomized is quite

Evaluating the reporting of RCTs in Vaccine

In Section 4, we discussed reports that described nonrandomized studies as randomized. The only way to detect this is from the contradiction in the article. It is unlikely that the authors of such reports intend to deceive, because if they did, then they would delete from the article any mention of the actual nonrandomized procedure used. In fact, because of the common practice of not documenting the randomization methods, we would speculate that nonrandomized studies are labeled as randomized

Discussion

Altman [52] reported that among scientists surveyed, 98.6% consider as “unethical the provision of a misleading explanation of how the study was done to make it look sounder than it was; 12% did not deem unethical incomplete reporting of research which made it impossible to replicate the work in other laboratories.” As we stated earlier, we find it unlikely that authors who provide conflicting randomization details do so in an attempt to deceive or mislead. We also find it unlikely that authors

References (56)

D.C. Hadorn et al.
Rating the quality of evidence for clinical practice guidelines
J. Clin. Epidemiol.
(1996)
D.H. Williams et al.
Reporting of assignment methods in clinical trials
Controlled Clin. Trials
(1994)
V.W. Berger et al.
Detecting selection bias in randomized clinical trials
Controlled Clin. Trials
(1999)
D.A. Grimes et al.
A “randomized” controlled trial without randomization
Am. J. Obstet. Gynecol.
(1996)
F. Mosteller et al.
Reporting standards and research strategies for controlled trials. Agenda for the editor
Controlled Clin. Trials
(1980)
M.C. Grothaus et al.
Selection of an immunogenicpeptide mimic of the capsular polysaccaride of Neisseria meningitidis serogroup A using a peptide display library
Vaccine
(2000)
J.F. Fries
Compression of morbidity in the elderly
Vaccine
(2000)
P.R. Pittman et al.
Anthrax vaccine: increasing intervals between the first two doses enhances antibody response in humans
Vaccine
(2000)
A.D. Horne et al.
Intent-to-treat analysis and preventive vaccine efficacy
Vaccine
(2000)
S.K. Kim et al.
Effect of immunological adjuvant combination on the antibody and T-cell response to vaccination with MUC1-KLH and GD3-KLH conjugates
Vaccine
(2000)

Cited by (45)

Group sample sizes in nonregulated health care intervention trials described as randomized controlled trials were overly similar
2020, Journal of Clinical Epidemiology
Citation Excerpt :
Rather than simple randomization, this procedure would ensure balanced group sizes if all envelopes are used. Others have reported that trialists sometimes fail to report that they have used a restricted randomization design or that they have used nonrandomized allocation procedures, such as the day of the week or alternating participants [14,19–24]. We attempted to select trials where this is not the case, but it is possible that some trialists may have misunderstood the definition of simple randomization, even with an explanation, and reported that they used simple randomization in these cases.
We evaluated whether sample sizes in different arms of two-arm parallel group randomized controlled trials of nonregulated interventions were systematically closer in size than would plausibly occur by chance if simple randomization had been applied.
We searched PubMed for trials of nonregulated health care interventions that did not report using restricted randomization from journals in behavioral sciences and psychology, nursing, nutrition and dietetics, rehabilitation, and surgery. We emailed trial authors to clarify randomization procedures.
We identified 148 nonregulated intervention trials that indicated they used simple randomization. Difference in trial arm sizes was smaller than would be predicted by chance if simple randomization had occurred in all trials (P < 0.001). Rather than approximately half of the trials being within a 50% prediction interval for the difference, 96% had differences within this interval. Results were similar and statistically significant (P < 0.001) for trials that were published in journals with impact factors ≥ 4 and when stratified by type of nonregulated intervention.
There is a need for education and better understanding of clinical trial methods to ensure that randomization procedures are implemented as intended and reported fully and accurately.
Characterizing permuted block randomization as a big stick procedure
2016, Contemporary Clinical Trials Communications
Citation Excerpt :
In other words, the rocket big stick we consider forces BBB after AAA (if the MTI is three), but does not force BBBBBB, as it could, because if it did (which may seem like the intuitive form of a rocket big stick), then that would have to be followed by AAAAAA, and we see that the entire remainder of the allocation sequence would be deterministic. This would be glorified alternation, and alternation is certainly frowned upon, since it is not even randomization [17]. If we accept the earlier arguments in favor of preferring the control of selection bias to the control of chronological bias, then we must conclude that the standard big stick procedure is superior to the rocket big stick procedure, and also that the standard version of Chen's procedure is superior to the rocket version.
There are numerous approaches to randomizing patients to treatment groups in clinical trials. The most popular is permuted block randomization, and a newer and better class, which is gaining in popularity, is the so-called class of MTI procedures, which use a big stick to force the allocation sequence back towards balance when it reaches the MTI (maximally tolerated imbalance). Three prominent members of this class are the aptly named big stick procedure, Chen's procedure, and the maximal procedure. As we shall establish in this article, blocked randomization, though not typically cast as an MTI procedure, does in fact use the big stick as well. We shall argue that its weaknesses, which are well known, arise precisely from its improper use, bordering on outright abuse, of this big stick. Just as rocket powered golf clubs add power to a golf swing, so too does the big stick used by blocked randomization hit with too much power. In addition, the big stick is invoked when it need not be, thereby resulting in the excessive prediction for which permuted blocks are legendary. We bridge the gap between the MTI procedures and block randomization by identifying a new randomization procedure intermediate between the two, namely based on an excessively powerful big stick, but one that is used only when needed. We shall then argue that the MTI procedures are all superior to this intermediate procedure by virtue of using a restrained big stick, and that this intermediate procedure is superior to block randomization by virtue of restraint in when the big stick is invoked. The transitivity property then completes our argument.
The intraocular pressure-lowering effect of prostaglandin analogs combined with topical β-blocker therapy: A systematic review and meta-analysis
2010, Ophthalmology
Citation Excerpt :
For each publication, a quality score was calculated, where yes was scored as 1 point for a certain quality item and no and do not know or uncertain were scored as 0 points. For scoring quality items on masking, allocation concealment, and intention-to-treat analysis, suggestions from Berger and Bears,10 Berger and Christophe,11 Berger et al,12 and Berger and Weinstein13 were used. The statistical analysis was described in detail previously.6
To estimate the intraocular pressure (IOP)-lowering effect of prostaglandin analogs (PGAs) when added to topical β-blocker (BB) therapy.
Systematic review and meta-analyses of randomized clinical trials.
Twenty-nine articles reporting on 33 study arms and 3 control arms.
Articles published between January 1, 1990, and August 18, 2009, were identified in relevant databases. The pooled IOP-lowering effects at the 1- to 3-month follow-ups were calculated by performing random effects meta-analyses.
Absolute and relative change in IOP for mean diurnal curve and highest and lowest IOP decrease on the diurnal IOP curve.
Adding 0.005% latanoprost in the evening to 0.5% timolol twice daily resulted in a pooled change of –6.3 mmHg (95% CI, –7.1 to –5.5 mmHg, mean IOP curve); switching to the fixed combination of 0.5% timolol and 0.005% latanoprost in the morning resulted in a pooled change of –2.8 mmHg (95% CI, –3.3 to –2.3 mmHg, mean IOP curve). Starting with any fixed combination of 0.5% timolol and a PGA in the morning resulted in a pooled change of –8.4 mmHg (95% CI, –9.1 to –7.6 mmHg, mean IOP curve) and varied between –9.1 mmHg (95% CI, –9.9 to –8.2 mmHg, highest) and –7.9 mmHg (95% CI, –8.5 to –7.2 mmHg, lowest); starting with any fixed combination of 0.5% timolol and a PGA in the evening resulted in a pooled change of –8.6 (95% CI, –9.2 to –8.0 mmHg, mean IOP curve) and varied between –10.1 mmHg (95% CI, –11.0 to –9.2 mmHg, highest) and –7.3 mmHg (95% CI, –8.1 to –6.4 mmHg, lowest).
The concomitant use of latanoprost and timolol leads to a larger additional IOP reduction when compared with the fixed combination. There is no difference in mean IOP-lowering effect between evening and morning dosing of a fixed combination of timolol and a PGA, although the largest IOP decreases are seen with evening dosing. These findings are explained by differences in study design. When time points of IOP measurements close to the peak or trough moment of a drug are included, the IOP-lowering effect will be overestimated or underestimated, respectively.
Proprietary or commercial disclosure may be found after the references.
Methodology of clinical studies dealing with the treatment of envenomation
2010, Toxicon
Citation Excerpt :
Their objective is to test a therapeutic hypothesis and permit a causal imputation by way of a strict methodology. However, and contrary to widespread opinion, randomization – even if correctly implemented (Altman and Schulz, 2001; Altman et al., 2001; Berger and Bears, 2003) – does not by itself define a RCT. Other equally important conditions must be met to ensure the validity of the result.
A total of 142 clinical studies have been devoted to the treatment of envenomations, of which 115 address snake bites, 20 scorpion stings, and 8 other animals (one addresses both snake and spider envenomation). Antivenom use was studied in 118, of which 82 addressed efficacy, 43 evaluated safety, 23 studied dosage and 8 explored other issues. Besides anecdotal clinical reports, three classes of clinical studies are distinguished: (a) observational clinical studies (55 of the total) which analyze series of cases, (b) comparative clinical studies (36) which compare therapeutic products or treatment regimens without a gold standard for comparison and (c) randomized clinical trials (RCT, 51). The goals, methods and constraints of design of RCT are determined by whether explanatory (analytical) or pragmatic considerations are prioritized. Explanation-oriented RCT rely on strict group comparability before and during treatment, in order to ensure the internal validity of the study. The pragmatically-oriented RCT aims at establishing the superiority of a treatment over another, the goal being to maximize the external validity of the trial (that is, its application in current practice). We found that all clinical studies of treatment of envenomation lean markedly toward the explanatory end and suggest that, given some particularities of envenomation as a medical condition, a more pragmatic approach may be of value, particularly under the conditions prevalent for clinical studies in developing nations.
Bias in experimental nursing research: Strategies to improve the quality and explanatory power of nursing science
2010, International Journal of Nursing Studies
In a guest editorial in this journal, Rahm Hallberg [Rahm Hallberg, I., 2006. Challenges for future nursing research: providing evidence for health-care practice. International Journal of Nursing Studies 43, 923–927] called for research which has greater explanatory power to determine the effectiveness of nursing interventions. In this paper we critique the suggestion made by the evidence-based nursing movement that randomisation per se is the principal route to better quality nursing research. In contrast, we evaluate the new CONSORT criteria for pragmatic RCTs, which assess the quality of strategies to reduce selection, performance, attrition and detection biases, allowing many different types of comparative studies to be covered by application of the checklist. We propose that randomisation alone is a necessary but insufficient strategy and that nursing researchers rise to Rahm Hallberg's challenge by adopting the extended criteria to assist in the critical appraisal, design and reporting of all experimental research in nursing.
The quality of reporting of randomized controlled trials of HuatuoZaizao pill for stroke
2023, Frontiers in Pharmacology

View all citing articles on Scopus

View full text

Published by Elsevier Ltd.

When can a clinical trial be called ‘randomized’?

Abstract

Introduction

Section snippets

What is a randomized study?

Salient features of randomization

Nonrandomized studies that claimed to be randomized

Evaluating the reporting of RCTs in Vaccine

Discussion

J. Clin. Epidemiol.

Controlled Clin. Trials

Controlled Clin. Trials

Am. J. Obstet. Gynecol.

Controlled Clin. Trials

Vaccine

Vaccine

Vaccine

Vaccine

Vaccine

Vaccine

Vaccine

Vaccine

Vaccine

Vaccine

Vaccine

Vaccine

Vaccine

Vaccine

Vaccine

Vaccine

Vaccine

Vaccine

Controlled Clin. Trials

Controlled Clin. Trials

Measuring effects without randomized trials?

Med. Care

Comparing like with like: some historical milestones in the evolution of methods to create unbiased comparison groups in therapeutic experiments

Int. J. Epidemiol.

Pros and cons of permutation tests

Statistics Med.