Heterogeneity in pragmatic randomised trials: sources and management

Centres involved in a pragmatic trial should be drawn from a similar range of patient care settings as those in the target population for which the designers intend the findings of the trial will apply [7]. If study centres are limited and highly selected, heterogeneity will be reduced and may no longer fit the target population. For instance, centres should not exclusively be university hospitals when the disease of interest is common, and patients are cared for in both community and university hospitals (e.g. NUTRIREA-2 trial [8], Table 3).

An option is to maximise the number and range of included centres, perhaps reducing the number of patients per centre. In trials conducted across a health system, it may even be possible to recruit centres in random sequence until the required sample size is reached, thereby vouchsafing representativeness of the available sample and thus applicability to the target population (e.g. IRIS trial [9], Table 4).

In a cluster randomised trial, heterogeneity in selected centres has two further consequences. First, more variability in outcome between centres increases the intraclass correlation coefficient, and as a result, a larger sample size is required. Second, variability in cluster (centre) size also increases the required total sample size [10].

Finally, although differences in patient characteristics between centres may reflect a different patient case-mix between centres [11], which is a welcome source of heterogeneity, such differences may also be due to the differential application of eligibility criteria, which is an undesirable source of heterogeneity [12]. Indeed, in a cluster randomised trial, such a phenomenon would be a source of bias because of differences in characteristics of included participants between the groups being compared; in an individually randomised trial, this situation may induce a centre effect, which would not be due to the intervention but rather to differences in following the trial procedures.

A pragmatic trial aims to recruit patients from an available population who are as similar as possible to the target population. This target population corresponds to the population that would receive the study intervention once it has been shown to be effective and scaled up in the usual healthcare setting. Eligibility criteria should not exclude patients who are less likely to respond to the treatment or those not likely to complete the follow-up. Success in representing the target population in the patients recruited for the trial contributes to the applicability of the trial’s results [13] to the target population. Inclusion and exclusion criteria are often more restrictive in trials of drug interventions than those assessing devices, surgery or other complex interventions; they are also more restrictive in industry-sponsored versus public agency-funded trials [14]. As an example, the TiME trial [15] had very few selection criteria for patients, thus promising very good applicability, besides the fact that it limited the risk of identification and recruitment bias (Table 5).

Even though sample size formulae may be the same, the reasoning about sample size differs in pragmatic and explanatory trials. First, intervention effects are expected to be smaller in pragmatic than explanatory trials, in part because of the inclusion of patients with a wider range of characteristics, for example those with comorbidities, who are less adherent, and/or who have both less severe conditions, and thus benefit less, as well as those whose condition is more severe and possibly intractable. Other features that might promote homogeneity and thus apparently greater effect sizes in explanatory trials include selecting caregivers and centres based on volume and experience [16]. Second, sample size parameters need to be carefully and realistically specified. A priori specifying a standard deviation that is lower than the post hoc estimate is a common problem [17] and results in optimistic sample size estimates and risks of insufficient statistical power. Therefore, attention should be paid to whether standard deviation estimates are derived from previously conducted explanatory trials—and therefore likely to be too low—or from administrative routinely collected data, for instance, which should adequately capture real-world heterogeneity.

A centre effect is to be expected in a pragmatic trial because of centre and participant heterogeneity, as previously discussed. The intervention delivery may also be tailored to the centre context, and such heterogeneity, which, in our opinion, should be welcomed because interventions are applied with heterogeneity in real practice, also contributes to a centre effect. Accordingly, to prevent imbalance between arms and improve power, individually randomised multicentre pragmatic trials should stratify randomisation on centre [18] (e.g. NUTRIREA-2 trial, Table 3). Prognostic factors may also be considered as stratification variables (e.g. ALIC⁴E trial [19], Table 6), notably when the sample size is small, thus limiting the risk of baseline imbalances [20]. Similarly, for cluster randomised trials, restricted randomisation such as, for instance, stratified randomisation or randomisation by minimisation, is advisable to limit chance imbalances (e.g. IRIS and TiME trials, Tables 4 and 5) [5].

Although not specific to pragmatic trials, unadjusted analyses of trials using stratified randomisation raise two issues. First, there is inconsistency if factors used to stratify randomisation are not taken into account when analysing the results. Second, ignoring stratification factors in the analysis leads to over-estimated standard errors, wider confidence intervals, inflated p-values and diminished power [21]. Although this is true for any randomised trial, it is a particular concern in pragmatic trials in which between-centre heterogeneity is expected to be higher, as discussed above. Accounting for centre effects is therefore advisable ant it has been shown that random-effects models offer better properties than fixed-effects models [21].

Subgroup analyses aim to identify interactions between treatment and pre-specified patient or centre characteristics [22]. Because pragmatic trials aim at informing decision-making rather than promoting an understanding of the mechanism of action, subgroup analyses should only be done if the same subgroups are meaningfully part of usual clinical care or policy decision-making, which requires that the distinction between these subgroups is readily accessible to clinicians (e.g. age, blood pressure), (e.g. APTS trial [23], Table 7) or policy-makers (e.g. subgroups defined by equality, diversity, and inclusion groups.

Although heterogeneity in the delivery of interventions is an undesirable feature of an explanatory trial (in which interventions must be standardised), in pragmatic trials, as in future usual care in the target settings, interventions may well be tailored to individual patient needs or the local context in which care is provided [24], especially for complex interventions [25] (e.g. OPERA Trial [26], Table 8).

Hawe et al. refer to standardisation by function as compared with standardisation by form (e.g. rather than using a common information kit, how information is provided may differ among centres while the function of the information remains constant across centres), acknowledging that mechanisms that are assessed (i.e. the very components of the intervention) can take different forms from one context to another [25]. Nevertheless, the core components of an intervention need to be specified [27]; otherwise, the interpretation of the results may be complex because one would not know what intervention is being evaluated.

Tailored interventions may contribute to a centre effect [18] or even a provider effect [28], but depending on the research question and trial intention, flexibility in interventions is relatively unproblematic as long as in the trial interventions are delivered by providers in a similar range of ways and in settings that match the target clinical settings. Doing so will introduce desirable heterogeneity in participant outcomes because it mimics reality in that interventions are rarely perfectly standardised in usual care.

Monitoring the extent of tailoring as well as co-interventions raises a further dilemma. On one hand, we want to better understand what actually happened, and this knowledge may help to scale up the intervention after the study has demonstrated benefit. This is the very aim of a process analysis, which is both desirable and recommended [29] (e.g. OPERA Trial, Table 8). On the other hand, any intrusive data collection is undesirable, because it may distort usual clinical practice and patient response. Indeed, patient and health provider behaviour should not be altered outside of the provision of the intervention, to limit as much as possible a Hawthorne effect [30]. Ideally, process measures and outcome assessments should be as unobtrusive as possible, perhaps obtained using administrative or electronic medical record data whose collection is part of the usual care.

Control interventions are typically non-protocolised usual care or, in comparative effectiveness research, another already widely used active treatment. The use of a usual-care control has several consequences. First, the control can be “no treatment,” but it should rarely be a placebo [31] because placebos are not used in usual clinical care outside of trial contexts. This unnatural comparison group may alter the results of the trial in unknowable ways. Moreover, a placebo control could contribute to an unnatural and undesirable homogeneity among patients allocated to the control group, by reducing recourse to self-prescription with medicines or other treatment modalities (e.g. ALIC⁴E trial, Table 6). It may also affect outcome assessment, which raises other issues, notably related to the risk of detection bias (cf. Outcome section). We acknowledge that not using a placebo may be a challenging issue for a regulatory agency and therefore, if relevant, encourage trialists to have preliminary discussions with these agencies to justify the need for avoiding placebos. Second, there may be different approaches to usual care in different centres of the target setting. This situation may be accommodated by more than one control group or a single control group that permits unrestricted implementation of a variety of different treatments used in routine care and thus averages out all the kinds of usual care provided [32]. Third, a usual care control means that we expect patients and providers to behave as they would outside a trial context. However, for both patients and providers, behaviours can be altered by trial enrolment, known as the Hawthorne effect [30]. Changes in patient and provider behaviours may affect patient outcome heterogeneity, probably by reducing it. This raises an unsolvable conundrum: except in rare situations, which must be approved by an ethics committee, both patients and providers must be informed that they are involved in a randomised controlled trial. This information procedure is a mainstay of ethical clinical research but may alter behaviours as compared with usual, unobserved, non-trial care. This situation is a strong argument for incorporating consent procedures in the flow of care [33], minimising the obtrusiveness of intervention and data collection in order to minimise participant awareness of the trial and thus minimise the Hawthorne effect.

Lack of compliance is common outside a trial context. Sample size calculation should take into account usual-care levels of compliance [34] (e.g. APTS trial, Table 7). Moreover, in pragmatic trials comparing usual-care interventions without blinding, patients from one group may sometimes be easily able to access another study group intervention, which may result in contamination. If this contamination is symmetrical between arms, then it increases variability and decreases the effect size estimate. If this contamination is not symmetrical between arms, which is the most plausible situation, it creates a bias, which can attenuate or exaggerate the effect size estimate. In both situations, the issue cannot be dealt with merely by increasing the sample size. Cluster randomisation may limit contamination, but it may also induce bias arising from the identification or recruitment of individual participants if this processes happen after randomisation [35]. This could be a worse problem than group contamination in the individually randomised version of that trial [36].

In explanatory clinical trials, compliance with intervention and control protocols by both providers and patients is enhanced by trial monitoring often followed by direct contact between a research assistant and the non-compliant patient or provider [37]. However, in pragmatic trials, efforts to promote compliance are undesirable unless such efforts are viewed as part of the intervention itself and would be scaled up in usual practice. The guiding principle is that outside of the study intervention—which should be provided similar to how it would be provided in future usual care should it be shown to be effective in this trial—other behaviours of providers and patients should be unaltered. Trial monitoring is deeply ingrained in the minds of both researchers and study sponsors and setting it aside when performing a pragmatic trial requires a paradigm change. Thus, in pragmatic trials, compliance should not be enhanced but rather considered an outcome and assessed unobtrusively [4]. In the TiME trial (Table 5), although the stated goal of pragmatism had been impaired owing to efforts made to enhance adherence and assess compliance, compliance turned out to be of major interest. Indeed, intervention fidelity was so poor that any difference between groups in haemodialysis session duration (the intervention assessed) vanished over time, which led authors to discontinue the trial.

Co-interventions, defined as additional treatments that are not part of the assessed intervention, are another source of heterogeneity. In an explanatory trial, possible co-interventions are listed in the study protocol; some of these may be allowed, but others are prohibited. In a pragmatic trial, co-interventions are not generally considered protocol violations: they are left to the discretion of patients and providers in the trial because this flexibility would apply to usual care in the target setting, once the intervention is in widespread use, and where similar co-interventions will be in use. Measuring them is of interest, but it remains a secondary objective aimed at understanding, and as much as possible, it should be done in an unobtrusive way.

Statistical analysis of a superiority trial is expected to be according to intent-to-treat, and this holds true for pragmatic trials [7, 38]. Indeed, per-protocol, completers, on-treatment or complier average causal effect (CACE) analyses aim at understanding what could be observed with optimal compliance and are more suited to explanatory trials [39]. Some argue that per-protocol analyses are of interest if the intervention is expected to be scaled up in settings where adherence to treatment is expected to be better than in the conducted trial [40]. However, this situation casts doubts on the representativeness of the selected settings. One may also argue that per-protocol or CACE analyses are of interest from a patient perspective because they may help patients decide between treatments, though the necessity for perfect compliance to achieve the effects in such analyses needs to be acknowledged. Thus, such analyses should remain secondary analyses.

Missing data is an important issue in intent-to-treat analysis. Missing data may be more prevalent in a pragmatic than explanatory trial in which monitoring is more stringent, except if data are obtained from well-completed medical or administrative registries [41]. Therefore, statistical methods to handle missing data, such as multiple imputation or covariate adjustment, should be used [42] (e.g. ACUDep trial [43], Table 9).

As an ancillary objective of a pragmatic trial, one may seek to better understand the assessed intervention. Thus, at the end of the study, a process analysis “[that] explore[s] the way in which the intervention under study is implemented” [29] may bring a complementary view taking into account contextual issues [44] (e.g. OPERA trial, Table 8). In the same way, per-protocol [40] or CACE analyses may help explain whether lack of treatment effect is due to lack of compliance, whereas subgroup analyses may help identify subgroups of patients who benefit most from the treatment. In a pragmatic trial, all these analyses are generally secondary ones, which means that no specific effort should be made to collect additional data for them if that extra data collection jeopardises the primary purpose of the study, perhaps by distorting the clinical setting and adding extra investigations or disruptive data collection. However, pragmatic trials aim at answering the questions that decision-makers need answered, so one cannot exclude the possibility that subgroup analyses may be part of the primary objective, for example, to investigate aims relevant to health equity.

In pragmatic trials, the primary outcome must be directly relevant to patients or the primary stakeholder because it needs to inform decision-making by patients, caregivers and policy-makers [2, 7]. The primary outcome of a pragmatic trial should ideally correspond to an outcome routinely assessed in usual care and is regarded as clinically important and therefore likely to influence providers’ decisions (e.g. TASTE Trial [45], Table 10).

Outcome assessment raises a conundrum. Some suggest that standardisation (i.e. applying standardised measurement methods), blinding and adjudication should be avoided because they do not correspond to usual practice [7]. Standardisation aims at reducing heterogeneity in outcome assessment, whose consequence is mainly a loss in power. Heterogeneity in outcome assessment also increases the risk of misclassification, which, may be a source of bias [46, 47]. Standardisation may occur for outcomes derived from interviews [48] but also for clinical examinations [49] or even in electronic health records [50]. Blinding and adjudication also aim at reducing the risk of bias (e.g. RESTART Trial [51], Table 11).

Problems arise mainly for non-objective outcomes. Subjective outcome assessment is indeed known to be potentially influenced by the beliefs, in relation to the treatments, of patients themselves, their caregivers or clinicians [52]. Moreover, in the absence of blinding, this influence may not be the same in the groups being compared. However, another view of this is that these subjective beliefs in relation to the effectiveness of interventions would be active in clinical practice, after the trial has shown one of the tested interventions as more effective and been implemented widely. In that case, the subjective beliefs in the intervention have been well captured in the trial and thus reflect the future usual-care situation accurately. In this quite common situation, eliminating the effect of subjective belief in the trial would eliminate necessary heterogeneity and result in an incorrect estimate of the effect size.

Actually, standardisation, blinding and adjudication do not have the same consequences. Although blinding as well as standardised data collection by researchers may indeed affect patient and care-provider behaviours, adjudication is less problematic because it can be performed after data collection, with blinding to the arm of the patient whose record is being assessed and therefore without bias. However, adjudication, as we most often know, is performed by outside and selected expert clinicians often using information or expertise not available to the clinician in usual care in some future setting. This might produce trial results that differ from results based on usual-care clinician assessments thus reducing the relevance of the trial for decision-making. Although this trial may not be biased (the finding is true for the patients and outcome measures of the trial), it is less applicable to the usual-care situation.

The data monitoring committee is expected to think differently when investigators have clearly articulated their intended goal of pragmatism [50]. The committee should pay more attention to protecting external applicability and avoiding co-interventions delivered by the research team (not the patient and care-provider co-interventions) that are not visible when reading the intervention description in the trial protocol. Depending on the unique circumstances of each trial and intervention being assessed, it may nevertheless keep its original function of monitoring for safety concerns.

Many pragmatic trials, especially of complex non-clinical interventions such as service delivery changes, may not collect data other than at the end of the trial, and so ongoing data monitoring is not relevant because the intervention is low risk. Hence, safety signals are considered unlikely and will not be formally monitored with trial data. This situation may suggest that instead of a data safety or monitoring committee, a more comprehensive trial management committee may be an appropriate supervisory structure, paying more attention to issues such as intervention implementation, patient and centre recruitment, although provision should be made for processes to deal with data confidentially should the need arise during the trial.

If ongoing safety data collection is planned for a pragmatic trial, unobtrusive data sources such as administrative and electronic medical record data may be preferred because they have no effect on the flow of care. However, collecting from these sources may also have substantial time-lags before reliable datasets are assembled and cleaned. Therefore, safety monitoring for acute intervention-related injury, requiring a quick turnaround for action, may have to depend on clinical suspicion. Because intensive safety monitoring may disrupt the usual flow of care, a highly pragmatic design may not be suitable for trials evaluating interventions whose side-effect profile is not yet clear.

Although vulnerable patients, including those with co-morbidities, are commonly excluded in explanatory trials, a more inclusive approach may be adopted in pragmatic trials, provided adequate protections are in place. For patients with co-morbidities, protections may include flexibility in administration of the study intervention to meet individual patient needs (e.g. dose reduction) and additional clinically indicated follow-up visits. When patients have diminished capacity to provide consent, a surrogate decision-maker may be required. This may also be the case for emergency research such as trials conducted in intensive care units.

Written informed consent for trial participation is standard for explanatory trials. Pragmatic trials are commonly conducted in primary care settings and usually involve routine medical interventions. Although the ethical principle of respect for persons requires that the autonomy of participants be respected, a more clinical approach to consent in pragmatic trials may achieve the same goal with less intrusion (and thus less propensity to increase homogeneity). Kim et al. [33] describe one such clinical approach to consent called “integrated consent”, whereby informed consent to participation in a pragmatic trial is sought by the health provider in the clinic, during the usual course of care delivery. The health provider discloses key features of study participation verbally and records the patient’s consent or refusal in the electronic health record. In a cluster randomised trial, when the study intervention is a cluster-level intervention (thus, indivisible at the level of the individual) and poses only minimal risk to participants, research ethics committees may grant a waiver of consent when the science would be compromised by seeking consent [58].