Principles and procedures for data and safety monitoring in pragmatic clinical trials

For either pragmatic clinical trials or more traditional explanatory trials, monitoring of recruitment or enrollment addresses the same fundamental question: Will this trial enroll a sufficient number of participants to yield a credible and reliable result? Addressing that question requires monitoring the rate of enrollment and comparison to original expectations. Monitoring may also include comparing the clinical characteristics of those enrolled to expected characteristics (especially characteristics related to primary study outcomes) and comparing actual outcomes or event rates in participants assigned to a control or usual care group to the anticipated outcomes or event rates previously used to estimate sample size. Procedures for this monitoring would usually differ little between pragmatic clinical trials and explanatory trials.

Responses to slow or inadequate enrollment, however, may differ between pragmatic trials and explanatory clinical trials. Rather than relying on volunteers or referrals, pragmatic trials may automatically enroll all eligible patients served by participating providers, clinics, or health systems. Population-based enrollment intends to increase both efficiency of trial enrollment and representativeness of trial participants. Eligible participants may be identified directly from data in electronic health records. One fortunate consequence of this population-based recruitment is that trial eligibility or enrollment is often more predictable or stable [10]. Investigators can apply trial eligibility criteria to historical records data to accurately estimate the number or rate of eligible patients in each study site [10]. One unfortunate consequence of this procedure, however, is that investigators or study staff have fewer options for increasing enrollment if actual experience falls short of prediction. When all eligible participants in a defined population are identified or enrolled automatically (e.g., they meet inclusion/exclusion criteria defined in electronic records) rather than actively recruited (e.g., through advertisements or outreach to treating clinicians), investigator or study staff effort will have no effect on enrollment rate. Increasing enrollment will instead require extending the enrollment period, relaxing eligibility criteria, and/or including additional providers, clinics, or health systems. A pragmatic trial DSMB observing an inadequate rate of enrollment may, therefore, be quicker to recommend adding study sites rather than waiting in hope that more vigorous advertising or outreach at existing study sites will yield greater returns.

Cluster-level randomization is a common feature of pragmatic trials and can create an additional layer of complexity in assessing the likelihood that recruitment will yield an adequate sample size. Statistical power or precision depends not just on the number of patient participants, but also on the number of clusters randomized, the distribution of cluster sizes, and the correlation of outcomes within those clusters [11, 12]. While some of these may be measurable in advance, cluster randomization creates the potential for imbalance in cluster size or imbalance in within-cluster correlation between randomization groups. Consequently, monitoring of enrollment in cluster-randomized trials may require more than simply plotting observed number of patients enrolled against expected.

For example:

Monitoring delivery of study treatments typically considers two questions. The first involves detection of adverse effects, especially previously unrecognized adverse effects of new treatments. The second involves fidelity or quality of treatment delivery.

Regarding detection of adverse effects, explanatory clinical trials often involve new treatments, where experience is limited, and discovery of unknown adverse effects is an important goal. That discovery could involve surveillance for any rare or unexpected events in patients receiving a new treatment. It could also involve comparing rates of more common events in patients receiving that new treatment to rates in those receiving a standard treatment or placebo. When monitoring for unexpected events, any inference regarding the relationship to study treatment often depends on the clinical judgment of study staff, investigators, or independent monitors. Comparisons of rates between randomization groups typically requires interim analyses of timely data from blinded research assessments.

In contrast, pragmatic trials often involve treatments in widespread use, where safety profiles are well established. Any apparent signal of a previously unknown adverse effect must be weighed against prior experience. If the cumulative sample size of previously completed trials far exceeds a planned new trial, then a new and unexpected “signal” of an adverse effect is more likely to be a chance finding. Consequently, attempts to identify previously unrecognized risks of study treatments are expected to have lower value in pragmatic trials involving well-studied treatments or treatment components.

The procedures employed to identify or quantify adverse effects in explanatory clinical trials may be impractical in pragmatic trials. When study outcomes or potential adverse events are ascertained from health system records, access to data is often delayed, and clinical detail may be limited. For some hospitalizations, only insurance claims data (limited to diagnosis and procedure codes) may be available to the research team. Timely review of clinical data to assess “relatedness” of adverse events to study treatments may not be possible.

For more common adverse effects, valid inference regarding relatedness requires comparison to an untreated control group. In many pragmatic trials, contact with participants is unequal across treatment arms. If ultimate trial outcomes are ascertained from health system records, study staff may have no contact with participants assigned to a usual care control group. If contact with study staff differs markedly between treatment arms, between-group comparison of adverse events reported to or discovered by study staff would certainly be biased.

Monitoring for adverse effects should also consider variable intervention fidelity or adherence and risks in specific clinical populations. Adverse effects of study treatments may be more often observed when or where those treatments are delivered most vigorously. Patients enrolled in pragmatic trials may differ from those in previous explanatory trials, so new adverse effects may still emerge.

For example:

Regarding effective delivery of interventions, the relevant question for any clinical trial is whether differences in treatment exposure between groups support valid inference, especially in the event of a null result. A finding of no difference between treatment groups may be uninterpretable if the difference in actual treatment exposure is too small [13]. Explanatory clinical trials typically intend to assess or compare efficacy of treatments when optimally delivered [5, 14]. Consequently, monitoring of those trials should attend to the quality or fidelity of treatment delivery and take appropriate corrective action when quality deviates too far from the optimal. Pragmatic trials do allow, or even embrace, variable uptake and imperfect fidelity [1, 5] because this variation increases external validity study. Acceptability and ease of use are central components of effectiveness. But there are limits to pragmatism, and monitoring should consider credible boundaries for uptake or adherence. Failure to deliver intervention components with minimally adequate fidelity or adherence may undermine valid hypothesis testing. Consequently, explanatory clinical trials and pragmatic trials might use similar methods or procedures for monitoring treatment quality or fidelity but have different thresholds for intervening when fidelity or quality fall short of expected [6].

For example:

Assuring that investigators and study staff act to minimize risk is distinct from monitoring adverse effects of study treatments. The latter question concerns inference or causation (Does the study treatment cause harm?), while the former concerns potential conflict of interest (When a risk or urgent need is discovered, do researchers place duty to participants over duty to the study protocol?). Attention to that potential conflict of interest is a fundamental responsibility of investigators and independent safety monitors in both explanatory and pragmatic trials. Lower levels of control over treatment in pragmatic trials, however, may limit investigators’ or monitors’ ability to identify or reduce risk to participants.

In explanatory trials, investigators or research clinicians are often directly involved in delivery of study treatments and monitoring of treatment effects. In pragmatic trials investigators and other study staff are often more removed from clinical care provided to study participants. Consequently, any actions to minimize risk are limited by how rapidly and accurately researchers can identify risk, what clinical actions study staff can take directly, or how well study staff can communicate risk information to treating clinicians.

For example:

The integrity and utility of any trial result depend on the integrity and timeliness of data regarding the trial outcome(s). In explanatory trials, the research team often controls the entire chain of custody for outcome data. Outcomes are ascertained or measured by study staff, recorded in a study-specific database, and transmitted to a study data coordinating center. Monitoring the integrity and quality of study data typically involves auditing of case report forms and validation of the study database against original records.

In many pragmatic trials, outcome data are extracted from records generated by routine health care operations [1, 5, 15]. Researchers do not control every step in the chain of custody for those data. This separation of researchers from data generation and collection reduces the chance that researchers’ expectations or preferences will influence assessment of outcomes. But it allows for other sources of error. Data systems may change, or recording may be influenced by changes in business processes or EHR environment [15]. Apparent changes in study outcome rates could be artifacts of changes in data systems or indicate true changes in practice patterns. Monitoring of data quality includes both awareness of those potential disruptions and monitoring trends in outcome data streams for discontinuities [6, 16].

In explanatory clinical trials, monitoring of data integrity sometimes includes following the audit trail from an analytic database back to original clinical documents. When study data are extracted from health systems’ clinical or administrative records, obligations to protect patient privacy or health system proprietary information [17] may preclude direct access to original or primary data sources.

For example:

In explanatory trials, DSMBs review interim analyses to determine whether available data either clearly demonstrate a clinically meaningful difference (in benefit or risk) between treatments or clearly indicate that a clinically meaningful difference will not be detected within the proposed study sample size or follow-up period [19]. The ethical obligation to terminate study enrollment or delivery of study treatments sooner than planned is clearest when interim analyses indicate that continued enrollment would assign participants to a treatment already shown to have either inferior benefit or greater risk than an available alternative. That central obligation does not differ between explanatory and pragmatic trials. Plans for interim analyses in pragmatic trials must consider both practical and ethical differences from explanatory trials.

On the practical side, reliance on outcome data extracted from health system records means that data regarding benefits or harms may not be available for months after positive or negative outcomes occur. Consequently, the likelihood of detecting a clinically important difference between treatments prior to completion of enrollment may be small. Pragmatic trials may be concerned with downstream costs of alternative treatments. Premature determination of futility may preclude accurate assessment of costs or other downstream effects.

In addition, the ethical implications of terminating enrollment in a pragmatic trial may be different from those in an efficacy trial. Pragmatic trials often focus on implementation or policy questions, and a “policy-meaningful difference” is harder to define than a clinically meaningful difference [6]. Implementation decisions must consider magnitude of benefit, anticipated cost, and competing health system priorities. Consequently, early termination of a pragmatic trial as soon as the benefit of a new program or service exceeds a boundary of statistical significance does not necessarily guarantee that health systems will adopt or implement that new program. A larger sample or longer period of observation may be necessary to assess costs or broader benefits relevant to implementation decisions. In contrast, any evidence that a new program or service produces significant harm would certainly justify premature termination of trial enrollment or intervention delivery. Consequently, rules or boundaries for premature termination in pragmatic trials may often be asymmetric—more likely to stop for early indication of harm than for early indication of benefit. In cluster-randomized trials, interim analyses should appropriately account for clustering of outcomes within randomization units and attend to the possibility of bias due to imbalance between clusters.

For example:

Monitoring trial integrity and participant safety is an obligation of investigators, other study personnel, and external safety monitors. While NIH guidance emphasizes the need for independent review of trial conduct [4], review by an external DSMB may not always be necessary. Many questions may require review by an independent monitor or safety officer, but only a smaller list of questions require confidential review of unblinded data by an independent board.

Independent review is needed for questions regarding potential harms of study interventions and adequacy of actions to protect participants from harm or address urgent clinical needs. Independent review may also be necessary to assess adequate rate of study enrollment, adequacy of intervention quality or fidelity, and quality or integrity of study data. In most cases, however, addressing these questions does not require any comparison between treatment groups. Consequently, there is no need for confidential review of unblinded data by a DSMB. Review by an independent safety officer with appropriate clinical expertise is often sufficient.

An external DSMB is, however, generally necessary for any confidential review of interim analyses to detect benefit, harm, or futility. Results of such analyses should be concealed from investigators and study team members [6, 20]. As discussed above, formal interim analyses requiring DSMB review are less often necessary for pragmatic trials than for explanatory clinical trials. A reviewing DSMB should have adequate statistical expertise to interpret interim analyses and adequate relevant clinical and policy expertise to weigh the consequences of premature termination to current participants, potential future participants, and other people living with the condition of interest. In pragmatic trials, expertise regarding health system data and operations will also be essential. DSMB members must understand real-world practice conditions and sources of error in real-world data.

The core ethical and regulatory obligations of pragmatic trial investigators do not differ from those of investigators leading more explanatory clinical trials. But specific features of pragmatic trials may require different procedures to satisfy those obligations. Table 2 lists features of pragmatic trials that may influence data and safety monitoring plans.

While pragmatic trial departures from traditional explanatory trial design are intended to increase relevance and generalizability of findings, they also influence how investigators and independent monitors can monitor participant safety and assure trial integrity. Consequently, we recommend that investigators developing data and safety monitoring plans for pragmatic trials not simply replicate the monitoring procedures, analyses, and reports typically used in explanatory clinical trials. Instead, investigators should return to the core questions outlined in the original NIH policy:

Plans for monitoring, interim analyses, reporting, and (if appropriate) premature termination should clearly address each of these specific questions.