Traditional efficacy double-blind randomised controlled trials (DBRCTs) are considered the “gold standard” study design for assessing the efficacy and safety of new medicines; however, their conduct in highly selected patient populations and in highly controlled settings limits the generalisability of their findings to patients seen in everyday clinical practice. |
Pragmatic effectiveness trials conducted in the routine clinical care setting allow for the evaluation of the effectiveness of medicines in the presence of real-world factors related to patients, actual medication use, and healthcare systems, thus providing a more complete picture of the benefit/risk profile of a medicine to support healthcare decision-making. |
In this article, we discuss the key features and advantages/limitations of pragmatic effectiveness randomised controlled trials (RCTs) compared with traditional efficacy DBRCTs, using the Salford Lung Study (SLS) programme as an illustrative example. |
SLS was the world’s first prospective, phase III, pragmatic RCT to evaluate the effectiveness of a pre-licensed medication in a primary care setting using electronic health records and through collaboratively engaging general practitioners and community pharmacists in clinical research. |
Key learnings from SLS that may help inform the design of future pragmatic effectiveness RCTs include: (1) ensuring that the trial setting and operational infrastructure are aligned with routine clinical care; (2) recruiting a broad population of patients with characteristics as close as possible to patients seen in routine clinical practice, to maximise the generalisability and applicability of the trial results; (3) ensuring that patients and local healthcare professionals (HCPs) are suitably engaged in the trial, to maximise the chances of successful trial delivery; and (4) careful study design, incorporating outcomes of value to patients, HCPs, policymakers and payers, and using pre-planned analyses to address scientifically valid research hypotheses to ensure robustness of the acquired data. |
Introduction
Efficacy and Effectiveness RCTS: Overview and Major Differences
Traditional efficacy DBRCTs | Pragmatic effectiveness RCTs | |
---|---|---|
Patients | Strict inclusion/exclusion criteria Patients with significant comorbidities and severe disease typically excluded Highly selected (“ideal”) patient population Limited relevance to patients in routine clinical care setting Good inhaler technique mandated High adherence mandated | Broad inclusion criteria, minimal exclusion criteria Patients with comorbidities and severe disease included Broad, heterogeneous patient population Greater relevance to patients in routine clinical care setting Variable inhaler technique Variable (and frequently poor) adherence |
Study design/conduct | Designed to test efficacy under near-ideal conditions (i.e. where confounding factors are minimised) Provide data on the efficacy and safety of a medicine, albeit in a highly controlled setting Typically recruited in ambulatory care/outpatient centres Randomisation (± stratification) and masking (e.g. double-blind, double-dummy) to limit bias due to systematic differences between treatment groups Can assess experimental medicine versus a placebo or “gold standard” comparator Treatment per protocol; generally treatment modifications are not permitted Frequent study visits/monitoring Adherence to treatment actively monitored and encouraged | Designed to test effectiveness in the presence of real-world factors Provide data on the overall treatment strategy in a real-world setting Conducted in routine clinical practice in primary care; patient management reflective of usual clinical care Randomisation (± stratification) to limit bias due to systematic differences between treatment groups; typically open-label in design Usually assess experimental medicine against usual care or established standard-of-care Treatment modifications permitted based on physician’s clinical opinion Few mandatory study visits; limited disruption to patients’ normal routine No monitoring or active encouragement of treatment adherence; patients’ health behaviours as normal |
Outcomes/data | Data have high internal validity, limited external validity Endpoints often designed to enable regulatory approval/licensing | Data have high external validity Often include additional endpoints of interest, e.g. healthcare resource utilisation, patient-reported outcomes |
Transferability/generalisability | Treatment effect in the real world has to be estimated Culturally accepted as most informative and therefore transferable, although external validity is weak | Data more generalisable to the overall disease population Effect of healthcare systems and access to medicines and cultural factors may need to be considered |
Existing diagnosis (often pragmatic or clinical) Access to medical care Non-adherence to prescribed medication (over- or under-treating) Poor inhaler techniquea Poor compliance with treatment advice and follow-up Comorbidities/coexisting medical conditions Polypharmacy Cigarette smoking and/or recreational drug use Variability in health literacy Diversions and distractions caused by life and social events, crises, shift-work patterns, accidents and injuries |
Scenario | Pros | Cons | |
---|---|---|---|
Process of care | In DBRCTs: Protocols demand atypical conduct from patients and HCPs, often requiring frequent, rigorous and prolonged study assessments Optimal treatment compliance and inhaler device technique is strongly encouraged Treatment pathway is strictly defined, and deviations from prespecified treatment may result in patients being withdrawn from study (restricting freedom of choice around patient care) | Rigorous assessment in efficacy DBRCTs allows for robust data acquisition and for patient safety to be closely monitored By strongly encouraging optimal treatment compliance, DBRCTs are able to provide an accurate profile of treatment efficacy and safety for an intended dose | A downside of the highly controlled environment in which efficacy DBRCTs are conducted is the impact on the behaviour of both patients and physicians, which does not truly reflect the characteristics of a medicine when used in the real-world setting |
In pragmatic effectiveness RCTs: Care is aligned with that received in routine clinical practice, with minimal scheduled study visits/assessments and minimal disruption to patients’ everyday lives Physicians have freedom of choice to modify patients’ treatment as deemed necessary | Study design permits freedom of choice around patients’ care and allows the normal behaviours of patients and HCPs to be expressed; thus, acquired data may be more generalisable to patients seen in everyday clinical practice | Data from pragmatic effectiveness trials more likely to be confounded by extraneous variables, not controlled for by virtue of the trial being conducted in a real-world setting | |
Data collection | In efficacy DBRCTs, patients’ data are typically entered into eCRFs by investigators/dedicated study team members during or following scheduled study visits The challenge for pragmatic effectiveness trial design is to adequately balance the delivery of highly accurate and complete data while minimising the level of interference that data entry and verification pose to routine practice. Effectiveness trials may rely on data extraction from patients’ EHRs or other spontaneous reporting systems, as well as eCRFs for data capture In contrast to traditional efficacy DBRCTs, where neither investigator nor patient has knowledge of the assigned treatment, pragmatic effectiveness RCTs are typically open-label in design | Compared with eCRFs, EHR-based data capture has advantages in terms of allowing for remote data collection in real time (avoiding recall or transcription bias) and in providing the opportunity for long-term follow-up of the trial population after the study has completed | Disadvantages of EHR-based data capture include missing data, lack of representation of endpoints of interest, and potential issues with accessibility for research purposes in certain regions/countries Conducting effectiveness trials in routine clinical care means that some investigators may be inexperienced in clinical research and processes must be implemented to manage this to ensure robust data collection (accordingly with resource, logistics, and cost implications) An open-label study design creates the potential for bias in the acquired data due to behavioural effects that arise as a consequence of knowledge of the treatment that has been administered, e.g.: Physicians may be biased in assigning causality of AEs to an investigational medicine rather than to a well-established standard-of-care treatment where common AEs are well known—this could have a positive or negative effect on AE reporting There may be increased HCP/patient vigilance with a new medicine, resulting in higher rates of healthcare contacts than with a more familiar treatment option Patients may merely have a preference for, and revert to taking, a more familiar treatment, thus impacting on adherence to the assigned study treatment The Hawthorne effect—where individuals modify an aspect of their behaviour in response to their awareness of being observed—may apply and confound data collected in both traditional efficacy DBRCTs and pragmatic effectiveness RCTs; however, this effect may be less likely in effectiveness trials that are designed to minimise disruption to everyday clinical care |
Trial eligibility criteria/patient population | The stringency of eligibility criteria for traditional efficacy DBRCTs versus pragmatic effectiveness RCTs will dictate the nature of the patient populations recruited, and this can have a profound effect on the data collected during such trials Patients in traditional efficacy DBRCTs are usually recruited in ambulatory care/outpatient centres, tend to be healthier than the non-trial disease population, and often participate in multiple trials In high-recruiting research centres, investigators may hold a database of patients who are “ready to enrol”. Such patients will meet trial inclusion/exclusion criteria as standard, are quick to learn and maintain excellent inhaler technique, intellectually capable, highly compliant with study protocols and procedures, and are familiar with a range of treatment devices and study assessments Pragmatic effectiveness RCTs seek to recruit a broad participant population with characteristics as similar as possible to patients who will eventually be prescribed the medicine in routine clinical practice. To achieve this, such trials typically employ minimal inclusion/exclusion criteria | The highly selected patient populations in traditional efficacy DBRCTs allow for testing of the efficacy of a medicine under conditions where confounding factors are minimised; thus, data have high internal validity | The strict entry criteria/requirements for adherence to protocol in traditional efficacy DBRCTs may preclude otherwise eligible patients from participating (e.g. patients from deprived areas, for reasons including difficulties with/costs of getting to the research site, or working and family commitments). This has led to the concept of “persistent participators” in efficacy DBRCTs—a population that is non-representative of patients treated in real-world practice Findings from efficacy DBRCTs have limited applicability/generalisability of the acquired data to the wider disease population—low external validity An example would be the collection and interpretation of AEs. In a pragmatic effectiveness trial, by virtue of enrolling a broader population of patients (including those with comorbidities and more severe disease), it is likely that a higher incidence and/or wider variety of AEs will be reported than in an efficacy DBRCT evaluating the same medicine These limitations are somewhat circumvented in pragmatic effectiveness RCTs In pragmatic effectiveness RCTs, recruiting a broad participant population may introduce additional variability to the data set, and a greater number of patients may need to be enrolled to power the study to demonstrate treatment effect, compared with traditional efficacy RCTs |
Outcomes | Endpoints in traditional efficacy DBRCTs (registrational trials in particular) are often dictated by outcomes of interest to regulatory authorities, often require frequent assessments and diary cards/electronic diaries, and serve as constant reminders of disease state and treatment response In pragmatic effectiveness RCTs, it is desirable to select endpoints that are relevant to patient-centric goals for treatment and that physicians routinely use to assess patients and make treatment decisions, so as to optimise external validity and transferability of the data, and enhance value to clinicians, payers, and policymakers In respiratory trials, for example, such endpoints would include exacerbations, hospitalisations, mortality, validated patient-reported outcomes, and quality-of-life measures | In effectiveness trials, it is desirable to minimise the impact of study assessments by selecting endpoints and a frequency of measures that ideally can be gathered with little or no impact on the patient or HCP, and where observer bias is controlled—crucial for an open-label study design | In pragmatic effectiveness RCTs, certain endpoints may be precluded because of the intensive monitoring that would be required (e.g. serial lung function, daily diaries) |
Data analysis/interpretation | In routine clinical practice, a patient’s treatment will be adjusted at the discretion of the treating physician Treatment modifications are rarely permitted in traditional efficacy DBRCTs, but are allowed (albeit with potential restrictions) in pragmatic effectiveness RCTs, with implications for the analysis and interpretation of the study data (particularly important for safety evaluation) In effectiveness RCTs where treatment can be modified, careful consideration must be taken as to whether specific study endpoints will be evaluated as ITT (i.e. according to randomised treatment group) or by actual treatment received Effectiveness endpoints will typically be analysed as ITT, which is equivalent to comparing the treatment strategies being investigated in the effectiveness RCT Safety data ought to be presented both by randomised treatment group and also by actual treatment | In traditional DBRCTs, efficacy and safety endpoints are typically analysed according to the ITT principle. Interpretation of data is more straightforward; randomisation group equates to treatment group and data can be analysed accordingly (e.g. safety events can be attributed to randomised treatment) In effectiveness RCTs, analysis by actual treatment received allows for assessment of true exposure risk of a medication | In effectiveness RCTs, variation in the treatment being taken produces an additional level of complexity for data analysis, in that it precludes the direct comparison of randomisation groups from being equated with the safety of treatment A compared with treatment B |
The Salford Lung Studies in COPD and Asthma: What Were Their Novel “Real-World” Aspects?
Maximising External Validity
Maintaining Scientific Rigour
Safety Data Collection
SLS Approach to Effectiveness RCTS: Advantages and Limitations
How Might SLS Inform HCPS and Decision-Makers?
Learnings From SLS: How Can These Be Applied to Future RCTS?
SLS design aspect | Issue(s) | Solution(s) |
---|---|---|
Recruitment of GP investigators | Almost all GPs had not previously taken part in clinical trials GPs are busy and do not have space in their practices or the time to conduct clinical studies | Appointed GP ambassadors who recruited practices and “sold” the value of the study in their locality Recruited a team of approximately 50 research nurses to support GP investigators |
Patient recruitment was initially slow in the SLS | No previous experience of recruiting in this environment GPs very busy, limited time for recruitment | Personal contact from patient’s own GP critical for recruitment Adopted a project management approach to patient recruitment and consent, local advertising and pharmacy approaches to patients |
Medicines supply and GCP management, pharmacy involvement | Pharmacies are usually in competition for business Few pharmacists had prior clinical research experience Requirement for pharmacy study-specific SOPs | Created a pharmacy steering group to oversee training and SOP development, endorsement of pharmacy chain superintendent pharmacists [34] |
Study endpoints, analysis and powering | No prior data on which to base our power calculations, our statisticians had not previously dealt with studies like this Endpoints had to be of value but be measured with minimal interference to patient care | Considerable debate with the SLS Scientific Committee to decide on endpoints Numerous reviews of statistical plans and endpoints—very different to a DBRCT, where our confidence is higher |
Randomisation and stratification of patients by asthma severity | Since very few baseline investigations were performed, it was impossible to stratify according to lung function or usual measures of disease severity | Novel approaches to stratification were developed, such as the issuing of a “dummy prescription” by GP at baseline assessment, which allowed us to stratify according to intended treatment |
Electronic collection of pharmacy dispensing data | Pharmacy systems in the UK are primarily stock control and labelling systems, and many different systems are used | Bespoke solution created, which took an incredible amount of work |
Safety monitoring to GCP standards | This had not been done previously and as we were using EHR triggers to detect certain study endpoints and safety signals, we had to think completely differently to safety monitoring in a DBRCT | Worked with the sponsor’s pharmacovigilance team to build a robust safety system Had a consultant physician-led safety team (two physicians and four nurses) monitoring signals on a daily basis [19] |
Data quality and standards | Use of EHRs and effort required to ensure that data was of high enough standard to meet GCP requirements Sponsor’s experience from data governance was a need to take EHR to a higher level | Implemented a much higher than usual investment in data cleaning and data quality The EHR needed an additional programme to collect relevant (and delete irrelevant) data. Level of quality and governance not adequate to start with and required an audit → fix → audit approach |