Background
NIH Pragmatic Trials Collaborative Project
Trial name | PI/affiliation/sponsor | Trial title | Significance | Setting/target population | Recruitment strategy | Intervention | Primary and (secondary) outcomes | Design/analysis (sample size) |
---|---|---|---|---|---|---|---|---|
ENGAGES | M. AVIDAN, MD (Washington University)/NIA | Electroencephalograph Guidance of Anesthesia to Alleviate Geriatric Syndromes | Reduce postoperative delirium associated w/cognitive impairment and falls | Hospital/elective surgery patients age 60 + years | Drawn from patients enrolled in SATISFY-SOS study (consent obtained by RA in pre-op clinic) | EEG-guided anesthesia vs. usual care | Postoperative delirium (patient-reported health-related quality of life; postoperative falls) | Block randomization (patients) Intent-to-treat (N = 1232) |
HUSH | D. BUYSSE, MD (University of Pittsburgh)/NHLBI | Pragmatic Trial of Behavioral Interventions for Insomnia in Hypertensive Patients | Reduce insomnia disorder using non-drug treatment in primary care | Primary care/adult patients w/HTN, hypnotic medication, or insomnia diagnosis | PCP referral via Research Recruitment Alert (RRA); telephone screen; electronic consent | Two CBT interventions (one online) vs. usual care | Self-reported sleep; health indicators (symptoms, health, and patient/provider satisfaction; sleep, depression, anxiety, fatigue) | Stratified block randomization (age and sex) Intent-to-treat (N = 625) |
PART | H. WANG, MD (University of Alabama at Birmingham)/NHLBI | Pragmatic Trial of Airway Management in Out-of-Hospital Cardiac Arrest | Identification of best approach for out-of-hospital cardio-pulmonary arrest | Community-emergency/non-trauma cardiac arrest – adult patients | N/A | Endotracheal intubation and supraglottic airways approaches | 72-hour hospital survival (return of spontaneous circulation, airway management performance, clinical adverse events) | Cluster-crossover (randomization at EMS level – no consent) Intent-to-treat (N = 2612) |
PROOF Check | M. GONG, MD; O. GAJIC, MD (Albert Einstein College of Medicine of Yeshiva University)/NHLBI | Prevention of Severe Acute Respiratory Failure in Patients w/PROOFCheck | Prevent acute respiratory failure leading to organ failure | Hospital/all at-risk patients | High-risk patients identified by APPROVE (EMR-based) | Clinician notification of high-risk and PROOFCheck (bundle of care practices) vs. usual care | Hospital mortality (organ failure, ventilator-free days, 6- and 12-month mortality, ICU and hospital length of stay, ability to return home on discharge) | Stepped-wedge, cluster randomized (hospital level – no consent) Intent-to-treat (N = 7778 minimum) |
REDAPS | S. HALPERN, MD (University of Pennsylvania)/NIA | Default Palliative Care Consultation for Seriously Ill Hospitalized Patients | Determine effectiveness and cost of inpatient palliative care consult services | Hospital (w/integrated EHR) Patients ≥ 65 years w/end stage renal disease, advanced COPD, or advanced dementia | Intake assessment (nurse); EHR algorithm generates default palliative care order | Opt-out default for palliative care services vs. usual care (opt-in) | Composite measure hospital mortality and length of stay (hospital and ICU mortality; pain, transfer to ICU and CPR after randomization; days of mechanical ventilation; discharge disposition; 30-day hospital readmission; total hospital costs) | Stepped-wedge, cluster randomized (waiver of informed consent) Intent-to-treat (N ≥ 23,000) |
Objectives
The PRECIS-2 tool
Domain | Assessment considerations |
---|---|
Eligibility | To what extent are the participants in the trial similar to patients who would receive this intervention if it was part of usual care? For example, score 5 for very pragmatic criteria essentially identical to those in usual care; score 1 for a very explanatory approach with lots of exclusions (e.g., those who do not comply, respond to treatment, or are not at high risk for primary outcome, are children or elderly), or uses many selection tests not used in usual care |
Recruitment Path | How much extra effort is made to recruit participants over and above what would be used in the usual care setting to engage with patients? For example, score 5 for very pragmatic recruitment through usual appointments or clinic; score 1 for a very explanatory approach with targeted invitation letters, advertising in newspapers, radio plus incentives and other routes that would not be used in usual care. |
Setting | How different are the settings of the trial from the usual care setting? For example, score 5 for a very pragmatic choice using identical settings to usual care; score 1, for a very explanatory approach with only a single center, or only specialized trial or academic centersb
|
Organization | How different are the resources, provider expertise, and organization of care delivery in the intervention group of the trial from those available in usual care? For example, score 5 for a very pragmatic choice that uses identical organization to usual care; score 1 for a very explanatory approach if the trial increases staff levels, gives additional training, require more than usual experience or certification and increase resources |
Flexibility in Delivery | How different is the flexibility in how the intervention is delivered from the flexibility anticipated in usual care? For example, score 5 for a very pragmatic choice with identical flexibility to usual care; score 1 for a very explanatory approach if there is a strict protocol, monitoring and measures to improve compliance, with specific advice on allowed co-interventions and complications |
Flexibility in Adherence | How different is the flexibility in how participants are monitored and encouraged to adhere to the intervention from the flexibility anticipated in usual care? For example, score 5 for a very pragmatic choice involving no more than usual encouragement to adhere to the intervention; score 1 for a very explanatory approach that involves exclusion based on adherence, and measures to improve adherence if found wanting. In some trials e.g., surgical trials where patients are being operated on or intensive care unit trials where patients are being given intravenously administered drug therapy, this domain is not applicable as there is no compliance issue after consent has been given, so this score should be left blank |
Follow-up | How different is the intensity of measurement and the follow-up of participants in the trial from the typical follow-up in usual care? For example, score 5 for a very pragmatic approach with no more than usual follow-up; score 1 for a very explanatory approach with more frequent, longer visits, unscheduled visits triggered by primary outcome event or intervening event, and more extensive data collection |
Primary Outcome | To what extent is the primary outcome of the trial directly relevant to participants? For example, score 5 for a very pragmatic choice where the outcome is of obvious importance to participants; score 1 for a very explanatory approach using a surrogate, physiological outcome, central adjudication or use assessment expertise that is not available in usual care, or the outcome is measured at an earlier time than in usual care |
Primary Analysis | To what extent are all data included in the analysis of the primary outcome? For example, score 5 for a very pragmatic approach using intention-to-treat with all available data; score 1 for a very explanatory analysis that excludes ineligible post-randomization participants, includes only completers or those following the treatment protocol |
Methods
Study participants
Results
PRECIS-2 ratings
Qualitative findings
Change in trial design
Shift in rating
Pragmatic trial themes
Domain (N) | Interpretation and PI responses | Domain clarification |
---|---|---|
Eligibility | 1. Less pragmatic due to additional effort needed to identify appropriate patients and to validate correct identification: “It’s a little more work to figure out patients who are chronically vented who are excluded…I just thought it was going to be very, very easy and you don’t have to think about it. But it turns out actually I have to have my staff validate it.” (PI) | Eligibility refers to the extent to which the trial population matches the population intended for the intervention. The issue of effort to engage participants is more relevant to Recruitment Path, which addresses whether effort to recruit participants is greater than for patient engagement in usual care |
2. Less pragmatic due to a higher proportion of patients excluded than originally anticipated: “Once we started applying the criteria, we recognized…there are some people who we’ve excluded and I think they’re for good reasons… we haven’t changed the criteria, it’s just that as we’ve been applying them, we realized that it excludes a larger percentage of people perhaps than we thought.” (PI) | For the Eligibility domain one should consider the extent to which trial participants are similar to those who would receive the intervention if it were part of usual care (rather than volume of participants excluded) | |
Setting | 3. Setting is similar to usual care. “The setting is really the identical setting to usual care. But I probably scored it a bit down [at T1] it’s very representative of a usual setting.” (PI) | Setting receives a more explanatory score if there is only a single center, or only a specialized trial or academic center. Multi-center trials can be rated 3–5 |
Flexibility-Delivery | 4. More pragmatic because clinician notification (re: eligibility) was more automated than anticipated. “When we were in the planning phase…not clear how we were going to notify the [___]. It turns out the hospital itself had an outside vendor trying to figure out actually how to link …that [mechanism] is part of the hospital infrastructure now.” (PI) | Resource requirements are addressed under the Organization; the issue of resources required to conduct the study is not relevant to intervention delivery or adherence |
Flexibility-Adherence | 5. More pragmatic because notification (re: patient eligibility) was more automated than anticipateda: “When we were in the planning phase, [it wasn’t] clear to us exactly how we were going to notify the ___ – ” (PI) | Resource requirements are addressed under the Organization (see above). This domain should not have been rated as there is no monitoring of patient adherence |
6. More pragmatic because no participants are excluded due to adherence: “We’re not excluding anybody based on adherence, but we are encouraging adherence and are providing feedback on adherence.” (PI) | This domain addresses how flexibly participants in the trial are monitored and encouraged compared to usual care. This domain is not applicable to 2 of the trials as there is no compliance issue after consent has been given. The domain should be left blank (unrated) | |
7. More pragmatic because the intervention is executed in emergency care and adherence is minimal: “Our intervention really is executed and then it’s done, so the adherence of it is actually very minimal and the remainder of care given thereafter is just standard of care.” (PI) | ||
8. Less pragmatic because there is no usual care comparison: “There’s no way to know what would happen in usual care because the intervention’s never been tried in usual care. But I would foresee if our results prove favorable that the implementation in the real world would be identical to what we’re testing.” (PI) | The issue of usual care comparison is relevant to Flexibility of Intervention Delivery rather than Adherence | |
Follow-up | 9. Less pragmatic as collection of follow-up requires more effort than anticipated: “In clinical care, one would not necessarily seek out follow-up on patients,…what made us think that it was less pragmatic was the manner by which you seek out that information.” (PI) | Does not apply to this domain, which is concerned only with burden of follow-up on the participants, not whether the follow-up data are routinely collected |
10. Less pragmatic as collection of follow-up is less automatic than anticipated: “I have to apply in a separate IRB to a statistics department to get that long term follow-up. And that requires linking of the patient’s data. So that’s just a little less automatic…more work for me. For patients it’s the same.” (PI) | This domain is concerned only with burden of follow-up on the participants, not burden on research team or effort needed to collect the follow-up data | |
Primary Analysis | 11. Less pragmatic because the primary outcomes are not a standard measure: “The analysis is a standard analysis one would do for this type of a trial, but [not] a standard comparison that one would make on a daily basis.” (PI) | Pragmatism of primary analysis is based only on the degree to which all data are included in the analysis of the primary outcome |
Reflections on the tool
Rating scale | Would have been useful in design phase (N) | Highlighted areas important for trial to achieve goals (N) |
---|---|---|
Strongly agree | 2 | 1 |
Somewhat agree | 3 | 2 |
Neither agree or disagree | 0 | 1 |
Somewhat disagree | 0 | 1 |
Strongly disagree | 0 | 0 |
Discussion
Eligibility
-
For example, in the PART trial [32] testing airway management for resuscitation from out-of-hospital cardiopulmonary arrest (OHCA) endotracheal intubation (ETI) supraglottic airways (SGA), participants included were all adults who needed airway management following cardiac arrest. Exclusions were vulnerable populations who had “Do Not Resuscitate” orders, traumatic cardiac arrest, and children, which are routine exclusions and, therefore, very pragmatic. The only group that would be treated in usual care that were excluded were pregnant women and prisoners but these are usual Institutional Review Board (IRB) exclusions as protected groups.
Organization
-
For example, in the HUSH trial [31], cognitive behavioral therapy (CBT) delivery for insomnia was being tested in primary care using three methods requiring different resources. Thus, the Organization domain should have been rated separately for the two interventions being compared to usual care. One (Sleep Healthy Using the Internet) included a self-guided Internet version of CBT and the second (Brief Behavioral Treatment of Insomnia) involved a provider who would likely need additional training.
Flexibility of Intervention Adherence
-
In the ENGAGES trial [30], testing if an electroencephalography-guided protocol in elderly patients undergoing major elective surgery decreases the incidence of postoperative delirium, it could be argued that this domain was not applicable as the patients had given consent to the operation and being part of the trial. The official guidance [9] in the PRECIS-2 toolkit is that the domain should not be rated, as follows, “In some trials, e.g., surgical trials where patients are being operated on or intensive care unit trials where patients are being given intravenously administered drug therapy, this domain is not applicable as there is no compliance issue after consent has been given, so this score should be left blank.”
-
Similarly, in the PART trial [32] this domain is also not applicable as there was no compliance from the patient who either got ETI or SGA airway management resuscitation for OHCA.
-
For the PROOFCheck trial [33] the domain is also not applicable as patients were not involved in compliance; confusion may have occurred because adherence for physicians (the interventionist) was tested on use of checklists to determine which patients required mechanical ventilation to prevent severe acute respiratory failure (ARF). This domain is relevant for the HUSH trial [31]. For patients in the SHUTi arm of the trial adherence was encouraged using automated emails only, whereas in the Brief Behavioral Treatment of Insomnia (BBTI) arm the provider reviews progress with the participant for 15–30 minutes each week for 3 weeks to adjust sleep/wake times, which might be related to usual encouragement from a doctor. Due to the different ways of encouraging adherence, however, each of these interventions may be rated differently by trialists. Further clarification may be needed to assist trialists to score this domain accurately. In the REDAPS trial [34], however, with an intervention to test out the default option for palliative care consultation, the PI anticipated that the intervention could be fully pragmatic “5” but had marked down to “4” due to uncertainty implementing into usual care.