Background
Methods
Description of the PROUD Trial
Background
PROUD Study design
PROUD effectiveness objective
PROUD study sample
Statistical analysis plan for the PROUD effectiveness objective
Design and analytic challenges related to identification bias
Research Question: Does the intervention decrease the number of days of acute care utilization among patients with OUD (patient-level outcome)? | |
Assumption: Assume the intervention has no effect on reducing acute care utilization Analytic sample: An open cohort of individuals with an OUD diagnosis documented in the EHR (pre- and/or post-randomization) Patients identified using pre-randomization data: • Suppose the number of patients with documented OUD pre-randomization is 100 in each trial arm (control and intervention) • Assume an average of 9 days of acute care utilization per year at baseline among these patients with OUD in each arm Patients identified using post-randomization data: • Control: 25 patients receive a new documented OUD diagnosis post randomization. These patients have an average of 9 days of acute care per year at baseline • Intervention: 50 patients receive a new documented OUD diagnosis. Of these, 25 are diagnosed as part of the intervention program and the other 25 are diagnosed through other mechanisms as in the control sites • Suppose patients diagnosed via the intervention program are sicker as compared to those diagnosed through other mechanisms, with an average of 12 days of acute care per year (versus 9) at baseline Estimated intervention effect: • Control: among 125 patients with a diagnosis, there is an average of 9 days of acute care per year of follow-upb • Intervention: among 150 patients with a diagnosis, there is an average of 9.5 days of acute care per year of follow-upb [= 9*125/150 + 12*25/150] Summary: We would estimate that the intervention results in greater acute care utilization relative to control, even if there is truly no effect. The bias could go in the other direction if patients diagnosed as part of the intervention program are healthier (rather than sicker) than patients diagnosed through other mechanisms. |
Framework to address identification bias in the statistical design
- Step 1. Identification of candidate analytic samples. Consider different options for the choice of analytic sample to be used for the primary analysis. Determine which options include patients identified using post-randomization data (and are therefore potentially affected by identification bias) and which rely solely on pre-randomization data (and therefore avoid identification bias).
- Step 2. Choice of unbiased analytic sample for the primary analysis. Conduct a power analysis to identify which of the unbiased, pre-randomization samples identified in Step 1 achieves the maximal power. Use this sample for the primary analysis. Details on the methods used in the PROUD case study are below.
- Step 3. Determine whether secondary analyses are needed to assess generalizability. If generalizability of results from using the analytic sample identified in Step 2 is of concern (e.g. for the reasons described above), alternate samples that may be affected by identification bias may be considered as secondary analyses.
- Step 4. Statistical analysis plan for secondary analyses in alternate analytic samples. Specify a statistical analysis plan for secondary analyses that include individuals identified after randomization that accounts for the potential for identification bias.
Power evaluation to select an unbiased analytic sample for the primary effectiveness analysis
Analytic sample | General considerations | Implications for effectiveness outcome No. days of acute care utilization (patient-level outcome) |
---|---|---|
Analytic samples not affected by identification biasa | ||
Sample 1. Patients with documented OUD before randomization | • Misses potential improvements in outcomes attributable to the intervention among new patients with OUD who were attracted to receive care after randomization due to the PROUD intervention. These patients could comprise a substantial proportion of patients with OUD treated due to the PROUD intervention (70%–90%) • Patients with a prior documented OUD may not reflect the general population of patients with true OUDb • Given that OUD is underdiagnosed, restricting to patients with documented OUD before randomization reduces the sample size and therefore power for patient-level outcomes, relative to an open cohort design that includes those diagnosed after randomization (Sample 4 below) | Estimates of intervention effects within this select population of patients with documented OUD before randomization may not generalize to the broader population of individuals with true OUD who may be treated as part of the intervention (and therefore may not detect the true benefit). |
Sample 2. All patients with primary care visits before randomization | • Same as bullet #1 for Sample 1 above • Sample includes patients with undocumented OUDs before randomization who might benefit from the intervention • Sample also includes many individuals without true OUD who would not be impacted by the intervention | • Since most individuals in the site population do not have OUD, the effect of the intervention on acute care utilization would be diluted, resulting in attenuation of the treatment effect toward the null • Relative to Sample 1, power could either be increased due to the higher sample size of patients with OUD or decreased due to including patients without true OUD in the analysis |
Sample 3. Patients with primary care visits before randomization who have documented OUD or are at “increased risk” of OUDc | • Same as bullet #1 for Sample 1 above • Need to develop a definition of “increased risk” of OUD that seeks to include as many patients who truly have OUD (maximizing sensitivity) while limiting the number of patients included who do not truly have OUD (maximizing specificity). For example, this definition could be selected to target a high specificity (Sample 3a), a high sensitivity (Sample 3b), or a balanced sensitivity and specificity option (Sample 3c) | • Relative to Sample 1, results in a larger sample size of patients with true OUD (higher sensitivity). This could increase power • At the same time could lead to attenuated intervention effect estimates relative to Sample 1, since more of the identified individuals would not have OUD (lower specificity) |
Analytic samples potentially affected by identification biasa | ||
Sample 4. Patients with documented OUD (before and/or after randomization) | • Patients diagnosed after randomization in the intervention arm may not be comparable to patients diagnosed after randomization in the control arm • Diagnosis of newly recognized OUD is expected to continue over time. Consequently, including individuals diagnosed after randomization could increase the sample size (and therefore power) as compared to Sample 1 | Individuals diagnosed with OUD after randomization in the intervention arm are likely to be different (either sicker or healthier) with respect to their propensity for acute care utilization than individuals diagnosed with OUD after randomization in the control arm. This could lead to bias (see Table 1). |
Sample 5. All patients with primary care visits (before and/or after randomization) | • Patients new to intervention sites after randomization may not be comparable to patients new to the control sites after randomization • As in Sample 2, sample includes many individuals without true OUD who would not be impacted by the intervention • Captures outcomes of all patients with OUD who could be treated: patients seen previously in the clinic (including those with and without documented OUD before randomization) and those attracted to receive care as part of the PROUD intervention | As in Sample 2, the effect of the intervention would be diluted in the entire site population relative to Samples 1 or 4 and power could either be increased or decreased. |
Sample 6. Patients with primary care visits who have documented OUD or are at “increased risk” of OUDc (before and/or after randomization) | • Patients identified as at “increased risk” of OUD after randomization in the intervention arm may not be comparable to patients identified as at “increased risk” of OUD after randomization in the control arm • As in Sample 3, need to develop a definition of “increased risk” of OUD that seeks to include as many patients who truly have OUD while limiting the proportion of patients who do not truly have OUD who are included | As in Sample 3, results in a larger sample size of patients with true OUD, but also could include many patients without OUD; thus, power could either be increased or decreased relative to Sample 4. |
True OUD prevalence (π) | ||||||
---|---|---|---|---|---|---|
0.01 | 0.02 | 0.04 | ||||
Samplea | Assumptions / Notes | Specificity | Sensitivityb | |||
1 | Documented OUD | • Assumes all individuals with documented OUD do in fact have true OUD (specificity = 1) • Sensitivity selected to be consistent with the observed proportion of patients with documented OUD based on Phase 1 data (0.5%) and the specific choice of the true prevalence (π) | 1 | 0.5 | 0.25 | 0.125 |
2 | All patients | By definition, sensitivity = 1 and specificity = 0 | 0 | 1 | 1 | 1 |
3a | High specificity | Selected to have slightly higher sensitivity than Sample 1 (1.2 times the value), at the cost of slightly reduced specificity | 0.95 | 0.6 | 0.3 | 0.15 |
3b | High sensitivity | • Sensitivity was selected based on a previously developed algorithmc to identify individuals with opioid abuse and addiction, among patients on long-term opioid therapy • We considered a lower specificity (0.5 vs 0.64c) given that our initial sample is the entire site population, not restricted to long-term opioid users | 0.5 | 0.85 | 0.85 | 0.85 |
3c | Equal sens./spec. | Selected to have lower sensitivity and higher specificity than Sample 3b | 0.6 | 0.6 | 0.6 | 0.6 |