Development of an unannounced standardized patient protocol to evaluate opioid use disorder treatment in pregnancy for American Indian and rural communities

Metrics were developed within the context of our study population—three rural Utah counties with predominantly white and/or AI populations. We followed a process of literature review, guideline review, and stakeholder engagement similar to other pregnancy-related research [37]. We then identified specific knowledge gaps resulting from lack of direct observation to generate testable hypotheses for these knowledge gaps and develop metrics to test each hypothesis.

“Patient profiles” are the standardized patient characteristics assigned to each USP. Our objective was to create patient profiles representative of white and AI pregnant women with OUD residing in rural Utah that would capture the data required for each outcome metric. We sought input from several sources, including prior studies, the health center network, and others with lived experience in rural Utah communities, AI communities, or both (see Fig. 1). Additionally, we engaged a research assistant of Navajo descent familiar with local cultural practices and knowledgeable about common social determinants of health, insurance status, and other contextual factors within the Navajo community. Given that other research team members had extensive experience treating pregnant patients with OUD of all races/ethnicities who reside in the sampling area, the team’s collective experience allowed for a basic understanding of environmental and cultural considerations in creating the USP profiles.

We created two identical patient profiles that differed only by patient race (white or AI) and race-related context (e.g., name, place of previous residence), shown in Table 1. Callers would represent either the patient or the patient’s male partner for the white profile, but only the patient for the AI (Navajo) profile. The health center network provided additional insights about developing AI/non-AI profiles as well. For example, a Navajo individual is likely to relocate from reservation lands, while a white individual is likely to relocate from non-reservation lands. Similarly, Navajo communities tend to follow a matriarchal order, so while a male partner calling on behalf of the patient would be appropriate for a white couple, it would be highly unusual for a Navajo couple.

After discussions and consensus among team members, we identified several patient profile characteristics to enhance the quality of our evaluation. First, we determined buprenorphine continuation, rather than initiation, would be more likely accepted by buprenorphine prescribers and capture the highest possible number willing to treat OUD in pregnancy. Second, in our literature review, we identified barriers and facilitators to seeking care, such as maternal concern for loss of child custody [81]; therefore, we incorporated these characteristics and other established risk factors to increase urgency of the request and evaluate whether these characteristics are recognized/addressed by clinical staff. Third, we limited the scope of extraneous medical issues to focus only on measures of interest. Only comorbidities that could potentially modify OUD and prenatal treatment, such as depression or other substance use, were included. A comprehensive description of USP profiles and script prompts are available in Additional file 1: Exhibit S1.

Pregnant individuals with OUD interface the healthcare system as new patients primarily through two encounters: scheduling an appointment and meeting with a provider during an initial visit. Because each may affect delivery of care, we designed two interrelated USP protocols to account for both. Together, these two protocols capture all metrics required for the six access and quality outcomes. The telephone appointment request was deemed as not human subjects research by the University of Utah IRB; the face-to-face visit was a quality improvement project and not subject to IRB review.

Clinical sites delivering primary care and obstetric care in three rural counties were identified through a commercially available database of provider listings by specialty (IQVIA), combined with verification of database information through internet search and consultation with stakeholder confederates. A total of 18 clinical sites meeting criteria were identified. See Additional file 1: Table S3.

We developed a coding protocol for telephone appointment requests and face-to-face visits using input from team members and prior USP studies [82]. Because outcome measures included both objective and subjective data, objective findings were captured through review of audio recordings using a post-visit data entry form, and subjective data were captured through a post-visit questionnaire completed by each USP immediately following the encounter. See Tables 2 and 3. We selected REDCap to tabulate and export data for analysis [83, 84] and later manual coding, review, and descriptive statistics. The overall objective in creating our analysis plan was to measure how often each objective performance metric is met and describe the degree of alignment between provider performance and societal guidelines for subjective performance metrics.

With the exception of disparities between white and AI USP encounters, all other metrics in our study are descriptive in nature. Therefore, for qualitative aspects of our study, our sample size of all 18 sites in the sample area is sufficient to achieve thematic saturation [85]. Disparities in outcomes are tested using two sample, two-tailed t-tests. Determining a sample size to appropriately power our study and identify disparities (if present) is difficult for two reasons. First, unlike most clinical trials, the variance of primary and secondary outcomes is rarely known (and is not known for outcomes in this study). Second, defining a clinically meaningful difference in many outcomes (e.g., appointment wait times) can be subjective. Because relatively few clinical sites exist within the rural region of our study, we chose to include all 18 sites (saturated sample). However, by making a few pragmatic assumptions, the reasonableness of the sample size can be estimated. If, for example, a difference of 7 days were present between the two groups with a mean wait time for the reference group and a standard deviation of 0–14 days and 7 days, respectively, 16 independently sampled sites would be sufficient for a power level of 0.8. Similarly, if a relative disparity of 20 percent—one likely to be clinically meaningful—were present for referral to treatment (e.g., 40 percent for AI compared to 50 percent for white), with a standard deviation of 16 percent, 11 sites would need to be independently sampled for each profile to achieve a power level of 0.8. Our sample size exceeds the minimum requirement based on these assumptions.

The telephone appointment request protocol was tested through six pilot calls made to clinics in or near the sampling area. Callers received two separate training sessions facilitated by USP experts on the research team. These calls were audited and reviewed by investigators of the research to team to establish protocol face validity. Additionally, we used the calls to align with regional clinical practices and correct logic in post-visit data entry forms. For example, USPs were sometimes transferred immediately from a front desk scheduler to a medical assistant or nurse without an opportunity to request an appointment. These adjustments were made after each call until intended metrics could be reliably assessed.

A second purpose of the pilot call period was to verify USP fidelity. Fidelity checks assure protocol delivery across different USPs is consistent and reliable [31]. Fidelity checks in this study are especially important to ensure reliability between male and female USPs. As a matter of practice, we perform the same fidelity check on the entire study sample after data collection to ensure a consistent result. Our fidelity check used (1) a qualitative approach to compare USP performance to the written protocol by auditing the calls and providing feedback, and (2) a quantitative approach to assess for statistical differences between responses of randomly assigned clinic sites for each USP. A summary of process evaluations and outcomes for our method development is presented in Table 4. As shown, adaptation of the USP method to this clinical context relied on expertise specific to the patient population and medical condition, as well as standard processes to assure quality and fidelity of data collection.

The novel coronavirus SARS-CoV-2 (COVID-19) pandemic emerged during the development of our USP protocol. We accounted for these restrictions in our telephone appointment requests by (1) defining COVID-19 as a distinct and separate barrier to care when requesting an appointment and asking what clinics would do “under normal operating conditions” if care was refused due to COVID-19; and (2) allowing for virtual visits to be scheduled instead of face-to-face visits if necessary. For face-to-face encounters, we continue to work with our stakeholders to assure safe timing.

Following completion of data collection and analysis, findings will be reported back to clinical partners to begin a process of quality improvement. Subsequent re-evaluation will then be used to assess changes in quality over time. We will further use this approach to adapt the USP protocol for evaluation in other communities.

OUD in pregnancy remains a critical concern in the US, especially among vulnerable populations residing in rural communities. We have described a novel USP protocol to assess healthcare access and quality for pregnant, rural-dwelling white and AI individuals that cannot be measured through administrative data. While much work remains to improve care for OUD in pregnancy and reduce disparities, this protocol represents a step toward gaining new insights and may serve as a roadmap for future healthcare access and quality research within the context of substance use disorders and pregnancy.