Impact of an educational program and decision tool on choice of maternity hospital: the delivery decisions randomized clinical trial

We conducted this randomized controlled trial in 2019–2021 using the Ovia Fertility and Pregnancy mobile applications (apps). These apps, only available in English at the time of the study, offer a series of tools including articles customized to the patient and their interests (e.g., an article explaining the size of the fetus at the user’s stage of pregnancy) (Figure S1.1). These apps from Ovia Health predate the study and were available in most app stores. We conducted the study using the apps because of their large preexisting user base and because they could be easily modified to include new functionalities relevant to the study. To recruit participants, we presented a post in the “newsfeeds” of the app users who had been trying to conceive for fewer than five months or were 28–104 days into their pregnancies based on last menstrual periods. We included those still trying to conceive because we recognized that most of these people would soon become pregnant and sought to identify participants who had not yet chosen an obstetric provider. If a user expressed interest, they were immediately randomized 1:1 based on the last digit of their app user ID, which was automatically generated upon sign up for the app. Participants were excluded if they reported that they had already selected hospitals or obstetricians/midwives for their pregnancies because the intervention would likely have little impact in this population. We also excluded people if they were enrolled in two other Ovia programs (a health plan/employer program or a high-risk depression pregnancy intervention), as we did not want to interfere with those programs. We also excluded participants who did not report their last menstrual periods, had due dates outside the study period (April 2020-June 2021), reported ages outside of 18–49, or indicated they did not live in the United States. Trial recruitment and exclusions are outlined in the CONSORT diagram (Fig. 1).

Our intervention builds on the learnings of a prior pilot trial in which a similar intervention using informational tools increased participants’ familiarity with cesarean delivery rates but did not increase the likelihood of users selecting low-cesarean rate hospitals [21]. In the current trial, people in the intervention group were provided an interactive tool that showed hospital star ratings for the 10 closest hospitals within 50 miles of any zip code (star rating determinations explained below) as well as a series of educational modules on why it was important to select a hospital with a higher “labor-friendly” star rating. The interactive tool used a 5-star rating system instead of showing users the actual hospital cesarean rates because prior interviews suggested that many people had trouble translating a numerical rate into an actionable choice and prior research highlighted that simplifying quality data was key to increasing accessibility [22, 23]. We chose the term “labor-friendly” (paralleling the “baby friendly” designation) based on feedback from interviews that this phrase was appealing and accessible as well as evidence that hospitals with low cesarean section rates were more likely to provide additional support for labor (e.g., doulas) [24‐26]. The 8 educational modules provided to the intervention population answered questions such as “Why does it matter if a hospital is labor-friendly?” and “What labor-friendly hospital is right for you?” with videos and articles.

People randomized to the control group were also given access to a hospital look-up tool as well as educational modules. The interactive hospital look-up tool provided to the control population was similar to the intervention tool but did not display any star ratings. The 2 control educational modules explained the different criteria someone could use to select a hospital and encouraged readers to go on hospital tours. (Screenshots of educational materials and tools for both arms available in supplemental materials).

Investigator blinding was less relevant in this study because researchers did not have any personal interaction with participants during the trial and all outcomes were self-reported by participants.

We used two data sources for hospital cesarean rates. For five states (Alabama, California, Massachusetts, Vermont, West Virginia), we used publicly reported primary cesarean rate data for all hospitals published by the state governments [12‐16]. For all other states and D.C., which had no comparable public data, we used the self-reported nulliparous, term, singleton, vertex (NTSV) cesarean rates submitted by hospitals to the Leapfrog Group. Among known hospitals in these other states and D.C, 54.3% of hospitals did not submit data to Leapfrog. In Supplemental Materials, we detail dates of the data and differences in how the cesarean rate was measured based on data source (Table S1).

Within each state, we categorized hospitals into quintiles and assigned them star ratings based on their quintiles (e.g., 5 stars if the cesarean delivery rate was in the lowest quintile for the state). Hospitals for which cesarean rates were unavailable were marked “no data” and listed last on the interactive tool.

Our primary outcome was the star rating of the delivery hospital each participant selected using the interactive hospital selection tool during pregnancy (hospital choice during pregnancy) (Figure S1.5 how participants entered these data). Participants were shown posts in their newsfeeds in the apps each week from the start of their enrollment asking them to report their chosen hospitals through the tool until they submitted choices. They were offered the opportunity to be entered into a $100 lottery for reporting.

Our secondary outcomes were survey questions focused on the importance of cesarean rate data and whether these data were used in the participants’ decisions (Table 1 provides the wording of questions). Starting in the second month after entry in the trial, participants had posts in their feeds asking them to fill out this survey. Again, participants were entered in a lottery for $100 for responding. If they responded to the survey more than once, we used their later responses.

All users of the app (trial participants and non-participants) were asked to fill out a post-delivery survey that is sent to all app users regardless of whether they were part of this trial. Post-hoc, we added two secondary outcomes from this survey: hospital choice and self-reported delivery method (language of questions in supplemental materials). We linked each participant’s hospital choice to its star rating.

Limited demographic data were collected by the Ovia apps, including enrollees’ ages and zip codes of residence. Using each participant’s zip code, we linked our dataset with data from the American Community Survey on median annual household income and the education levels of women ages 18–45 to estimate the incomes and education levels of the participants [27, 28]. Additionally, we used participant zip codes to determine which participants lived in urban areas as defined by the United States Census Bureau [29].

We compared the demographic characteristics of participants in the trial to those of pregnant people nationally using data from the Census and CDC [30‐33]. We compared the star ratings of hospitals by conducting Welch’s t-tests, both for the primary outcome of choice during pregnancy and the secondary outcome of choice after pregnancy. We grouped Likert survey responses into binary categories and then used chi-squared tests to compare responses. We compared delivery methods between the control and intervention groups using a Pearson’s chi-squared test across all participants who indicated their delivery method as a post-hoc secondary outcome analysis.

In exploratory post-hoc subgroup analyses, we examined if the intervention’s impact was mediated by participant socioeconomic status (as measured by the median income in zip code), as we hypothesized that the impact of the intervention would be smaller among lower-income people given that other factors (e.g., insurance restrictions) may limit their hospital choices. We also conducted sub-group analyses to assess whether the intervention had a greater impact in communities with more choice of hospitals. Specifically, we stratified our analyses by whether the participant lived in a state that reported hospital cesarean rates and therefore fewer hospitals were missing star ratings, how many hospitals with a three star rating or higher were within 10 miles of the participant’s zip code, and whether there were hospitals with a difference in star ratings of at least two stars within 10 miles of the participant’s zip code. We calculated the star rating differences between the control and intervention arms in each subgroup. To determine if these differences were significant, we used linear regression models regressing on whether the participant was in the intervention or control, the sub-group (e.g., number of hospitals with a star rating of three or higher within 10 miles), and the interaction term between the two (variable of interest). All analyses were performed using R version 3.6.2.

The trial was approved by the Institutional Review Board of the Harvard T.H. Chan School of Public Health and registered at clinicaltrials.gov (Registration number NCT02987803, registered 09/12/2016). All participants consented through their agreements to the terms of use and privacy policy for the Ovia apps. A Data Safety Monitoring Board (DSMB) comprised of experts in the study content and statistical methods reviewed interim results in September 2020.

Among people enrolled in the trial, 10.2% (9.8% intervention, 10.5% control) reported their choices of hospitals during pregnancy. The average gestational age was 12 weeks at the time they provided this information. Among those who reported a hospital choice, 7.9% updated their hospital selection at least once. Compared to those who did not report this outcome, participants who reported this outcome lived in higher income communities (27.0% vs. 25.5% lived in zip codes with median incomes higher than $75,000) and were more likely to live in urban areas (84.1% vs. 77.6% in urban areas) (Table S3).

The average star ratings of the hospitals selected by people in the intervention population were significantly higher than in controls (average star rating 2.61 (SD 1.60) intervention vs. 2.24 (SD 1.44), p < 0.001) (Fig. 2A). Of the participants who reported their hospitals, 19.0% of the intervention group selected 5-star hospitals while 10.8% of the control group selected 5-star hospitals.

Among participants, 8,035 (6.7%; 6.6% intervention, 6.7% control) reported the hospitals in which they gave birth after their deliveries.

The average star rating of the hospitals where the intervention population gave birth was significantly higher than that of the control population (average star rating 2.30 (SD 1.42) intervention vs. 2.19 (SD 1.42) control, p = 0.001) (Fig. 2B). Of the participants who reported their hospitals, 11.5% of the intervention group selected 5-star hospitals while 9.2% of the control group selected 5-star hospitals.

Among the 1,681 (1.4%) of participants who reported their hospital choices both during pregnancy and after delivery, 60.5% and 54.5% reported the same hospitals at both instances for the control and intervention groups, respectively (Table S4).

People in the intervention group were more likely to believe that the choice of hospital impacts the likelihood of having a cesarean delivery (38.5% vs. 33.1%, p < 0.001, response rates of 6.9% and 6.7%). However, there were no differences in respondents’ beliefs that hospitals in their communities had different care quality levels (91.0% vs. 90.0%, p = 0.10, response rates of 7.3% and 7.2%) or that cesarean delivery rates are important to consider when choosing a hospital (76.2% vs. 74.3%, p = 0.050, response rates of 7.3% and 7.2%) (Table 1).

Among the participants randomized to the intervention, 50,611 (83.9%) did not open any educational modules offered as part of the intervention and 50,241 (83.4%) did not use the hospital-look up tool. Among those in the intervention group who reported a delivery hospital, 3,419 (57.8%) did not open any educational modules. There were no clear differences in the mean star ratings of hospitals chosen during pregnancy stratified by the number of educational modules opened (0 modules 2.62, 1 module 2.62, 2 modules 2.65, 3 + modules 2.54) (Table S5).

We conducted exploratory analyses on the differential impact of the intervention across several subgroups (Fig. 3). We hypothesized that participants with more resources and higher education would be more likely to respond to the intervention. Using median income of the zip code of residence as a proxy for resources and education, we did not observe any substantive difference in the impact of the intervention between participants who lived in higher and lower education zip codes (p = 0.85).

To further understand the influence of hospital choice, we examined the relationship between the number of hospitals with a 3-star or higher rating nearby and the impact of the intervention (0.31 difference in star ratings between intervention and controls among participants with no hospitals with a 3 star or higher rating within 10 miles, 0.36 for one hospital, 0.44 for more than one hospital, test of interaction p = 0.03). Relatedly, we compared participants who had a choice between hospitals with disparate star ratings (0.42 difference in star ratings between intervention and controls among participants where the maximum star rating is at least two stars greater than the minimum star rating among hospitals within 10 miles of the participant, 0.34 if this is not true, p = 0.25).

This randomized trial was motivated by the hypothesis that making cesarean delivery data more interpretable and accessible would encourage and enable pregnant people to use these data in the selection of hospitals to deliver their babies. Participants subject to the intervention were more likely to believe that cesarean rates were important in choosing a hospital and were more likely to select a hospital with a relatively low cesarean rate. However, there was no difference in participant likelihoods of having cesarean deliveries.

Our findings help inform the scientific literature on whether and how patients use publicly reported quality data to select a provider. With some notable exceptions in choices of nursing homes or health plans, prior reviews have highlighted that few patients are aware of publicly-reported quality data, and these data rarely impact provider selections [34‐37]. Our results support prior laboratory studies that patients are more likely to use quality data to inform their provider choices if the data are more interpretable (e.g., star ratings, simplified presentations, patient-friendly language) and if they understand how the information can impact their own care [23].

The findings inform ongoing efforts to publicly report cesarean rate data, which are motivated by the goal of empowering pregnant people to make more informed delivery hospital selections. Our findings argue that simplifying data presentation by using star ratings and more patient-friendly terminology increases usability and therefore the impact of these data. Because the intervention encompassed many different components, we do not know which component was most important.

It is also important to emphasize that despite the inclusion of the educational program in the intervention explaining that the choice of a low-cesarean rate hospital could reduce their personal risks of delivery by cesarean, most participants in the intervention arm still did not believe this to be true. This belief will be a substantial barrier to influencing people’s delivery hospital choices that public reporting efforts alone cannot solve.

Additionally, despite a shift in hospital choices, we did not observe any difference in self-reported cesarean rates. This does not support the assumption underlying our study, and the rationale for public reporting of cesarean section data by states and other groups in general, that if more people shift to lower cesarean-rate hospitals, the overall cesarean rates in the population will fall.

Although there was a difference in the star ratings of hospitals selected during pregnancy (on average chosen in the first trimester), this difference was only modest, and the difference in star ratings of hospitals actually used for delivery was smaller. This shift between hospital choice early in pregnancy and actual delivery might be attributable to constraints based on proximity of the hospital or insurance. Further, only a minority of those randomized to the intervention engaged with the educational modules or hospital look up tool. Future research is needed to understand how to increase engagement in hospital cesarean rate data and what are the key barriers in using this information.

This pragmatic clinical trial was able to recruit a large and diverse national study population of over 100,000 participants using a mobile app. The intervention included a simple, low-cost, innovative tool that gives pregnant people readily comprehensible and actionable information. The intervention can be implemented relatively easily.

However, our results should be interpreted in the context of several key limitations. First, response rates were low for all outcomes, particularly survey results, and we reported outcomes on only those who responded. Second, the hospital data we used to create star ratings had limitations: the Leapfrog Group data lacked ratings for many hospitals, and the data were several years old at the time of the trial. However, when we limited the population to those states with more complete data, the effect of the intervention was similar. Third, many participants had limited hospital choices and therefore could not act on the data provided in the intervention [39]. Many hospitals had no data, more than a third of participants did not live within ten miles of a rated hospital, and almost half of participants did not live in zip codes within 10 miles of a hospital with a 3-star rating or higher. Fourth, we had limited demographic data, so we are unsure of the representativeness of our study population across many categories, particularly race. The absence of race/ethnicity data is particularly notable given the substantial racial inequities in maternal health [40]. Fifth, we do not have clinical data to determine for which births in the study population a cesarean delivery was indicated. We also did not ask participants whether they electively chose to have cesarean sections, but this is quite rare (< 1%) and therefore unlikely to drive our findings [41]. The study was conducted in part during the COVID-19 pandemic, and it is unclear how the pandemic changed patients’ delivery plans [38]. Additionally, we recognize that patient hospital choice is driven by a myriad of other considerations, including logistics, distance, continuity of care, and access to specialized services [42]. In this context, hospital cesarean section rate data can only have a limited impact. Finally, the intervention combined both educational modules and a selection tool, and we do not know which of the components drove the differences observed.

In this randomized controlled trial of a large and diverse national study population, an intervention composed of educational modules, translations of hospital cesarean rates into patient-friendly star ratings, and an interactive tool to help select a delivery hospital increased the likelihood that people would plan to deliver at hospitals with lower cesarean rates. However, there was no change in participants’ likelihood of having a cesarean delivery.