MomsTalkShots, tailored educational app, improves vaccine attitudes: a randomized controlled trial

The MomsTalkShots app (described in more detail elsewhere) [49] was designed as a website to be accessible via multiple internet browsers on smartphones, tablets, and computers. The app begins with registration, then administers a survey, then immediately after survey completion provides educational videos that are algorithmically responsive to its users’ vaccine intentions, KABs, and demographics [49]. The videos incorporate introductions and conclusions from obstetricians and pediatricians of different races/ethnicities with narrated animation to communicate messages in an interesting and engaging manner. The videos were designed based on approaches shown to be effective in training healthcare providers to improve their vaccine discussions with patients: for patients already intending to vaccinate, taking a presumptive approach; and for patients with concerns, establishing empathy, then carefully addressing the concerns within the context of the risk of disease, the benefits of vaccination, and the ability to protect through vaccinating [50‐54]. Videos covered both maternal and infant vaccination, though videos provided to pregnant women focused more on the former and videos provided to new mothers focused more on the latter. Videos were also available in a gallery for rewatching upon logging in between survey timepoints. A feature allowing pregnant women to refer the app to their close contacts was evaluated elsewhere and was shown to increase influenza vaccine uptake among family and friends to cocoon the infant [55, 56].

MomsTalkShots was the patient-level component of a multi-level intervention to improve maternal and infant vaccine uptake that also included provider- and practice-level interventions, described in more detail elsewhere [47, 48]. The impact of these interventions was studied using a two-by-two factorial design, randomizing at both the patient- and the practice-level (provider-level interventions were implemented and analysed in tandem with practice-level interventions). The practice-level intervention implemented an adaptation of the Centers for Disease Control and Prevention (CDC) Assessment, Feedback, Incentives and Exchange (AFIX) model, [57] and the provider-level interventions included provision of the Continuing Medical Education (CME) module VaxChat [58] and the book The Clinician’s Vaccine Safety Resource Guide [50]. The target sample size for the overall trial of 1896 was calculated using PASS 11 (NCSS, LLC, Kaysville UT), assuming a cluster randomized design with intra-class correlation coefficient of 0.01 (based on preliminary data from a previous pilot study), [59] to provide a power of 80% (α = 0.05) to see a doubling in maternal Tdap vaccinations (not for the purpose of this analysis) [48].

Study staff recruited pregnant women from waiting rooms of a geographically and socio-demographically diverse set of prenatal care practices in Colorado and Georgia between June 2017 and July 2018 and followed them through March 2019. Practices were chosen to capture diversity in patient demographics, urbanicity, provider types, and practice size. Eleven of the 22 participating practices were assigned to receive the practice- and provider-level interventions (via covariate constrained cluster randomization) [60]. Patient eligibility criteria included: between 18 and 50 years of age, English-speaking, and gestational age of 8–26 weeks (since Tdap is recommended at 27–36 weeks gestation) [61]. Upon enrollment, study staff lent participants an electronic tablet to complete the baseline survey via the app while in the waiting room. Two follow-up surveys were made available to all participants via the app approximately 1 month and 1 year after their infant’s birth, respectively, to complete using their device of choice at home. A $20 incentive was provided for completion of each survey. Further detail and data on recruitment is reported elsewhere [48].

All participants received three surveys, but only the half randomly assigned to receive the patient-level intervention subsequently received the tailored educational videos (MomsTalkShots) immediately after the baseline and initial follow-up surveys. We used SAS PROC PLAN to generate patient-level randomization schedules for each of the 22 participating practices, using block randomization with a block size of 8; these randomization sequences were then programmed into the app such that participants were randomized upon registration, blinded to study investigators.

Surveys used multiple choice questions to assess vaccine intentions, and Likert scale statements to assess latent vaccine KAB constructs, including: confidence in vaccine safety and efficacy, perceived susceptibility to and severity of vaccine-preventable diseases, self-efficacy (one’s belief in their ability to execute behaviors necessary to reach specific goals), [62] descriptive (what people do) and injunctive (what people approve) norms, [63] perceived vaccine knowledge, and trust in sources of vaccine information. These constructs were chosen after review of relevant behavioral models and scales, [8, 64] and multiple survey items were dedicated to each construct. Likert scale response options included: strongly agree, agree, disagree, and strongly disagree; knowledge and trust statements also included “don’t know”; and trust statements regarding pediatricians and naturopathic/chiropractic doctors also included “I don’t have a pediatrician yet” and “I don’t see this type of doctor,” respectively. Specific vaccine safety concern statements were administered only to those who expressed a lack of confidence in vaccine safety (to avoid creating new concerns).

Likert scale responses to survey items were encoded as follows: 1 - strongly disagree; 2 - disagree; 3 - don’t know; 4 - agree; 5 - strongly agree. Those who agreed (or strongly agreed) with survey items expressing confidence in vaccine safety were encoded to have disagreed (or strongly disagreed) with survey items expressing specific vaccine safety concerns. Those who had not (yet) seen pediatricians and/or naturopathic/chiropractic doctors were treated as missing for these statements.

Summary scores for constructs combined the scores from their constituent statements. Only constructs with multiple statements assessed at two or more timepoints were analysed. All scores were standardized by dividing into the maximum score then multiplying by 100. This standardization allowed comparison of construct summary scores with different numbers of contributing statements.

Dichotomous variables were encoded for each of the survey items as follows: 1 - agree or strongly agree; 0 - disagree, strongly disagree, or don’t know. Dichotomous construct summary scores were created by calculating the mean of each scaled continuous construct summary score and encoding as above the mean (“high”) or below the mean (“low”).

Analysis was guided by the statistical analysis plan documented in the study protocol and recorded on clinicaltrials.​gov. Multilevel mixed-effects linear regressions were performed for scaled statement and construct scores, with a random intercept for the clinic. Interim and final time point scores were each regressed on their corresponding baseline score, to control for potential differences in baseline scores between groups. Dummy variables were initially included for MomsTalkShots, the practice/provider intervention, and their product (to test for potential interaction). However, the interaction term was found not to be statistically significant, so the model was reparametrized without a dummy variable for interaction. Regressions were stratified by baseline intention to vaccinate. Dichotomous/logistic analysis otherwise emulating the above methodology was performed to increase interpretability. Statistical significance was considered p < 0.05.

Per protocol, because our surveys included so many individual items, we focused our analysis on constructs, to reduce redundancy and the chances of type 1 error. However, individual survey statements were also analysed, to provide additional insight into the individual components driving changes in constructs. The analyses of individual survey statements are reported as appendices.

Since the practice/provider intervention did not appear impactful, and the impact of MomsTalkShots did not significantly differ between intervention and non-intervention practices, we focused on MomsTalkShots regardless of the practice/provider intervention.

The results of the dichotomous analysis (Additional file 1: Appendices 1–2) were highly consistent with the results of the continuous analysis (Additional file 1: Appendices 3–4), aside from small differences in significance due to loss of power, so we present the results of the dichotomous analysis for greater interpretability.

Of the 3904 pregnant women found to be eligible after screening, 1391 declined to participate, and 303 did not finish enrollment. Ultimately 2087 enrolled pregnant women were randomized and provided enough data to contribute to this analysis. Informed consent was obtained from all participants.

Nearly three quarters of the participating pregnant women (n = 1524; 73%) completed the initial follow-up survey (at least 1 month after their infant’s birth), and over half (n = 1117; 54%) completed the final follow-up survey (at least 1 year after their infant’s birth) (Additional file 1: Appendix 7). Follow-up survey completion did not differ by intervention group, but was consistently lower among women not intending to vaccinate at baseline (Additional file 1: Appendix 8).

Half of participating pregnant women were from Georgia, and half from Colorado. Nearly half (46%) were pregnant for the first time. About 85% provided their race/ethnicity and 83% their education: of these, 64% were White, 16% were Black and 11% were Hispanic, and 72% had at least an undergraduate degree, respectively. Sociodemographic characteristics were similar between study arms, [48] and are described in more detail elsewhere [46, 48, 65].

Over half (56%) of women intended to receive both influenza and Tdap vaccines during pregnancy, 16% intended to receive one but not the other, 14% intended to receive neither, and 13% were unsure; 81% intended for their baby to receive all recommended vaccines, 11% intended for their baby to receive some or no recommended vaccines, and 8% were unsure (Table 1). Baseline vaccine intentions were similar between study arms, [48] and are described in more detail elsewhere [46, 48, 65].

Herein we report statistically significant results among all participants and stratified by vaccine intent.

One month after their infant’s birth, 61% of MomsTalkShots recipients had high perceived risk of maternal influenza disease, compared to 55% of controls (Odds Ratio: 1.61, 95% Confidence Interval: 1.23–2.09); 73% had high confidence in maternal influenza vaccine efficacy, compared to 63% of controls (OR: 1.97, 95%CI: 1.47–2.65); and 55% had high perceived vaccine knowledge, compared to 48% of controls (OR: 1.39, 95%CI: 1.13–1.72) (Table 2). One year after their infant’s birth, 62% had high perceived vaccine knowledge, compared to 50% of controls (OR: 1.74, 95%CI: 1.36–2.24); and 64% had high trust in vaccine information from obstetricians and pediatricians, compared to 55% of controls (OR: 1.53, 95%CI: 1.17–2.00) (Table 3).

One month after their infant’s birth, 71% of MomsTalkShots recipients intending to vaccinate at baseline had high perceived risk of maternal influenza disease, compared to 64% of controls (OR: 1.57, 95%CI: 1.12–2.20); 86% had high confidence in maternal influenza vaccine efficacy, compared to 77% of controls (OR: 2.09, 95%CI: 1.36–3.22); and 59% had high perceived vaccine knowledge, compared to 53% of controls (OR: 1.36, 95%CI: 1.08–1.70) (Table 2). One year after their infant’s birth, 65% had high perceived vaccine knowledge, compared to 55% of controls (OR: 1.60, 95%CI: 1.22–2.10); and 70% had high trust in vaccine information from obstetricians and pediatricians, compared to 63% of controls (OR: 1.47, 95%CI: 1.09–1.97) (Table 3).

One month after their infant’s birth, 38% of MomsTalkShots recipients intending not to vaccinate at baseline had high perceived risk of maternal influenza disease, compared to 32% of controls (OR: 2.07, 95%CI: 1.15–3.71); and 44% had high confidence in maternal influenza vaccine efficacy, compared to 28% of controls (OR: 2.62, 95%CI: 1.46–4.69) (Table 2).

One year after their infant’s birth, 47% of MomsTalkShots recipients with uncertain vaccine intentions at baseline had high perceived vaccine knowledge, compared to 12% of controls (OR: 6.89, 95%CI: 1.52–31.25); and 71% had high specific safety concerns for infant vaccines, compared to 91% of controls (OR: 0.24, 95%CI: 0.06–0.98) (Table 3).

MomsTalkShots decreased perceived risk of infant pertussis: 1 month after their infant’s birth, 39% of MomsTalkShots recipients had high perceived risk of infant pertussis, compared to 42% of controls (OR: 0.75, 95%CI: 0.57–1.00) (Table 2). While MomsTalkShots decreased perceived susceptibility; it also increased perceived severity (Additional file 1: Appendix 1), and the reduction in perceived susceptibility was limited to those who received Tdap during pregnancy (Additional file 1: Appendix 5).

Our study design included randomization at both the practice/provider and the patient levels. We hypothesized that both interventions might impact KAB constructs, and there might be an improved impact when combined. However, only the patient level intervention (MomsTalkShots) appeared to have an impact; the number of construct associations found with the practice/provider intervention was less than expected by chance alone and the interaction between practice/provider and patient interventions was non-significant. So we focused our per protocol analysis on constructs in the main analysis to reduce redundancy and the chances of type 1 error. There were many insignificant associations that we did not explicitly comment on in this manuscript; they may indicate MomsTalkShots did not impact certain vaccine perceptions, especially among those not intending to vaccinate at baseline. Although we analysed each KAB construct both as a continuous score and a dichotomous indicator of an above-average score, we focused exclusively on the dichotomous analysis in the Results and corresponding tables, despite continuous measures typically providing greater power and precision than dichotomous measures derived from them. However, we found the dichotomous analyses to be much more interpretable and thus more useful for the main text. We have presented both analyses fully in the Appendices for transparency; aside from small differences in significance due to loss of power, the results of the dichotomous analysis (Additional file 1: Appendices 1–2) were highly consistent with the results of the continuous analysis (Additional file 1: Appendices 3–4), justifying our approach. MomsTalkShots’ decrease in perceived risk of infant pertussis being driven by women who received Tdap during pregnancy illustrates the potential effect modification of vaccination, which was not included in our analytic approach beyond further exploration of an unexpected result. Small numbers of women intending their children to receive no recommended vaccines did not provide enough power to justify stratifying by this group alone, so it was combined with those who intended their children to receive some (but not all) recommended vaccines. Although this made our stratification less precise, it increased our power and simplified our analysis. Even with this combination, power was still limited, leading to many potential associations that were close to but not quite statistically significant at the prespecified p < 0.05 cutoff. This analysis focused on pregnant women and mothers and thus did not account for others who may be heavily involved in vaccine decisions for children, though data on partners, family and close friends of these pregnant women are published elsewhere [55, 56]. Our sample was comprised mostly of highly educated white women, despite efforts to recruit from a geographically and socio-demographically diverse set of prenatal care practices. Finally, loss to follow-up with differential rates by baseline intention to vaccinate may have biased our data, especially from the final follow-up at 1 year after birth, as those who intended to vaccinate were more likely both to follow-up and to interact positively with MomsTalkShots.

Evaluation of MomsTalkShots through a RCT has high internal validity and provides compelling efficacy data. Our study included recruitment by study coordinators in obstetric practices using financial incentives, whereas in a real-world setting, strategies would be needed to encourage pregnant women and mothers to use MomsTalkShots without these incentives. While scale-up is not a challenge from a technical perspective, dissemination and support from healthcare providers, public health authorities, and other partners would be critical. Other similar interventions do not offer this potential combination of effectiveness and scalability. In-person training of providers to improve their vaccine communication with patients has been shown to be effective, but scale-up would be cost- and time-intensive [52‐54]. Several other educational vaccine apps and websites have been developed [70, 71, 76‐80]. These include: ImmunizeCA, a smartphone app which helps Canadians manage their family’s immunizations by generating customized immunization schedules and reminder alerts for each family member [77]; ReadyVax, a smartphone app providing access to evidence-based vaccine information for providers and patients [79]; and HPV Vaccine: Same Way, Same Day, a smartphone app which teaches evidence-based vaccine recommendation practices including motivational interviewing skills using simulated role-play scenarios [80]. However, only one – a web-based social media intervention during pregnancy – demonstrated a significant positive effect on vaccine uptake, and its scalability is a challenge due to its reliance on public interactions with vaccine experts, whose time is limited and expensive [70, 71]. MomsTalkShots is the only app or website that tailors information on vaccine attitudes, concerns and demographics. Further research is needed to identify characteristics beyond tailoring that make such apps and websites effective versus ineffective among various populations and settings.

The need for effective interventions to improve vaccine confidence and uptake has only increased since the conclusion of this study. The Coronavirus disease 2019 (COVID-19) pandemic has caused widespread morbidity and mortality, while disjointed government response has led to confusion, the unfortunate politicization of vaccination, and vaccine hesitancy coming to the forefront of public consciousness [81, 82]. COVID-19 vaccines were at first particularly perplexing for pregnant women, given pregnant women’s exclusion from clinical trials but increased risk for severe illness from COVID-19 [83]. As of June 2022, 71% of US pregnant women were fully vaccinated against COVID-19, compared to 77% of adults overall [36]. However, most (95%) vaccinated pregnant women had been fully vaccinated before becoming pregnant. Promising early data on efficacy [84‐86] and safety [87‐89] of mRNA COVID-19 vaccines in pregnancy were eventually published, and the American College of Obstetricians and Gynecologists (ACOG) strongly supports vaccination against COVID-19 during pregnancy [90].

Adaptation of MomsTalkShots for new vaccines and populations beyond English-speaking pregnant women in the US has the potential to improve vaccine knowledge and perceptions more broadly. We are currently updating and expanding MomsTalkShots to become “LetsTalkShots”, which will cover vaccines across the lifespan, including routine adolescent and adult vaccines such as HPV, influenza, and shingles. Crucially, MomsTalkShots has also been adapted to improve COVID-19 vaccine knowledge and perceptions by providing easily accessible, individually-tailored messages to assuage common concerns about COVID-19 vaccines (e.g., new technology, rushed timeline for development, long-term safety, fertility, safety in pregnancy) and appeal to populations with lower acceptance (e.g., pregnant women, ethnic minorities, younger age, less education, conservative political ideology) [91‐93]. This new iteration of MomsTalkShots, called “LetsTalkCovidVaccines”, is free and accessible for all at http://​letstalkcovidvac​cines.​com/​. The content, design, and distribution of LetsTalkShots and LetsTalkCovidVaccines will be regularly assessed and upgraded, to reflect updates in science, incorporate new topics of concern, improve the user experience, and expand access, reach, and impact.