Animated, video entertainment-education to improve vaccine confidence globally during the COVID-19 pandemic: an online randomized controlled experiment with 24,000 participants

Our primary outcome is vaccine hesitancy (including COVID-19 vaccine hesitancy). We will ask participants how much they agree or disagree with statements related to perceived vaccine safety as well as their attitudes, preferences, beliefs, and hesitancies regarding regular vaccines and the COVID-19 vaccine, respectively [16]. In the statistical analysis process, we will use nine questions to measure the degree of vaccine hesitancy in general and seven questions to measure the COVID-19 vaccine hesitancy specifically. Then we will normalize the final score to a range of 1–5 for both measures for comparison, which higher scores indicate higher degrees of vaccine hesitancy. These questions are shown in Table 1 and the data elicited from this survey will enable us to achieve objective 1.

Our secondary outcomes include participants’ behavioral intent to get vaccinated as well as participants’ level of hope. In order to reduce the social desirability bias often associated with direct questioning about sensitive items [15], we will use the unmatched count or list randomization approach to devise a series of list experiments [17]. Four list experiments will be used, as shown in Table 2. For each experiment, the control group will receive a list of three items. Participants are asked how many items they are likely to do in the coming months without stating which ones they chose. The treatment group will get the identical item lists but with one additional “sensitive” item relating to their behavioral intent to get vaccinated or encourage their loved ones to do so. For example, imagining that the control group select 1 out of the 3 items on average while the treatment group select 1.3 out of the 4 items, with the assumption that the average acceptance of these two cohorts is the same, we can conclude that the prevalence of participants who would get vaccinated against COVID-19, is 20%. We have designed the list of experiments in accordance with published best practices [17], and these data will be used to assess objective 2.

Second, we will assess participants’ level of hope using the Adult Hope Scale [18], a 12-item scale measuring participants’ level of hope (Table 3). The Adult Hope Scale is comprised of two subscales relating to Snyder’s cognitive model of hope: (1) Agency (goal-directed energy) and (2) Pathways (planning to accomplish goals) [11]. The 12 items include 4 Agency items, 4 Pathways items, and 4 distractors. Participants are asked to respond using an 8-point Likert-type scale ranging from Definitely False to Definitely True. As recommended in the literature, we will refer to the scale as “The Future Scale” within the survey experiment.

We also aim to measure the difference of the outcomes above between the three intervention arms a (storytelling-instructional-humor), b (storytelling-analogy), and c (storytelling emotion-focused). These data will be used to assess objective 3.

The study investigators will be responsible for data collection. Data will be collected on either the Gorilla or Kurundata platforms. The data retrieved from the platforms will be anonymous. Data downloaded will be stored on an encrypted and secure server. The data will be deleted two years after the study has been completed.

Participants will be automatically timed out from the online platform if they take more than 45 min to complete the study. Though participants can withdraw the study at any time, they will not be compensated for incomplete survey participation. Incomplete data will be excluded from our analyses. Since the participants are anonymous to us, there is no way to initiate follow-up in the time limit.

Data will be collected on the Gorilla or Kurundata platforms. Third parties except for ProA or Kurundata will not have access to the data. The data will be downloaded and safely stored for statistical analysis on a computing system maintained by the University of Heidelberg in Germany.

Because of the anonymized participant IDs, the study investigators can never meet or know the identity of the study participants. The study investigators will only have access to the participants’ anonymized ID and no other personal or confidential information, and the data will be deleted 2 years after the study has been completed. The study investigators will keep this information confidential.

We will use descriptive statistics to obtain summaries of the demographic data (age, sex, education status, country of residence, etc.).

For each participant, we will calculate their (COVID-19) vaccine hesitancy score based on their survey responses. Let \( \overline{K_k} \) denote the mean vaccine hesitancy for each trial arm, where k ∈ {a, b, c} such that a represents the intervention arm a (video with storytelling-instructional-humor approach), b represents the intervention arm b (video with storytelling-analogy approach), c represents the intervention arm c (video with storytelling-emotion-focused approach), and d represents the control arm. To evaluate the effectiveness of the video, we will use ANOVA for continuous endpoints and χ² test for binary endpoints to assess the statistical significance of the difference between the control arm and the treatment arms. To investigate the factors that affect the endpoints, we will use generalized linear mixed models (GLMM) for the analysis of our endpoints. We use ordinary least squares (OLS) regression for our continuous endpoints; we will use (modified) Poisson regression for our binary endpoints; we will use negative binomial regression for our count endpoints. The reason for the choice of modified Poisson regression for our binary endpoints is that this analysis has good statistical properties and generates risk ratios, which are far easier and safer to interpret than the effect size measures generated by alternative methods (such as odds ratios or marginal effects) [19‐22].

For the vaccination plan in the list experiment, we will calculate the mean score for the control list and treatment list, denoted by \( \overline{C_i} \) and \( \overline{T_i} \) respectively, where i is the index of the list. Then we can calculate the mean difference between the control list and treatment list within each trial arm, which is considered as the participants’ behavioral intent to get vaccinated in this trial arm, denoted as \( \overline{D_{ik}}=\overline{T_{ik}}-\overline{C_{ik}} \). Analogous to difference-in-difference analyses, we can identify the effect of each treatment.

For other secondary outcomes, we can use the same statistical procedure described above to assess the effectiveness of our E-E video. We will use R statistical software to undertake the analysis.

No interim analyses are planned.

Since we will conduct the trial in the USA and China, we will conduct both country-specific and cross-country pooled analyses for further comparison.

Participants who decide to withdraw from the trial will not be replaced

This document is the full protocol. Anyone interested in other data or documentation should contact the corresponding author.

The trial will be overseen by a trial steering committee (TSC). The TSC will have an independent chairperson and members but also includes the trial collaborators. Two TSC meetings are planned.

As this is an online survey, there is very minimal risk for study-related injury or harm. We will present non-threatening video interventions and questions about vaccine attitudes. There are no foreseeable risks to participating in the online study.

No ancillary and post-trial care is planned.

The results of this study will be disseminated through presentations at international conferences and publications in peer-reviewed journals. Results will be used by the study collaborators and their institutions (Stanford School of Medicine, Heidelberg University, Chinese Academy of Medical Sciences, and Peking Union Medical College) to improve the design and universal appeal of future educational and health promotion videos. All investigators who meet authorship criteria will be included as co-authors and anyone who contributed, but does not meet the criteria for authorship, will be acknowledged. No professional author services will be used.