Comparison of response rates on invitation mode of a web-based survey on influenza vaccine adverse events among healthcare workers: a pilot study

The sample for this study was drawn from members of a registry maintained by the investigator. The registry consists of adults aged 18 years and older, mainly (84.4%) healthcare workers, who participated in previous studies conducted by the Infectious Disease Epidemiology Research Unit at Mount Sinai Hospital, Toronto, Canada. The registry contains the volunteer’s name, year of birth, sex, and email and postal addresses. Eligibility for this study was limited to registrants with both an email and a postal address. The study was approved by the Sinai Health System research ethics board (REB #16–0267-E).

Eligible individuals were randomly assigned to one of eight groups. Four groups received an invitation by email with a direct link to the survey site while the other four groups received an invitation by post with the URL to the survey site included in the letter. Each invitation arm was randomized to receive one of four versions of the questionnaire: short or long with either open- or close-ended questions. A unique 6-digit personal identification (PID) code was assigned to each individual using a random number generator and used to ensure that only invited people were able to participate and to track responses so responding individuals did not receive unnecessary reminders. Individuals accessing the survey site first landed on the consent page. If they consented, they were brought to the PID code verification page. Once the code was verified, they were taken directly to the appropriate questionnaire. All questionnaires were implemented using SimpleSurvey® (OutSideSoft Solutions Inc., Quebec).

Invitations asked potential participants to help us learn “how to best ask questions” and their experiences with adverse events following immunization with influenza vaccines. The visual design and subject matter was consistent across the four versions of the questionnaire. Questionnaire version distinctions were made on the basis of i) overall length: short (24 questions) versus long (additional 4 questions) and ii) response format: open-ended versus close-ended (to five (short version) or six (long version) questions). Questions were developed through adaptation (with permission) of existing questionnaires including Nichol and Hauge (1997), Feemster et al. (2011), and Opel et al. (2011) [24‐26]. Questions included age, sex, occupation, work site, as well as influenza vaccination history, factors affecting the decision to be vaccinated, experience with side effects, and impact of side effects. To determine if there were signals that might predict adverse events in certain populations, there was a series of questions asking about non-influenza vaccines received and resulting side effects, as well as allergies or sensitivities to medications, foods, or environmental allergens. Skip logic was used to present respondents with follow-up questions based on previous responses. Open-ended responses were categorized by one person (XT) using the closed-ended items as a guide.

The initial invitations were sent by post or email on November 22, 2016 to allow for the maximum number of respondents to be vaccinated against influenza for the season but to allow for reminders and responses before the Christmas holiday vacation period. Each invitation was personalized (e.g., “Dear Jane”) and contained a direct URL to the survey, a brief introduction to the study to increase salience, and the PID code. The invitation emails were sent using the SimpleSurvey® interface which generated a list of email addresses to which the invitations had failed to send (e.g., address errors, inactive accounts). If errors were identified and a second email address was listed in the registry, a second attempt was made via email; if not, no further contact was attempted. If an invitation letter was returned by the postal service (e.g., invalid address), no further attempt was made to contact the subject. The research assistant answered telephone and email queries from potential respondents who had questions about the study or problems accessing the survey site.

Reminder emails were sent to all non-respondents seven days after the initial email while reminder letters were sent 14 days after the initial letter. The reminder letter was delayed by 7 days compared with the reminder email to allow for delivery by the postal system, which is commonly 2–4 business days. The online surveys were deactivated on December 28, 2016 and response data were downloaded the same day.

Response rates by invitation mode (i.e., post vs. email) were calculated based on the final disposition codes defined by the American Association for Public Opinion Research (AAPOR) using response rate 2 (RR2), which counts both complete and partial responses in the numerator [27].

(Abbreviation: RR, response rate; I, complete interview; P, partial interview; R, refusal and break-off; NC, non-contact; O, other; UH, unknown if household/occupied HU; UO, unknown, other.)

Adverse events following influenza vaccination were categorized into four groups: no adverse event, local adverse event only, systemic adverse event (most recent only), and recurrent systemic adverse event. An adverse event was defined as any untoward, unfavourable, or unintended symptom or disease following influenza immunization. A local adverse event was defined as any local reaction at the injection site including pain, soreness, numbness, bruising, redness, or swelling. A systemic adverse event was any of: upper respiratory illness (with or without sore throat, coughing, stuffed or runny nose, etc.), fever/feverishness, headache, new joint pain, malaise/feeling generally unwell, new & unexplained muscle aches, tiredness/fatigue or sweating within 7 days, a rash with onset within 48 h, or an allergic reaction with onset within 24 h of vaccine receipt [28]. A recurrent systemic adverse event was defined as at least one of the same systemic adverse event symptoms occurring following the receipt of the two most recent influenza vaccines. Respondent’s ages were categorized into four groups (18–34; 35–49; 50–64; 65+ years).

As shown in Fig. 1, of 1364 people in the registry, 1334 participants aged 22–94 years had both an email and postal address and were randomized to 8 groups by invitation mode, survey length (long vs. short), and response type (open- vs. close-ended). There was no significant difference by age or sex in those who were and were not eligible to be randomized. After excluding 6 ineligible individuals, there were 649 and 679 participants eligible for contact by email and post, respectively. The eligible sample had a median age of 48 years (95% CI: 47, 49) and 80% were female, which is reflective of the healthcare personnel in Canada [29].

There were no differences by median age (48 vs. 48 years; p = 0.89) or sex (78.7% vs. 81.6% female; p = 0.18) between the two invitation mode groups (email and post, respectively).

There were 559 (41.9%) potential participants with two email addresses available in the registry. After invitations were sent, 67 (10.3%) initial emails and 60 (8.8%) letters were returned as improperly addressed (p = 0.35). Twenty-three (76.7%) emailed invitations were successfully re-sent to an alternate email address. There was ultimately no statistically significant difference in the percentage of returned invitations by mode (44 (6.8%) email vs. 60 (8.8%) post) (p = 0.17).

As shown in Table 1, the overall response rate was higher for those sent the invitation by email (34.8%) than by post (25.8%; p < 0.001). Older participants were more likely to respond to the survey than younger ones, overall (median age 53 vs. 46 years, p < 0.001) and by invitation mode (p_trend < 0.001). Participants responding to emailed invitations were younger (median of age: 51 years) than those who responded to postal invitations (55 years; p = 0.005). Although males and females were equally likely to respond overall, females were more likely to respond to the email (36.0%) than the posted letter (24.4%; p < 0.001). This was driven by females younger than 50 years of age who were significantly more likely to respond to the email versus the postal invitation (32.1% vs. 15.9%, p < 0.001) while there was no difference for older women. Males 50 years and older were more likely to respond to the posted invitation (51.7%) than females in that age group (34.8%; p = 0.02) while there was no difference in response rates to emailed invitations (41.1 and 35.6%, respectively).

As shown in Table 2, there were some statistically significant differences in response rate by the length and/or format (open-ended vs. close-ended) of the questionnaire, with those randomized to the longer and open-ended questionnaire more likely to respond if they were invited by email. However, since the potential participants were not aware of the questionnaire they would receive, the differences are likely due to statistical chance. There was no statistical association between the length and type of questionnaire and the likelihood of providing an incomplete survey. Of the 6 (1.5% of 401) respondents who submitted partially completed surveys, 2 received long, open-ended, 3 received long close-ended, and 1 received short open-ended versions (p = 0.53).

For women, the odds of responding to the survey was higher (OR 1.77) by email than post (p < 0.001). On the other hand, there was no difference in the odds of responding by email or post for males while holding the effects of age constant (p = 0.90). While holding the method of response (email verus post) constant, both older men and older women were more likely to respond than younger ones (see Table 3).

As shown in Fig. 2, the number of responses from the emailed invitation reached a peak on the day the first invitation was sent with a second peak on the day the reminder emails were sent. In comparison, the peak for the posted invitations was 2 days after the first letters were sent with a second peak 2 days after the reminder letters were sent. The response rate after the initial invitation (up to date of second invitation) was 21.3 and 14.4% for those who received email and postal invitations, respectively (p < 0.001). Older participants were more likely to respond after initial invitations than younger groups (p < 0.001) but there were no difference by sex (p = 0.27).

Of the 401 people who completed a questionnaire, 96.5% had been vaccinated against influenza at least once during adulthood and 76.0% were vaccinated every year since their initial vaccination. Of those who had not been vaccinated during adulthood, the main reasons stated were: the effectiveness of the vaccines (N = 5), inconvenience (N = 3), and not feeling at high risk of getting influenza (N = 3). The top reasons for not being vaccinated every season were: the limited effectiveness of the vaccines (N = 28), inconvenience (N = 22), concern about side effects from influenza vaccines (N = 21), and not feeling at high risk of getting influenza (N = 14).

Of the 387 previously vaccinated participants, 100 (25.8%) reported an adverse event following receipt of their most recent influenza vaccine including 70 (18.1%) who reported systemic event(s) including muscle ache (N = 27), acute upper respiratory illness (N = 26), headache (N = 25), fatigue (N = 21) and malaise (N = 17). Recurrent systemic adverse events (same systemic event after their two most recent vaccinations) was reported by 22 of 387 participants (5.7%) and included muscle aches (N = 13), fatigue (N = 9), headache (N = 8), malaise (N = 7) and acute upper respiratory illness (N = 7).

As shown in Table 4, there were no differences (at P < 0.025, to adjust for multiple testing) in the percentage of respondents reporting recurrent systemic adverse events by invitation mode, question format (open vs. closed), length of the survey, age, sex, or adverse event recall of non-influenza vaccines. Of the participants who recalled their experience with non-influenza vaccines, 67.8% (208/307) reported no adverse events after any vaccines while 8.1% (25/307) reported an adverse events (local or systemic) after both influenza and non-influenza vaccinations. There was also no correlation between known allergies or sensitivities and the occurrence of recurrent systemic adverse events.

A higher percentage of respondents who reported only a local adverse event (27/29 or 93.1%) than those who reported a systemic adverse event (46/66 or 69.7%; p = 0.04) planned to be vaccinated again next season. Of the 66 people who reported systemic adverse events, an equal percentage of those with an event after only their most recent vaccination (29/45 or 64.4%) and those with recurring events (17/21 or 81.0%; p = 0.38) planned to be vaccinated again next season. Respondents whose adverse event affected their daily activities or required a medical consultation were significantly less likely to plan to be vaccinated next year (12/25 or 48.0%) than those with adverse events that did not (61/70 or 87.1%; p < 0.001). Participants who reported no side effects were not asked about their intentions to receive the influenza vaccination next season; a change will be made to the final survey to correct this oversight.