Improving Longitudinal Survey Participation Among Internal Medicine Residents: Incorporating Behavioral Economic Techniques and Avoiding Friday or Saturday Invitations

Low participation rates limit the generalizability of surveys. Prior studies show participation is better with financial incentives.^{1, 2} Beyond financial incentives, insights from behavioral economics reveal that psychological concepts, like regret aversion,³ can increase participation. For example, subsequent medication adherence improves when patients are entered into a “regret lottery”—they can win prizes if they adhere or are told how much they could have won, had they adhered.⁴ Other studies of survey participation have compared response rates for some, but not all days of the week.⁵ Using a longitudinal survey of internal medicine residents, we examined the effect of (a) whether the prize and regret message components of a regret lottery increased subsequent participation and (b) whether response rates varied based on the day of the week survey invitations were sent.

Residents from 63 internal medicine programs were invited to complete web-based surveys for the Individualized Comparative Effectiveness of Models Optimizing Patient Safety and Resident Education (iCOMPARE) trial, a cluster-randomized trial designed to test alternative resident duty hour rules.⁶ Residents received one survey, with 24 h to respond, every 2 weeks for 16 consecutive cycles from September 2015 through April 2016. Before each cycle, one first-year and one junior or senior resident at each program was randomly pre-selected to win a prize (either a $25 or $100 Amazon gift card) and randomly assigned a day of the week for survey invitation. At the end of each cycle, pre-selected residents who responded were emailed the prize. Pre-selected residents who had not responded received a “regret message” indicating what they would have won, had they responded. This procedure was repeated each cycle.

We fit a logistic regression model with survey completion (“responding”) as the outcome. The model included two binary variables for the prior cycle’s (a) lottery selection and (b) survey response, and an interaction term between the two. We used generalized estimating equations to account for clustering by resident, adjusting for resident year, trial arm, training program, cycle number, and cumulative prior response rate. In a secondary analysis, we distinguished the effects of winning or regretting by dollar amounts ($25 or $100). The University of Pennsylvania institutional review board approved the protocol. Analyses were conducted using Stata v14.0 (StataCorp LLP).

In total, 97,142 survey invitations went to 6228 residents, with a 40.0% overall participation rate (Table 1). Non-responders who received a regret message improved their subsequent response probability (OR 1.37, 95% CI 1.12 to 1.68; P < 0.001), as did responders receiving a prize (OR 1.44, 95% CI 1.09 to 1.90; P = 0.003) (Table 2). Each represents approximately a 4% increase in participation. No differences were found when comparing $25 versus $100 for either regret notices or receiving a prize (P = 0.08 and P = 0.83, respectively), although power for these tests was low. No differences were observed comparing the effect of regret messages with prizes (P = 0.77). The probability of response was lower when survey invitations were sent on Fridays (OR 0.72, 95% CI 0.65 to 0.79; P < 0.0001) and Saturdays (OR 0.84, 95% CI 0.76 to 0.92; P < 0.0001), corresponding to responses required by Saturday and Sunday, respectively.

Understanding the influence of behavioral economic techniques on survey response rates is an important advancement in survey science. Our findings suggest that regret messages were as influential as receiving prizes for increasing subsequent survey participation among residents. Additionally, survey responses expected on weekends had fewer responses. Our study has several limitations. First, this is a study of internal medicine residents and may not be generalizable to other populations. Second, this is a study of participation in repeated surveys. The effects of achieved or anticipated prizes or regret may differ when a single survey is administered. Third, the influence of regret messages and days of the week may have been different had we administered paper-based or mail-in surveys. Additional evaluations are needed to validate our findings in these other settings. However, at minimum, these simple techniques are likely to improve survey participation in a longitudinal survey of internal medicine residents. If behavioral economic techniques and targeting the ideal days of the week can increase participation in other survey designs, their low-cost approach (if not free) could be an important, cost-effective mechanism for improving the generalizability of survey-based findings.