Improving postal survey response using behavioural science: a nested randomised control trial

One of the most common data collection methods used in health research is the provision of postal questionnaires, especially when seeking information from large geographically dispersed populations [1]. Postal response rates are considered an important indicator of study quality as they can act as a metric of sample representativeness [1, 2]. Sufficiently high response rates help to reduce some forms of non-response bias, maximise sample size, and minimise research costs [2, 3]. However, adequate response rates are increasingly difficult to obtain with declining rates of participation in health research observed over time, worldwide [4‐6].

Systematic reviews have identified effective strategies to increase postal response rates in randomised controlled trials [1, 7‐11]. Providing incentives (money, gifts, or prize draws), pre-notifying participants, incorporating university sponsorship, using personalised messages, and sending reminders or a second copy of a questionnaire to non-respondents, have all been shown to increase participant response [1, 8, 12‐14]. The design of a questionnaire (content, length, format) can also be altered to achieve differential effects [1, 7, 9]. However, variable effects have been found for observational studies [11].

Dillman’s Tailored Design Method, established in 1970s, has been one of the most common frameworks employed to design research surveys and optimise response rate [15, 16]. More recently, theoretically driven behavioural science techniques have been tested in empirical studies in an attempt to improve participant engagement and response [17‐21]. Behavioural frameworks can act as tools for guiding and implementing applied techniques to inform content and design of written materials, such as letters and postal packaging. The ‘MINDSPACE’ Report [22], for example, contains a behavioural science checklist which outlines nine influences on behaviour which can be targeted in routine communications: (i) messenger (we are heavily influenced by who communicates information); (ii) incentives (responses to incentives are shaped by predictable mental shortcuts); (iii) norms (we are strongly influenced by what others do); (iv) default (we tend to follow pre-set options); (v) salience (our attention is drawn to what is novel and seems relevant); (vi) priming (we are influenced by sub-conscious cues); (vii) affect (emotional associations shape our actions); (viii) commitment (we seek to be consistent with public promises and reciprocate acts); and (ix) ego (we act in ways that make us feel better about ourselves). MINDSPACE and other behavioural frameworks are often implemented within research teams and government agencies [21‐23]. However, there is a paucity of evidence relating to their efficacy for improving survey response in health research.

Furthermore, most studies aimed at modifying postal response rates have isolated single techniques, testing the effect of each one individually [1, 7‐11]. Though these studies provide insight into specific explanatory mechanisms, they lack ecological validity when compared with routine practice, where multiple techniques are often combined. Also, more generally, the behavioural science literature suggests combining relevant behaviour change techniques to maximise effect sizes [24, 25].

The aim of this nested randomised controlled trial (RCT) was to evaluate whether using a pragmatic combination of behavioural science and evidenced-based techniques in a study invitation letter increased response rate to a health-related survey, when compared with a standard invitation letter. We hypothesised that the intervention letter would increase participant return of the postal survey.

The manuscript was written in line with CONSORT and TIDieR guidelines – see Supplementary Files 1 and 2, respectively, for completed checklists.

In total, 2702 individuals were invited to take part and mailed a survey; 1353 were randomised to the intervention and 1349 to the control arm. The mean age of the population was 37.5 years and the majority lived in the two most deprived IMD Quintiles in England (n = 1431, 56.2%; Quintiles 1 and 2). Around three quarters of participants were recruited through NHS Greater Manchester (n = 2090, 77.4%) and most had received their first HPV-positive with normal cytology screening result (n = 2202, 81.5%). Baseline characteristics were similar between the two randomised groups, with slight differences observed for some IMD quintiles (see Table 2).

Overall, 646 participants returned the completed survey, generating a response rate of 23.9% (n = 357, 26.4%, intervention; n = 289, 21.4% control). Figure 3 displays a flow diagram of the recruitment process.

Table 3 presents demographic characteristics for responders (n = 646) and non-responders (n = 2056). Supplementary File 4 presents a table of demographic characteristics stratified by intervention vs. control group for responders.

Almost all postal questionnaire studies incorporate an invitation or cover letter. We found that applying behavioural science and evidence-based methods to routine invitation letters improved postal response to a health-related survey, after adjusting for demographic and clinical characteristics. As survey participation rates continue to decline worldwide [4‐6], our findings provide support for the pragmatic and cost-effective adoption of combined techniques in routine research to increase postal response rates.

Consistent with previous systematic reviews evaluating the application of individual techniques, we found that combining several techniques positively influenced postal response rate [1, 7‐10]. The magnitude of effect observed in our study (adjusted odds ratio of 1.30 in favour of the intervention) is higher than found in some isolated techniques which similarly carry low or minimal financial costs, such as adopting personalisation or use of non-monetary incentives (odds ratios of 1.16 and 1.13, respectively [2, 9]). However, this is not the case when compared to all cost-effective isolated techniques, such as mentioning an obligation to respond or the use of university sponsorship, which demonstrate similar or slightly larger effects (odds ratios of 1.61 and 1.32, respectively [8, 9]). Furthermore, our approach appeared to yield a lower effect size than certain more financially expensive or resource-intensive strategies, such as providing monetary incentives, use of recorded mailed delivery, and pre-notifying participants (odds ratios of 1.87–1.99, 1.76–2.04, and 1.45–1.50, respectively [8, 9]).

Ultimately, however, findings which are based on isolated techniques cannot act as a direct comparator to our study. This is partly due to differences in the content used in the control arms of studies and variations in adjustments for confounders and contexts. For example, in our study, the control and intervention letters both utilised some techniques which have been shown to increase participant response, such as providing assurance of confidentiality and a conditional incentive of financial payment for participation in an interview [8, 9]. Similarly, we provided a second copy of our questionnaire at follow-up which has been shown to improve response [9]. Utilising these evidence-based techniques in our control letter mirrors standard research practice; however, this differs to several previous studies which avoid using techniques in control conditions or do not report control conditions. It is therefore possible that the effect sizes observed in our study could be subject to ceiling effects or reflective of additive effects. Conversely, our study questionnaire (sent to all participants) asked about a sensitive health topic and our study information sheet explicitly stated that participants could opt out, which have both been found to reduce the odds of response [8, 9]. Hence, overall, these heterogeneities in methodology and study contexts prohibit comparative conclusions relative to the previous literature.

Using a combination of techniques in our study also introduces the possibility of interaction and/or moderation effects between individual techniques, which we could not measure or test. Two or more techniques implemented in tandem may have led to differential impacts on response rate, when compared with the same techniques used in isolation. Hence, a core limitation of our pragmatic approach is that we are unable to determine optimal combinations of techniques and, similarly, whether certain combinations may have reduced response or counteracted positive effects. Further investigation is needed to test the magnitude of effects using different combinations of techniques (e.g., through adopting a factorial RCT design) and to assess the impact of potential interactions.

Response rate is known to be influenced by sociodemographic factors such as age, sex, educational attainment, ethnicity, marital status, and deprivation [5, 31, 32, 35, 36]. Living in a less deprived area (lower index of multiple deprivation score) yielded a small statistically significant effect size in favour of returning a survey in our study (adjusted odds ratio of 0.99), but we observed no effect for age in our adjusted analyses. We did not test for interaction effects between area-level deprivation and response to the survey, as this was not part of our planned analysis and due to the likelihood of issues with statistical power. Also, we did not have data on other important sociodemographic variables like ethnicity and education. Hence, even though our intervention increased survey response overall whilst adjusting for some demographic factors, we cannot rule out that bias remains for particular sociodemographic groups.

Improving response rates in survey-based studies remains a priority for health and epidemiological research. It is hoped that the gains yielded from better sample representativeness and lower non-response bias should ultimately translate into improved public and patient outcomes [37]. Implementation of behavioural science techniques in routine research practice may offer a low-cost solution for generating higher response rates and thus enhancing quality of care.

Using a combination of easy-to-implement behavioural science and evidence-based techniques in a study invitation letter increased participant response to a health survey. The major benefit of this pragmatic approach was the absence of substantive additional research costs, like providing financial incentives or additional follow-up mailing strategies. Further research is needed to investigate the optimal combinations of techniques for increasing postal response.