Methods to increase response rates to a population-based maternity survey: a comparison of two pilot studies

Surveys are an important research method for collecting population data that are unavailable from other sources. However, response rates to surveys have been declining over the last 30 years [1]. The declining trend is exemplified in questionnaire surveys into maternal and infant health. For example, the response rate to the National Maternity Survey (NMS), which uses postal questionnaires to survey mothers during the postnatal period, has fallen from 67% in 1995 to 47% in 2014 [2‐4]. This decline is consistent with other large UK maternity surveys, such as the Infant Feeding Surveys [5] and the Care Quality Commission Maternity Surveys [6].

A number of possible explanations have been proposed for the growing problem of non-response to survey research including greater time pressures, the increasing number of surveys in circulation, survey fatigue and privacy concerns [7]. There is a large literature reporting methods to improve survey response rates. A 2009 Cochrane review identified 481 trials that examined 110 different methods to increase response rates to postal and internet-based questionnaire surveys [8]. A number of strategies were found to substantially increase the odds of response to postal surveys including: offering incentives, pre-notifying individuals, shortening questionnaires, follow-up contact and employing more personable questionnaires. More recent research has consistently endorsed the use of incentives and multiple contacts [9‐11] whereas evidence for pre-notification, shorter and more personable questionnaires is less clear [10‐13]. Four methods (pre-notice letter, shorter questionnaire, postcard reminder, redesigned cover letter) were tested in a large randomised experiment conducted due to a persistent decline in response to the annual GP Patient Survey in 2015 [14]. With the exception of the pre-notice letter, all methods had a positive impact and, following implementation of these, the response rate to the GP Patient Survey increased, yet still remained below 40%.

Literature is emerging on methods to increase survey returns yet response rates remain low. If we are to continue to use postal and online questionnaire surveys to collect vital data on population health, the problem of declining response rates needs to be addressed. The aim of this paper is to evaluate the impact of five selected methods on the response rate to pilot surveys, conducted prior to a large-scale NMS in England.

Table 2 shows the baseline sociodemographic characteristics of the women selected for the samples in pilot surveys 1 and 2. Overall, there were no differences between the two samples of women in terms of age, marital status at the time of registering the birth of the baby, country of birth, IMD, or region of residence (p > 0.05).

Figure 1 shows the cumulative weekly response rates to pilot surveys 1 and 2 (additional file 3). The pattern of response was similar in both pilot surveys with pilot 1 response falling marginally behind pilot 2 response throughout most of the data collection period.

Table 3 shows the overall response rates to pilot surveys 1 and 2 and the difference between the response rates with 95% CI. The response rate to pilot 1 at the end of the survey was 28.7% and the response rate to pilot 2 at the end of the survey was 33.1%, which represents a 4.4% increase in response (95%CI:0.88 to 7.83, p = 0.02).

Figure 2 shows the percentage of responses received during each week of pilot surveys 1 and 2 (additional file 4). There were increases in response following each of the mail-out periods. The response rate was highest to the initial mail-out in both pilots with a diminished return after each subsequent mail-out.

Table 3 shows a breakdown of the response rates to each pilot survey by different time-points during the mail-out process. After the initial mail-out and prior to the first reminder, the response rate to pilot 2 was 1.7% higher than the response rate to pilot 1 (95%CI: -1.25 to 4.52, p = 0.25). After the first reminder and prior to the second reminder (in pilot 2 only), the response rate to pilot 2 was 0.6% higher than to pilot 1 (95%CI: -2.84 to 3.95, p = 0.73). Therefore, at these equivalent time-points in the mail-out process in pilot surveys 1 and 2, there were small increases in response to pilot 2 which were not statistically significant.

Table 3 also shows a breakdown of the response rates to each pilot survey by mode of response. In pilot 1, the majority of women opted to complete and return the postal questionnaire (26.8%) rather than to take part online (1.8%) or over the telephone (0.1%). The preference for taking part via post was also evident in pilot 2 with 29.7% opting to complete and return the postal questionnaire compared to 3.5% opting to take part online; no women utilised the telephone option in pilot 2. Although the numbers are small, the proportion of women choosing to take part online almost doubled from 1.8% in pilot 1 to 3.5% in pilot 2, which corresponds to an absolute increase of 1.7% (95%CI: 0.45 to 2.81, p = 0.01).

Table 4 shows a breakdown of the response rates to each pilot survey according to maternal sociodemographic characteristics. There was some evidence of a larger increase in the response rate to pilot 2 compared to pilot 1 by women with certain sociodemographic characteristics, for example, a higher proportion of women aged 30–34 years, married women, and women born in the UK and outside of the UK responded in pilot survey 2. However, the numbers of women included in these subgroup analyses were small and the confidence intervals for the differences overlap.

The analysis focused on unit non-response but item non-response was also assessed. The proportion of missing data was below 5% for all key non-optional items on the questionnaire.

In this study, a number of methods were evaluated for increasing response rates in an English population-based maternity questionnaire survey. Taken together, pre-notification, a shorter questionnaire, more personable study materials, an additional reminder and the inclusion of QR codes led to an increase in the response rate. Although the overall increase was modest, it was statistically significant and methodologically important against a persistent downward trend in response rates to surveys. The findings show that declining response rates may be ameliorated with the use of these selected survey methods. There was some evidence to suggest that the selected methods may have had a greater impact on women with certain sociodemographic characteristics, although the numbers of women included in the subgroup analyses were small. Further research is required to explore how different research methods might affect response rates in different sociodemographic groups.

The methods evaluated in this study were mostly found to have a limited effect on the response rate but, due to the timing of responses, the findings suggest that reminders might be important. The literature on survey methods confirms the use of multiple contacts as one of the most influential factors in improving questionnaire returns [8, 17]. Despite this finding, objections to contacting potential research participants on numerous occasions are sometimes raised on ethical grounds. Clearly it is essential to protect potential participants from feeling coerced into taking part in studies. However, we also have a responsibility to those individuals who do choose to participate. Quality research requires the recruitment of sufficient participants to address the research questions and to draw valid inferences from the data collected. A 2008 synopsis of best practices for survey research recommended ‘when conducting survey research, if follow-ups are not planned and budgeted for, the study should not be initiated’ (p.2) [17].

The other method that was shown to be potentially effective in this study was the use of QR codes to enable easier access to the online questionnaire. Although the number of women opting to take part via the online survey was small and the use of this method did not significantly impact the overall response rate, the online response rate almost doubled when they were included. Therefore, the use of QR codes may have facilitated participation via this mode. According to the marketing literature, QR codes are simple and effective tools which increase user engagement [18]. However, there is very little information in the academic literature to support or refute the effectiveness of this technology.

Other methods that have previously been shown to increase response rates to questionnaire surveys were not found to be effective in this study. Our finding that pre-notification did not have a significant effect on response is consistent with the GP Patient Survey experiment [14] yet contrary to Cochrane review evidence [8]. The evidence for the impact of questionnaire length on response rate is unclear, with some studies suggesting that response rates improve with shorter questionnaires [8, 19] and others suggesting that response rate is unrelated to questionnaire length [10, 11]. This study did not provide support for using shorter questionnaires, although it is possible that, despite shortening the questionnaire, it was still deemed to be too long [20]. Finally, using more personable study materials did not increase the response rate in this study, contrary to existing findings, which suggest that improving study materials is an effective method [8]. However, the literature is not clear on what exactly constitutes respondent friendly design; hence the extent to which the material was improved in this study is uncertain.

According to the literature, the most effective method to increase response rates to surveys is to offer incentives to participants [8‐11]. However, as with multiple contacts, the use of incentives raises ethical considerations [21]. Traditionally, offering remuneration to individuals for their involvement in research has been discouraged due to concerns around coercion [22], yet the use of incentives is now becoming increasingly recognised as an acceptable, and often necessary, strategy to aid recruitment. Furthermore, remuneration may be an indication of respect for the time and contribution that research participants make [23]. Nonetheless, offering incentives is not without issues; it may generate selection bias, undermine autonomy around consent, preclude participant anonymity, and substantially increase the cost of research studies [21]. Indeed, we were unable to offer incentives in this study due to limited resources and budgeting for incentives in similar large-scale surveys might not always be feasible.

The main strength of this study is that a statistically significant increase in response rate was achieved with the implementation of selected methods aimed at increasing response. This increase was achieved over a period of eight months against a downward trend in response rates to postal questionnaire surveys of maternal and infant health over more than 30 years [2‐6]. The methods shown to be potentially effective in this study can now be developed and incorporated into future population-based maternity surveys. Another strength is the consistency in the design and materials employed in the two pilot surveys enabling direct comparison of the results.

The main limitation is that women were not randomised to one of the two pilot surveys. However, the women selected for both pilots were random samples from the population of all births and comparison of the women indicated they did not differ on key sociodemographic characteristics. Another limitation is that we were unable to isolate the impact of the different methods, possibly with the exception of the additional reminder. Due to the fact that pilot surveys 1 and 2 were not carried out at the same time of year, it is also possible that factors we were unable to control for might have impacted upon the response rates. For example, pilot 1 was launched during the Autumn (Fall) and ran through the Winter whereas pilot 2 was launched during the Spring and ran through the Summer. Consequently, the climatic conditions and the holiday periods during the two pilots would have been different and seasonal effects might have affected the response rate.

Declining response rates may be ameliorated with the use of selected survey methods. Additional evidence from randomised controlled trials is required to offer clearer guidance on which methods are most effective for maximising postal and online questionnaire survey returns. Further research is also required to identify novel strategies for engaging populations in survey research.