Background
The ability to detect treatment effects in a randomized controlled trial is partly determined by the number of participants providing primary outcome data, which is often threatened by participant attrition. Therefore, trial protocols specify the expected loss to follow-up and the number of participants required for recruitment in order to compensate for this attrition. However, primary outcome data are often collected in trials through postal or online questionnaires, whereby meeting target response rates can be difficult. One explanation is that questionnaires might be overlooked or ignored by participants, owing to the volume of junk mail via post and email.
One of the largest pragmatic telehealth trials to date, The Healthlines Study, involved two linked, parallel, randomized controlled trials which sought to assess the effectiveness and cost-effectiveness of a telehealth intervention to support patients with two exemplar and common long-term conditions: depression (
n = 609) and increased risk of cardiovascular disease (
n = 641) [
1]. The primary outcome in the depression trial was positive response to treatment, as measured by improvement on the Patient Health Questionnaire (PHQ-9) [
2] at four months post-randomization, with additional follow-up questionnaires administered at eight and 12 months. Participants could choose to complete questionnaires online or by post, and received up to five questionnaire reminders. Approximately 3–4 months into the start of the 8-month follow-up in the depression trial, it was apparent that the response rate (approximately 60–70 %) was not only falling below the 4-month rate for the equivalent time period (approximately 85 %), but below the trial protocol target of 80 %. This downward trend was not evident in the cardiovascular disease trial, and so three simple interventions were introduced to try to improve response rates amongst the depression group alone.
Achieving high response rates might be particularly difficult in trials with longer follow-up periods [
3,
4] and telehealth trials [
5], as well as with patients with depression [
6,
7], who typically struggle with concentration and motivation. In these instances, the efficiency gained by simply sending out follow-up questionnaires when they are due might not outweigh the effort required from multiple, time-consuming completion reminders or eventual loss to follow-up. Instead, it might be more effective to employ strategies when sending out questionnaires, to increase response rates. Therefore, we investigated different strategies embedded within the depression trial of The Healthlines Study to boost response rates.
Given the risk of bias from follow-up attrition and, hence, the threat to validity and reliability [
8], it is not surprising that numerous strategies for increasing response rates to follow-up questionnaires have been developed and tested. Some of these strategies are resource-intensive, but generally effective [
9‐
12]. One popular resource-intensive strategy is contacting participants to give them advance notice about an upcoming questionnaire (i.e., pre-notification). In two systematic reviews, which included 28 [
10] and 47 trials [
9] employing this strategy, Edwards et al. demonstrated that pre-notification increased the odds of responding by about half. However, many of the studies in these reviews were nested within a variety of study designs. Furthermore, the reviews included studies in both healthcare and non-healthcare settings, and examined this intervention in relation to completing postal questionnaires alone [
9,
10]. Further investigation is required regarding whether the beneficial pre-notification effect will transfer to a trial of mixed completion methods, in which participants can choose to complete either postal or online questionnaires. Therefore, we evaluated response rate effects in those completing either postal or online questionnaires after eight months of participation in The Healthlines Study depression trial.
Another broad strategy that has been examined in several studies seeks to capture the attention of participants by adding some form of novelty or distinctiveness to the questionnaire or accompanying cover letter [
9,
10]. The reduction in researcher time and effort to carry out such response rate strategies is an obvious benefit over more resource-intensive methods, but the success in achieving heightened response rates is less clear. One Cochrane review noted that including a picture within the questionnaire cover letter tripled the odds of response amongst participants completing online surveys (two trials), whereas this same intervention seemed to have no reliable effect for those receiving a postal copy (four trials) [
9]. The differential effect might be because the emailed photographs were in colour and contained actual people [
13], which could have enhanced their distinctiveness and visual appeal [
14], whereas the pictures within the cover letter of the postal questionnaires appeared to be in black and white [
15] and included a mixture of photographs and drawings or graphics [
15‐
18]. Additionally, none of these studies recruited participants from primary care or were embedded within a host trial. Thus, we investigated whether a colour photograph containing the names of the local research team, with whom all participants had had some degree of contact, in the cover letter or email prompting questionnaire completion would boost 12-month follow-up questionnaire responses in the depression trial of The Healthlines Study.
A relatively easy and cost-effective means of reminding non-responders to complete questionnaires is through email reminders. Some research has examined altering email subject lines [
9], since this might affect whether an email is opened at all [
19,
20]. Although it only involved two trial comparisons, one Cochrane review found no evidence of an effect on the odds of responding when comparing a topic in the subject email line against a blank subject line, including ‘Survey’ in the subject line against a blank subject line, or even including a plea for help in the subject line or not [
9]. Since these comparisons involved a student sample, the findings might not extrapolate to participants in a health-related trial. We trialled whether an action-oriented email reminder subject line, containing visual distinctiveness with some words capitalized (‘ACTION REQUIRED’), might boost 12-month response rates in non-responders in the depression trial when compared with the reminder subject line (‘Questionnaire reminder’) used at previous follow-up time points within The Healthlines Study.
Taken together, the three embedded response rate studies – pre-calling, including a team photo, and using an action-oriented email reminder – attempt to address some of the shortcomings we noted in the previous studies, but do so in a host trial that encompasses study characteristics particularly subject to participant attrition – longer follow-up (12 months), a telehealth-based design, and in participants experiencing depression. We hypothesized that receiving each of these interventions would improve response rates, as well as reduce questionnaire reminders and response time to the questionnaire over those who received the standard study procedure.
Discussion
Participant retention is a methodological concern in trials, and so identifying successful strategies to improve follow-up questionnaire completion is important. This may be especially pressing in trials at greater risk of attrition, such as those with longer follow-up periods [
3,
4], those including telehealth interventions [
5], and those involving participants with depression [
6,
7]. In three response rate intervention studies embedded within a 12-month telehealth trial for participants with depression (The Healthlines Study), there was no indication that any of these interventions improved overall response rate. However, there was evidence that pre-calling participants had some beneficial effects. Compared with those receiving no advance notification, pre-called participants at eight months post-randomization were less likely to require a questionnaire reminder, required fewer reminders, and returned the follow-up questionnaire about eight days earlier than controls. This strategy might be helpful when the timing of outcome completion is important.
Contrary to expectation, none of the response rate interventions appeared to boost the overall number of completed questionnaires, although this is probably because of very high response rates, creating ceiling effects. The most likely explanation for the unexpectedly high and sustained response rates amongst this group of participants is that the study team were already doing many of the things that, according to systematic reviews [
9,
10], tend to result in better response rates. This use of multiple strategies to ensure participant retention reflects recommendations in existing literature [
22,
23], particularly with participant groups that are difficult to recruit and retain [
24]. In these ways, we had already optimized retention and response to some degree prior to introducing the response rate interventions. Nonetheless, we devised and carried out these studies because of an early indication at the start of the 8-month follow-up that response rates were below those of the previous follow-up and below the trial protocol target. Since participants were recruited from 43 practices over the course of approximately one year, there was overlap between the 8- and 12-month follow-ups; the response rate to the 8-month follow-up could not be determined prior to the start of the 12-month follow-up. Moreover, the pre-calling intervention from Study 1 was adopted as a standard procedure for all participants at the 12-month follow-up. This latter strategy, coupled with the multiple other response-boosting tactics already employed in the host trial, might have accounted for the better than expected response rate at 12 months.
Other studies have also failed to achieve an improved response rate, but did similarly enhance completion rates. In a recent study that included pre-notification calls as an embedded intervention within the host trial, there was a small, but non-significant effect on response rates [
25]. Interestingly, however, the pre-called group had a higher response rate at the next scheduled follow-up, suggesting a carry-over effect from the previous telephone contact. It is possible that this occurred in the current results of Study 2, since the same group of Bristol-based participants who received a pre-call at the 8-month follow-up were involved in the team photo study at 12 months. This might have diluted any potential effect of the photograph intervention. Indeed, 48 % of pre-called participants required a reminder and took about 14 days to return the questionnaire in Study 1 (versus 62 % and 22 days, respectively, for control participants), which closely aligns with the figures for both the intervention (49 % and 14 days) and control (46 % and 13 days) groups in Study 2. Furthermore, another study demonstrated a similar carry-over effect for those who received a questionnaire with a colour photograph compared with those who received the black and white version in two waves of subsequent questionnaires [
14]. Since the photograph intervention occurred at the final follow-up in the Healthlines host trial, it is not possible to examine whether this was also the case in our study. It is, nonetheless, plausible that these latter less resource-intensive strategies in isolation could be as effective at bringing about similar responding benefits as the more effortful pre-calling tactic. In a second example, Ashby and colleagues [
26] reported a non-significant increase in response rates of participants who were sent an email or text message (or both) just after being posted their next questionnaire, but did observe a faster response time. Completion rates in this embedded trial (89 % overall) approached ceiling levels, just like our follow-up time points. Perhaps the previously documented benefit on completion rates of pre-notification, as well as the inclusion of a photograph or email subject line intervention [
9,
10], applies to studies with poorer response rates to begin with.
As we noted earlier, different response rate strategies might bring comparatively different benefits and costs, in which a trade-off exists between the effectiveness of response rate strategies and the resources required to implement these tactics. In line with previous reviews [
9‐
12], the results of the current studies suggest that the resource-intensive strategy – pre-calling participants – was more effective than the more easily implemented distinctiveness-enhancing strategies (including a personalized photograph and using an urgent action email subject line). However, there are two related, but opposing, issues to consider. First, despite the reduction in reminders and faster completion time, pre-calling participants probably did not result in an overall net benefit, since additional researcher time was required to contact these participants by telephone prior to sending out the questionnaire. Although we did not actually measure these additional researcher costs, the majority of the participants in this study required multiple telephone attempts to establish contact, sometimes outside of normal working hours. It is likely, therefore, that the effort expended at the outset to pre-call participants offset the reduction in questionnaire reminders. It would be worthwhile for future studies to quantify such cost-benefit trade-offs. Second, given the elaborate reminder protocol adopted in the main trial, the researchers perceived a substantial benefit from the reduction in the number of questionnaire reminders required. Not only did the reminder protocol result in a heavy administrative workload, but researchers also disliked the feeling of chasing and nagging participants to complete questionnaires; a feeling that they perceived to be shared by the participants themselves. Therefore, while it was laborious to successfully complete the pre-notification telephone call, the researchers felt that the benefits of this outweighed the later reduction in questionnaire reminders.
Strengths and limitations
There are two key advantages shared by these three studies. Firstly, we employed an embedded study design, in which the response rate interventions occurred within an ongoing, complex, pragmatic trial (The Healthlines Study). Bower and colleagues [
27] note that embedded studies are ‘the most robust test of the effectiveness of a recruitment or retention method’, since they permit less biased and more externally valid evaluations of such strategies, but acknowledge that these studies are quite rare. A second shared advantage is that the current studies include a joint assessment of responses to both postal and electronic questionnaire completion. Thus, while it remains an open question, whether the results of some previous response rate intervention effects were replicable across different questionnaire completion methods, our results hold across paper-based and online surveys.
A number of limitations of these studies should be taken into consideration. Firstly, owing to the practical constraints of the embedded study designs, we did not calculate power calculations
a priori for these three studies. It is, therefore, possible that the studies did not include an adequate sample size to detect differences in effects. This is especially the case in Study 3, in which only 61 % (141/231) of randomized participants required a study reminder. Yet, both intention-to-treat and sensitivity analyses, whereby only participants exposed to the treatment allocation – a condition that is independent of group allocation, and so unlikely to introduce bias – revealed the same pattern of findings. In a similar embedded study, the authors calculated
post facto that 4,000 participants would be required to detect a small but significant response rate effect – a figure that they state would be difficult for studies to achieve [
27]. However, as these authors recommended, publishing such embedded studies will enable future meta-analyses. Secondly, our results might not be generalizable to other kinds of trial or patient population. Unlike the majority of previous trials, participants in the current studies were given the option of completing either postal or online questionnaires, according to their preference. This might have contributed to the high response rates we observed. In addition, participants who volunteer to take part in a telehealth trial might differ from other patient populations in terms of their accessibility to and confidence using technologies [
28]. Finally, the pre-calling study used alternate allocation of participants to the two study arms, which was, therefore, not truly random. While this is a methodological drawback, Table
1 clearly shows that the two groups were very well balanced across all characteristics, and comparably so with Studies 2 and 3, which did make use of simple randomization.
Acknowledgements
This paper summarizes independent research funded by the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research (Grant Reference Number RP-PG-0108-10011). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.
These embedded studies form part of The Healthlines Study research programme, carried out in partnership with and hosted by NHS Direct and Solent NHS Trust. The Healthlines Study is comprised of a large research team from the Universities of Bristol, Sheffield, Manchester, Southampton, University College London, and from the Royal College of Surgeons in Ireland, and we thank the members from these institutions who are not otherwise authors on this paper. We would like to thank all of the participating GP practices and patients recruited from these practices for taking part and making this research possible. To this end, we are grateful to the Primary Care Research Network (now, NIHR Clinical Research Network,
http://www.crn.nihr.ac.uk/) for assisting us with GP practice recruitment for the trials. We also acknowledge the contributions to intervention development and delivery of Professor Hayden Bosworth and Felicia McCant (Duke University), Professor Chris Williams (University of Glasgow), Jen Hyatt (Chief Executive, Big White Wall), Steve Bellerby and the rest of the NHS Direct IT team. This study was designed and delivered in collaboration with the Bristol Randomised Trials Collaboration (BRTC), a UKCRC Registered Clinical Trials Unit in receipt of National Institute for Health Research CTU support funding.
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
LE co-devised Study 1 with CS and KG, designed and carried out Study 2, conducted all analyses and drafted this manuscript. KG carried out Study 1, while KH and AF co-devised and carried out Study 3. AAM provided advice on the statistical analysis of the data. CS provided overall guidance for The Healthlines Study as chief investigator. All authors commented on the draft, leading to the final manuscript submission. All authors read and approved the final manuscript.