Maximising response to postal questionnaires – A systematic review of randomised trials in health research

Randomised trials of methods of improving response to postal questionnaires in health care research were identified. Seven electronic bibliographic health care and medical databases were searched for relevant trials (Table 1). The reference lists of identified trials and reviews were also searched. Authors of relevant trials and reviews were contacted to identify unpublished trials. Selected journals were hand searched. The BMJ 'Cite Track Alert' service [13] was used to alert for articles citing the most recent relevant review [1] and the 'Biomail' Medline search service [14] was used with the search terms of ('clinical trial') and ('follow-up' or 'questionnair*'). There were no language restrictions.

All identified randomised trials of any method of improving response to postal questionnaires in a health care context were evaluated for study inclusion. 'Health care research' is defined as the questionnaire being used in a clinical trial, survey or observational study of health state and containing questions relating to aspects of a person's physical, mental or social well-being (based on the WHO definition of health[15]). Only studies that recruited patient populations were included. A 'patient' is defined as a person who is receiving medical or surgical treatment [16]. Studies in which participants were recruited via GP patient lists but were not actively receiving medical treatment were excluded. A list of excluded studies is available from the authors. The criterion to assess the effect of the interventions was a comparison of the percentage of questionnaires returned after all follow-up efforts. All potentially relevant studies were checked for study quality independently by two reviewers.

Quality assessment was based on recommendations in the Cochrane Reviewers Handbook [17] and a Delphi List of quality criteria developed by Verhagen et al[18]. Where aspects of quality were unclear from the report the authors were contacted for clarification.

Data were extracted independently by two reviewers using a standard data extraction form. Details extracted included the country, main study method, patient characteristics, intervention used to improve response, number of participants randomised to the intervention and control groups and response rate in terms of number and percentage of questionnaires returned and procedures for follow-up. Where insufficient data were reported the authors were contacted for clarification. When studies used more than two categories to evaluate an intervention, (for example short, medium and long questionnaires) a dichotomy was created by combining the categories that were most similar. When this has been done it is indicated on the Data Extraction table (Additional file 1).

The results were pooled into sub-groups of similar interventions. The data were analysed using the Cochrane review manager software (RevMan version 4.2; Oxford, UK). We used the Mantel-Haenszel method to calculate the pooled odds ratios (OR) for binary outcomes for each strategy. This fixed effect method based on a weighted average of the results was used to combine studies. A sensitivity analysis was carried out by re-analysing the data using a random effects model. For all estimates we calculated 95% confidence intervals (CI 95%). Statistical heterogeneity between trials was assessed with χ² tests using P < 0.10 to reflect significant heterogeneity and the percentage of variation across the studies was measured using the I² statistic [19]. Publication bias was investigated using a funnel plot.

We identified 13 randomised trials including 25607 participants that fulfilled our inclusion criteria [20‐32]. Figure 1 gives a flow chart summarising the study selection process.

The studies evaluated five different methods of enhancing response to postal questionnaires. These methods were: questionnaire length, incentives (cash, prize draw, lottery or phone card), question order, reminder strategies and including an information brochure with the questionnaire. One paper reported results in two distinct patient groups (angina and asthma) and these are presented as separate studies [28]. Another paper described two separate interventions (questionnaire length and incentives) and these are also reported as separate studies [24]. Six papers contained information regarding missing data from the returned questionnaires [20‐23, 28] but used different interpretations of missing data. All the studies incorporated their randomised trial of methods of improving response into an existing research study. The majority of the studies nested their trial of enhancing response within a patient survey. None of the studies nested their study of methods of improving response into a randomised clinical trial. Additional file 1 gives details of extracted data. Five studies were deemed to be of 'good' quality, six were 'moderate' quality and quality was unclear from the report of four studies. See Table 2 for details of quality assessment.

Figure 2 shows the pooled odds ratios and 95% confidence intervals for the five different strategies investigated for improving response rates. Reminder systems had the most significant effect on response rates (OR 3.71, CI 95% 2.30 to 5.97 p = <0.00001) with more intense methods improving response by an average of 24%. Shorter questionnaires improved response rates but to a lesser degree (OR 1.35, CI 95% 1.19 to 1.54 p = <0.00001) with an average improvement in response of 9%. 'Shorter' questionnaires ranged from seven to 47 questions and 'longer' questionnaires ranged from 36 to 123 questions. The studies investigating questionnaire length compared two or more questionnaires. We used the authors own categorisation of 'shorter' and 'longer' questionnaires. The use of incentives (OR 1.09, CI 95% 0.94 to 1.27 p = 0.24), re-ordering of questions (OR 1.00, CI 95% 0.91 to 1.09 p = 0.92) and including an information brochure with the questionnaire (OR 1.04, CI 95% 0.94 to 1.16 p = 0.42) had no significant effect on response rates.

There was no evidence of significant heterogeneity between the trials in each intervention group. Sensitivity analysis using a random effects model gave virtually identical overall estimates of effect.

Losses to follow-up in health care research can have serious effects on study validity [33]. A recent Cochrane review [1] identified 292 eligible randomised trials of methods of improving response rates to postal questionnaires. The review concludes that methods such as unconditional incentives (ie incentives given regardless of whether the questionnaire is actually returned), shorter questionnaires and "user-friendly" questionnaires can substantially improve response rates. Caution should be taken when interpreting the findings of this review in a health care context as the majority of the included trials had no health care connection. The motivation of a patient to respond to a follow-up questionnaire in a health care study might differ from that of a member of the public selected to receive a general survey questionnaire. Tactics to encourage response may therefore differ. Health care study participants are actively involved in the research process and are often motivated by the potential health benefits associated with the study. Conversely, the amount of trauma and discomfort produced by the study treatment or procedures affects the willingness of the patient to remain under follow-up [34]. Ludemann et al. [35] found that patients in a clinical trial of laparoscopic fundoplication were less likely to respond to postal follow up if they had a poor outcome from the surgery. Saliency of a questionnaire to the recipient has been shown to be one of the strongest predictors of response. A salient topic is defined as "one which deals with important behaviour or interests that are also current" [36]. It is likely that a participant in a health care study receiving a questionnaire regarding their response to a treatment intervention, or their views on a therapeutic encounter, would find the questions highly salient. Response rates to non-salient questionnaire surveys of the general population rarely exceed 50% [37]. The average response rate across the included studies in our review (excluding two studies that only randomised non-responders to previous follow-up methods) was 65%.

Three studies investigated methods of follow-up to improve response [29‐31]. Although the methods of follow-up differed, all of the trials compared a more intensive follow-up procedure with a standard method. The three included trials compared telephone, postal or recorded delivery reminders compared to usual follow-up efforts. We therefore carried out an analysis of intensive versus usual follow-up. The results suggest that increased intensity of follow-up effort may improve response rates, but the differences between the interventions of the studies in this analysis mean that the result should be treated with caution. In one study [29] the use of telephone reminders appeared to be less effective than recorded delivery postal reminders. However, in another study [30] telephone reminders appeared to be more effective compared to normal delivery postal reminders alone. One of the studies had a very small sample size [30] but excluding this study and re-running the analysis had little effect on the results. Clinical researchers need to incorporate appropriate follow-up strategies within the budget constraints of their research activities. Due consideration for the patients' privacy is needed, however, to ensure that patients do not feel harassed by the follow-up efforts. Further research is required to determine the acceptability of repeated contact to the patient.

A recent review focuses on the effect of questionnaire length on response [38]. Out of twenty seven included trials, fourteen (52%) studied health related topics but only four (15%) studied patients rather than members of the general public. The authors extrapolate that shorter questionnaires should be used in clinical trials to improve response. Since none of the included studies looked specifically at clinical trials, such extrapolation should be viewed with caution. Our findings confirm that shorter questionnaires improve response in the health care setting. Questionnaires are often used in health care research to answer a research question. There is, however, an inevitable trade off between making the questionnaire comprehensive enough to answer the question adequately, and making it so long that it has an adverse effect on response. Careful consideration of the minimum data required when designing the questionnaire is essential. As yet there is insufficient evidence to suggest an optimal questionnaire length in terms of number of questions or pages.

Previous reviews looking predominantly at market research found incentives to be a useful way of improving response [1‐3]. The largest effect sizes are seen with monetary incentives. The use of incentives in health care research in Europe is uncommon. Trials often have strict budget constraints making the provision of incentives an unacceptable additional cost. Providing incentives in health care research can also raise ethical concerns [39]. The health care study participant may view their personal input into the study as the motivator to respond rather than merely responding to an incentive. This review has shown no evidence that incentives are effective in the health care context. This is an area, however, which requires further investigation. The studies included in this review used either small monetary incentives or monetary equivalent incentives (lottery ticket, prize draw or phone card). None of the studies investigated non-financial incentives such as pens. The inclusion of an incentive appropriate for the particular study may have a positive effect on response but this has not been tested. Until this area is investigated more fully no recommendations can be made on including incentives in health care research as a method of improving response.

Question order appeared to have little effect on response rate. The three studies looking at question order, however, investigated two different approaches. One study compared a traditionally ordered questionnaire with a chronologically ordered one [22] and the other two studies compared placing condition specific questions either before or after generic questions [28].

This review was strict in its definition of a 'patient' and excluded studies which were in the health care setting but involved the general public. It was anticipated that more studies would be found involving patients. The evidence available on which to base conclusions was therefore limited. The review could be repeated including health care research studies of the general public to give a broader perspective of methods of improving response in the health care setting. Previous studies have investigated this area evaluating methods of improving response such as postage stamps [40] and questionnaire length and incentives[40, 41]. The market research literature has investigated many methods of improving questionnaire response. Edwards et al. [1] grouped these methods into the following strategies: Incentives, Questionnaire length, Appearance, Delivery, Contact, Content, Origin and Communication. All these methods need to be tested on patients in the health care setting before extrapolations of their usefulness can be made. All of the trials included in our review looked at the effect of an intervention in isolation of other interventions. Future studies could use factorial designs to investigate the addition of different methods to improve response.

In any future research it is important that the methods of improving response are well documented and tested in situations that reflect their intended use ie patient populations in health care studies. The effects of the interventions on completeness of the returned questionnaires also requires investigation.