A randomized trial of mailed questionnaires versus telephone interviews: Response patterns in a survey

The allocation sequence for the randomization was generated by a computer assistant using the SAS routine RANUNI and a subsequent sorting procedure. The allocation lists were delivered to the head of the project.

Neither participants, nor surveyors or assessors were blind to group assignment.

Those randomised to receive a mailed questionnaire were sent one at their current address. Non-respondents received two postal reminders at intervals of three weeks. The second reminder included a questionnaire. For the group randomised to telephone contact, phone numbers were searched electronically in the database of the national phone company and manually in additional databases. Persons with a confidential phone number were sent a letter with an invitation to call back. The telephone interviews were computer assisted; they were preceded by a letter and were primarily conducted Monday-Thursday between 4 pm and 9 pm and Sunday between noon and 4 pm. The interviewers were student assistants and paid by the hour. They received a short course, a written manual and conducted three test interviews. The interviewers were instructed to accept item refusals and act neutral, but whether they complied completely with these instructions is uncertain, just as it is uncertain whether all response categories consistently were read out. Non-respondents were followed up with at least three calls at different times of the day/evening. Those unwilling to participate in a telephone interview were offered a mailed questionnaire. The latter are treated as non-respondents here. In both of the randomised groups, the study was presented as concerning health, well-being and, for those having a job, the psychosocial work environment. No incentives were used in either of the two randomised groups.

The questionnaire was 27 pages long and contained 257 items. The duration of telephone interviews was typically 30–45 minutes (cf. English translation[13]). This paper deals with the 47 items concerning health and well-being that all respondents were asked, irrespectively of their labour market status. These 47 questions are on the first 8 pages of the questionnaire. The questions have between three and six response categories and are primarily from the Stress Profile[14] and the SF-36 questionnaire[15]. The order of the questions was the same as in additional file 1: Table A.

The study has been notified to and registered by the Danish Data Protection Agency (Datatilsynet). According to Danish law, questionnaire based studies do not need approval by ethical and scientific committees, nor informed consent.

The original protocol for the survey did not specify any objectives or hypotheses (or primary and secondary outcome measures) concerning the effect of mode of data collection. We thus tested the hypotheses that we identified in the literature and listed in the introduction, and we estimated any differences. Questions concerning loneliness, well-being, self-esteem, unhappiness, nervousness, worry, depression, or trouble sleeping are mentioned in the literature as sensitive[1, 7, 16]. We categorised our 47 items as belonging under one of the four mentioned self assessments: well-being, self-esteem, depression or the additional heading stress, or under one of two behaviours: smoking habits and use of medicine. Thus all 47 questions are either sensitive or concerning socially undesirable behaviour. The categorization appears from the additional file 1: Table A (cf. the "Theme" column) and 2 and the analyses are presented accordingly. The distinction between self assessments and behaviours were added post-hoc.

We used the chi-square test to test the hypothesis of telephone interviews resulting in higher response rates than mailed questionnaires (Hypothesis 1) and logistic regression to assess differential bias between mode of data collection, age and gender; marginal associations between mode of data administration and demographic data were examined using the chi-square test of independence.

We tested the hypotheses of higher rates of missing responses among questionnaire respondents than among telephone respondents (Hypothesis 2) and of greater willingness to answer sensitive questions in questionnaires than in telephone interviews (Hypothesis 3) by Fishers exact test (two-sided for all items since both hypotheses are relevant for all items).

We used the chi-square test to test whether telephone respondents had a preference for extreme (i.e. the highest possible and the lowest possible) response categories compared to questionnaire respondents (Hypothesis 4) and whether the level of reported health and well-being differed according to mode of data collection (Hypothesis 5) (The null hypotheses of these hypotheses are the same, while the alternative hypotheses differ). The tendency to use the extreme response categories among phone respondents relative to mail respondents was quantified by the odds ratio for extreme replies versus the pooled middle categories, and the tendency to reply more positively was quantified by the odds ratio estimated in the proportional odds model. Responses in the "don't know" category were treated as missing values.

The same statistical tests are performed on 47 items so the traditional 5% level of statistical significance is replaced by a 0.1% level, i.e. a Bonferroni correction of the 5% level[17].

Participants were sampled from the CRS early August 1997 and randomised to mode of data collection August 18^th. Questionnaires were sent out around October 1^st and telephone interviews conducted concurrently until January 1998. Data on mode of data collection, gender and year of birth were available for the entire sample and indicated a successful randomization (Table 1). The flowchart indicates reasons for non-response.

Overall, the response rates for the two modes of data collection were similar; 58.1% among persons randomly assigned to questionnaires and 56.2% among persons randomly assigned to telephone interviews (p = 0.26). However, some age and gender differences were hidden (Table 2). Women and persons in their thirties were more willing to respond to the mailed questionnaire. A logistic regression analysis confirmed these interactions between mode of data collection and respectively gender (p = 0.001) and age (p = 0.01) (Estimates not shown). The specific difference for women in their thirties (69% versus 50%) was even larger than shown in Table 2. Only persons responding by the mode to which they were randomised, were included in these analyses; cross-over respondents were treated as non-respondents.

Self reported demographic data were compared for the two types of respondents. No differences were found in the marginal distribution of schooling (p = 0.13), labour market status (p = 0.43), number of children living at home (p = 0.20), or number of children under the age of six living at home (p = 0.77). Significant differences were found for vocational training (p = 0.004) and cohabitation (p = 0.001). Among telephone respondents, there was a larger percentage of persons with a further and higher education than among mail respondents (11.8% versus 7.3%) and a larger percentage of persons who had always lived alone (9.1% versus 5.3%).

For all items there were more missing values among questionnaire respondents than among telephone respondents (Table 3). For smoking habits the difference was non-significant; for most of the self assessment questions and for use of medicine the difference was significant (Additional file 1: Table A). The lowest rate of missing replies (around 1%) in the mailed questionnaire is consistently achieved for stand-alone-questions (e.g. smoking habits), an intermediate rate (around 2%) is consistently achieved for the first question in blocks of questions with common introduction, and the poorest rate (3 – 6%) for the remaining questions.

The "don't know" category was used much more frequently among questionnaire respondents (Table 4). Being the middle response category, the "don't know" option might also be regarded as a neutral response, at least in the mailed questionnaire.

For the health behaviours we found no significant differences in the response patterns (Table 5).

For the 41 health self assessment items the response pattern was statistically significant different for 32 items (Additional file 2: Table B). All significant differences were in the same direction with regard to extreme responses (Hypothesis 4): more use of the extreme responses in telephone interviews. The median estimated odds ratio for the 32 items was 1.67 for more extreme responses in telephone interviews. The median estimated odds ratios within each theme were in the range 1.55 – 2.12 and the largest estimated odds ratios were in the range 2.20 – 2.57 (Table 5). Odds ratios above 1.2 were estimated for the six 6 "depression" and "stress" items that did not differ significantly between the modes. The pattern was slightly more complex regarding the tendency for positive responses (Hypothesis 5): of the 32 significantly different response patterns, 26 were shifted toward a more positive reporting among the telephone respondents (Additional file 2: Table B). The median estimated odds ratio for the 26 items was 1.73 for more positive responses in telephone interviews. The median odds ratios (within each theme) were in the range 1.60 – 1.99 and the largest odds ratios were in the range 1.82 – 2.97 (Table 6). Odds ratios above 1.2 were estimated for the six 6 "depression" and "stress" items that did not differ significantly between the modes. Of the six items shifted in the opposite direction, four were grouped under the self-esteem heading.

Adjustment for the observed differences in gender and age response rates did not alter the magnitude of the estimates appreciably (Data not shown).

If the "don't know" category was analysed as the neutral middle response, the estimates were increased and the conclusion was strengthened (Data not shown).

No differences were found in the overall response rate but a breakdown on age and gender showed that women and persons in their thirties had a higher response rate to the mailed questionnaire compared to the telephone survey. The rate of missing responses is clearly higher among questionnaire respondents than among telephone respondents. In relation to questions on behaviour, no difference between the two modes of data collection was found in the use of extreme response categories and no shift in the level was found between the modes. In relation to questions involving self assessments, telephone respondents had a strong preference for extreme response categories compared to questionnaire respondents and most items were shifted towards a more positive reporting among the telephone respondents.

When it comes to utilizing extreme response categories and to shift in level of response, the odds ratio estimates were above 2 for several items, i.e. as large as many effects studied in epidemiology.

The unique opportunity in the presented data set to study mode sensitivity on a range of questions has so far been unused. The difficulties we met trying, retrospectively, to construct a flowchart and to verify the coding of variables emphasise the importance of completing these tasks concurrently with conducting the study[18]. These deviations from the ideal, however, do not seem to cause bias in the conclusions of our study. Our categorisation of individual items under the themes mentioned in the literature as sensitive might have been done differently by other researchers. However, we have preferred to maintain our pre-analysis categorisation. We do not suspect this categorisation to have any major influence on our results.

To counteract the multiplicity of analyses and outcomes we applied a conservative Bonferroni correction of the p-values. Nevertheless, the vast majority of statistical tests were significant and the emerging pattern of differences was fairly consistent.

The sample was a stratified sample of an adult population. The results are thus fairly generalizable compared to samples of more specific subgroups. However, the analysed questions concerned only health behaviour and self assessment items so generalization to other types of questions should be done cautiously.

Compared to most previous studies, our randomized design is strong.

It has previously been reported both that mailed questionnaires resulted in a lower[2, 6, 8], a higher[4], or the same[7] response rate as telephone interviews. The response rate will probably depend highly on the specific strategy and persistence in each trial arm[19]; the telephone response rate may also dependent on administrative accessibility of phone numbers and changes in phone technology.

The highly significant difference[4, 7‐9] and the missing rate of almost nil among the telephone respondents[4, 8] are in accordance with other findings.

The cognitive demands on telephone respondents are higher than the cognitive demands on questionnaire respondents. Questionnaires enable the respondent to control both the pace and the order of interview[7]. A tendency for telephone respondents to prefer extreme response categories has also been reported in other studies[7, 8] that primarily covered self assessments.

Due to the few missing responses among telephone respondents, the hypothesis of greater willingness to answer sensitive questions and questions concerning socially undesirable behaviour in questionnaires than in telephone interviews could not be confirmed by our data.

A majority of the questions considered here are sensitive according to the literature, and if interviewers are instructed not to accept item refusals it may be at the cost of sensitive questions being answered randomly or perhaps overly positive.

As already mentioned, most of the self assessment items were shifted towards a more positive reporting among the telephone respondents. However, the pattern was slightly more complex, since six of the mode-sensitive items shifted in the opposite direction. These six items all appear in blocks of questions formulated as statements with response categories indicating how well the statement fits the respondent's situation ("Correct", "Almost correct", "Somewhat correct", "Only slightly correct", "Incorrect"). Some statements are positive (E.g. "I feel I understand most of what is going on in my everyday life") and some are phrased using a negation (E.g. "So far, I have not had any clear direction or purpose in life"); five of the deviating items included negations and four of these concerned self-esteem. In Danish, the questions concerning self-esteem are strikingly longer and more long-windedly phrased than the rest of the questions. The long-windedness, the mix of positive statements and statements using negations, or both might explain the deviation from the overall pattern. However, these are post-hoc speculations.

According to the theory of response order effects, a preference for the first response categories is to be expected in visually presented questionnaires whereas a preference for the last mentioned response categories is to be expected in telephone interviews[20]. A shift in response pattern supporting this theory was observed for half of the items. (This theory coincides with the hypothesis of a shift toward a more positive reporting among the telephone respondents for questions using a negation and coincides with the hypothesis of a shift toward a more negative reporting among telephone respondents for questions positively phrased). Thus the mechanism of more positive reporting among the telephone respondents is stronger in our study than the proposed response order effect.

In our study response patterns were similar for questions on health behaviour regardless of mode of data collection; whereas, response patterns were significantly different for questions on health self assessments. This finding was not one of our pre-specified hypotheses so an independent test of our finding would be welcome. Also similar tests and quantifications of mode sensitivity in a wider set of item types, e.g. items unrelated to health issues and items including opinions and values would be useful.

Costs were not booked in our study but ought to be booked in future studies. It is important additional information in decision making when comparing the two data collection methods in relation to differences in response rates, patterns of missing responses and response patterns and when optimizing reminder procedures. Calibration of the measurements from the two modes of data collection is also an important issue. This has to some extent been dealt with elsewhere[21].