Response rates in postal surveys of healthcare professionals between 1996 and 2005: An observational study

Studies with low response rates may be less likely to be published in journals with space limitations. We therefore compared surveys published in Biomed Central, which publishes only electronically and has no restriction on article length, with surveys indexed in "standard" bibliographic databases Medline, Embase and Psycinfo. These databases were selected to give comprehensive international coverage and to include both medical and psychosocial disciplines.

The search strategy is detailed below.

#1 survey$ or questionnaire$.tw.

#2 clinician$ or dentist$ or doctor$ or family practition$ or general practition$ or GP$ or FP$ or gyn?ecologist$ or hematologist$ or haematologist$ or internist$ or nurse$ or obstetrician$ or occupational therapist$ or OT$ or pediatrician$ or paediatrician$ or pharmacist$ or physician$ or physiotherapist$ or psychiatrist$ or psychologist$ or radiologist$ or surgeon$ or therapist$ or counse?lor$ or neurologist$ or optometrist$ or paramedic$ or social worker$ or health professional$ or primary care).tw.

#3 1 and 2

Inclusion criteria are described in Table 1. Studies using a postal or electronic survey methodology (published during 1996-2005) were identified by searching the databases mentioned above. References were downloaded, duplicates were removed and Biomed references were excluded from the standard database set.

All references published in electronic media were screened for inclusion but references in the standard databases were sampled before screening. All screening was performed by JVC. We estimated that 272 studies were required to detect a change of 5% at significance p < 0.05, 80% power with a cluster size of 100 and an intracluster coefficient of 0.08 [12, 13], assuming a baseline response rate of 61% [2]. We assumed that only 1 in 7 references would fulfil the inclusion criteria and therefore screened 2000 references, which were randomly selected using computer generated sequences of 200 random numbers per year.

Data abstraction parameters are set out in Table 1. Number of survey participants, type of healthcare professional, questionnaire length, use of reminders and financial incentives were selected as predictors of response rate based on data from studies of health professionals and the general population. Publication type was chosen in order to examine the potential for publication bias. Country of study population was included as country specific factors such as healthcare system and net remuneration could moderate the effects of financial incentives. Non-response analysis was deemed present if researchers compared demographic variables between respondents and non-respondents, demonstrated sample representativeness or contacted a sample of those who did not reply.

We examined the effects on response rate of publication type, healthcare profession, country of study population, length of questionnaire, number of reminders and number of study participants using multivariable, multilevel logistic regression that allowed for clustering of healthcare professionals within studies [14]. The primary outcome measure was whether or not a healthcare professional had responded to a questionnaire; these individual responses were combined in the response rate for each study.

Multilevel modelling was used because the likelihood of response by different healthcare professionals within the same study is likely to be more similar than that by healthcare professionals in different studies. The multilevel model gives more weight to larger studies. It assumes that the response rate in a study with specific characteristics is sampled from a normal distribution and estimates the mean and variance of this distribution.

Initially, multilevel univariate logistic regression was performed, considering in turn each covariate, categorised as in Table 2. One third of the studies (108) did not provide information about length, so unreported length was considered as a separate category in the analysis. Covariates were selected for the multivariable model using a forwards stepwise procedure and the likelihood ratio test statistic (LRTS) to compare nested models. All variables significant in the multivariable analysis were tested for removal with a backwards step at each stage. To lessen the probability of chance findings due to multiple hypothesis testing, the p-value p < 0.01 was deemed to be significant for entry and removal of covariates. Categories within variables were collapsed if this made no significant difference. The final multivariable model excluded studies with missing values on the included covariates and was therefore based on 339 (96%) of the included studies. We checked for significant interactions between covariates in the final model. Odds ratios (OR) and their 95% confidence intervals (CI) are reported. From the final model, we estimated the overall mean response rate and its 95% confidence interval (which contains the actual mean with a probability of 0.95).

Cook's distance [15] was used to identify studies with undue influence.

From 123,538 references downloaded from the standard databases, 2,000 were randomly sampled; 277 fulfilled the inclusion criteria. Of the 494 references from Biomed Central, 75 fulfilled the inclusion criteria. The median number of participants in these 352 studies was 275 (interquartile range: 150 to 498).

Two very large studies, originating in Canada [16] and the US [17] with 61,751 and 51,672 participants respectively were excluded from the regression analysis because preliminary analysis indicated that they had undue influence on the results. The remaining 350 studies surveyed 365,490 healthcare professionals.

Two thirds of the included studies were in doctors, nearly a third did not report questionnaire length and a third required less than 10 minutes to complete (Table 2). Nearly all studies reported the number of reminders used: half did not use reminders. Thirty-five different countries were represented. Thirty seven percent of studies were based in the US and 23% were in the UK/Ireland. Other European countries accounted for 14% of studies; the only other countries with more than 10 studies were Canada (10%) and Australia/New Zealand (6%).

Various characteristics of studies were significantly correlated. On average: longer studies were larger and had more reminders (Spearman's rank correlation (ρ) = 0.22, 0.20; p = 0.0006, 0.002 respectively); US-based studies were longer and larger (ρ = 0.17, 0.18; p = 0.01, 0.009 respectively); studies in Europe were less likely to report the length of the study (ρ = 0.13, p = 0.02); studies of doctors were shorter (ρ = 0.21, p = 0.001). Among studies published electronically, three-quarters were published between 2004 and 2005 and all except two were conducted in Europe. Too few studies (3%) reported the use of financial incentives or a solely electronic design (3%) to allow further analysis of these factors.

The simple mean of response rates (giving equal weight to studies of all sizes) was 56% (95%CI: 54.4% to 58.3%). The median response rate was 59% (interquartile range 42.2% - 70.8%). Only 56 studies (16%) reported response rates over 75%.

Univariate logistic regression (Table 2) showed no significant difference in response rates between surveys of different types of healthcare professionals (p = 0.18), or between surveys published in standard and electronic format (p = 0.51). Survey response rates tended to be higher in studies with more reminders and those of unknown length and lower in the US, Canada, Australia and New Zealand and in larger studies. Adjacent categories were collapsed (Table 3) if their response rates were not significantly different. Multivariable logistic regression (Table 3) confirmed the associations found in univariate analysis, but with a less marked relationship between response rate and country, as the lower response rate in studies in the US and Canada was partly explained by the larger size of these studies. We found no statistically significant interactions between covariates. After allowing for these associations, substantial unexplained variation in response rates remained. Some variation is to be expected because of the effect of sampling within studies. However, the intra-cluster correlation coefficient (ICC = 0.16) indicated that most of the unexplained variation (84%) was between studies. In sensitivity analyses, each of the very large, excluded studies [16, 17] was included in turn in the final multivariable model; this yielded similar estimates of the effect of the included factors, but with more variation between studies.

As Cummings' study was restricted to doctors, we compared Cummings' results with our results for all studies in doctors (including the largest study [16]). The simple mean response rate to questionnaires mailed to doctors (giving equal weight to studies of all sizes) in our study (1996-2005) was 57.5% (95%CI: 55.2% to 59.8%), based on 237 surveys. For surveys of doctors published between 1986 and 1995, Cummings et al [2] reported that the simple mean response rate for doctors was 61.2%, based on 257 surveys, but did not report confidence intervals on this estimate. If we assume that the variation between response rates in Cummings' study was similar to that for surveys of doctors in our study, then the 95% confidence interval on Cummings' response rate would be 59.0% to 63.4%. A two-sided t-test indicated that our estimate and Cummings' estimate are significantly different (p = 0.02), confirming a small decrease in the mean response between 1986-1995 and 1996-2005.

Fifty-eight of 350 studies (17%) reported some form of non-response analysis. Thirty-three compared socio-demographic characteristics of respondents and non-respondents. Frequently reported characteristics were age and gender, but also reported were: workplace location (hospital, GP, community), setting (urban, rural), size (single handed, multiple partners) and individual characteristics e.g. speciality, affiliation with professional bodies or university, years since graduation. Eleven studies assessed sample representativeness by comparing respondents' socio-demographic characteristics with those of a national database or large national survey. Four studies conducted telephone/personal interviews in subsets of non-responders. Three studies examined differences between early and late responders. Finally, four studies used multiple strategies, and three studies claimed to analyse non-response but did not report how.

We updated Cummings' study [2] of response rates to questionnaires mailed to doctors in the ten years from 1986-1995 by considering the following 10-year period. We included healthcare professionals other than doctors, but found no significant differences in response rates between professional groups. Although we selected surveys from the major health-related electronic bibliographic databases - Medline, Embase and Psycinfo - we did not include a database that was specifically focused on nursing e.g. Cinahl, largely to ensure comparability with Cummings' study [2]. Unlike Cummings, we modelled the association between study characteristics and response rate while allowing for a propensity towards similar responses by healthcare professionals within the same study. Despite examining a core set of recognised variables, much of the variation between studies could not be explained by factors that we considered and further exploration would require extensive contact with authors. In addition, due to poor reporting in primary studies, we were unable to examine the influence of factors, such as financial incentives, mail delivery systems and importance of survey topic, that are known to influence response rates in the general population.

Only 16% of studies achieved the response rate of 75% or over which is often regarded as the acceptable minimum [7]. Although our study confirmed the general consensus that reminders are an effective strategy to augment response rates [8, 9, 18], it was surprising that half the studies did not use any reminders. Even in the most favourable circumstances - studies outside the US, Canada, Australia and New Zealand, with reminders and less than 500 participants - the average response rate was only 65.5%.

It is unclear why larger studies and studies conducted in the US, Canada, Australia and New Zealand tended to have poorer response rates. The determinants of high response rates may be different in these countries and other countries; our study may not have captured the factors determining such differences. Smaller studies may have yielded higher response rates as they may focus more closely on issues salient to participants. The higher response rate in surveys that did not report length may be because surveys in Europe were less likely to report length but tended to have higher response rates.

Despite low response rates, only 17% of studies attempted any sort of assessment of the potential for non-response bias - a figure virtually identical to that seen by Cummings et al. [2]. Non-response analysis may be difficult and expensive: it requires assessing whether non-responders would have answered the questionnaire differently from responders, in some systematic way. If we assume that the propensity to non-response depends on the known characteristics of participants - e.g. basic demographic characteristics such as age, gender, profession - we can infer how non-responders would have answered, based on how responders with those characteristics did so. However, obtaining information even about such basic characteristics of non-responders may be problematic. If non-response depends on unknown characteristics of participants, then it is much more difficult to say how non-responders might have answered the questionnaire. In particular, if the reason for non-response is associated with the outcome of interest, bias will be inevitable. If this is suspected, it may be helpful to perform sensitivity analyses to explore how different assumptions about the determinants of non-response influence the conclusions [6]. This can be done either by explicitly modelling the probability of non-response, ideally using prior information from experts, or by "multiple imputation": replacing missing data by values randomly selected from a plausible distribution that reflects the postulated bias. However, all these methods of allowing for non-response are essentially informed guesswork and cannot compensate for the definitive knowledge provided by high response rates.