The feasibility of web surveys for obtaining patient-reported outcomes from cancer survivors: a randomized experiment comparing survey modes and brochure enclosures

Central cancer registries, which are mandated to collect data on all reportable cancer diagnoses within a defined geographic area for public health surveillance purposes [1‐3], are an important resource for researchers interested in ascertaining and recruiting individuals who have been diagnosed with cancer [4‐11]. Central cancer registries in U.S. states are population-based—that is, they are required to meet standards of complete ascertainment of incident cancers [12, 13]—and thus provide unbiased sample frames of cancer survivors in their catchment areas [14, 15]. These registries have been utilized for a variety of studies, including obtaining patient-reported outcomes and assessing quality-of-life among cancer survivors [16, 17], investigating etiology [18], identifying outcomes of treatment [19], and promoting preventive behaviors [20].

There is some concern about the potential for low participation rates when surveying through cancer registries [21, 22]. Generally speaking, survey response rates have been declining over time [23‐28], and there is some evidence that lower response will result in more nonresponse bias [29]. Numerous studies have documented concerns about demographic differences between those who do not respond and those who do participate [22, 26‐31]. In a recent evaluation of 10 years of recruitment efforts conducted via a central cancer registry, we found that a number of study-related and individual demographic variables predicted response outcomes [32]. Some studies have utilized randomized designs to evaluate the effect of survey administration methods used by registries on response, including sending the questionnaire in the initial recruitment packet rather than first obtaining consent [33], questionnaire length [7], type and amount of incentives offered [6, 7], and inclusion of phone contacts in addition to letters [9].

Generally speaking, web surveys have become popular for the advantages they offer in terms of lower costs, quicker data collection, automatic data entry, and the ability to require responses to all questions. However, across a variety of populations, response rates to web surveys have long been lower than those to paper surveys [34‐39]. The feasibility of web-based surveys in registry-based research has not been evaluated, which may in part be due to the fact that registries do not routinely collect email addresses of cancer patients. However, survey researchers have increasingly adopted alternative strategies for administering web surveys when email addresses are not available. One such strategy that has been used in a variety of populations is the web-push design, which uses postal mail to contact sample members and encourage response to a web questionnaire, while withholding a paper response option until later in the survey cycle [40]. Given recent research showing the success of this approach, as well as the potential web surveys have for increased data quality and quicker data processing, we sought to assess the feasibility of a postal-mail administered web survey to collect patient-reported outcomes via a central cancer registry. Using a randomized design, we compared response to a web survey to that of a paper survey in a population of individuals diagnosed with cancer ascertained via a cancer registry and examined the demographic profile of respondents for each mode.

As a secondary research question, we also assessed the effect of including an informative brochure describing the cancer registry on response outcomes. Such brochures, which explain how a person’s name was obtained for the study, are required by some registries when contacting cancer survivors for research recruitment. Our registry has not previously utilized such a brochure, so we aimed to examine its effects on response in order to inform future procedures and provide guidance to other registries for maximizing recruitment outcomes.

Twenty-four individuals were determined to be ineligible for the study after randomization and were not included in any further analysis. Reasons for ineligibility included: individuals we later learned had been deceased at the time of study selection, cases sampled shortly after diagnosis that were later determined (after the registry’s standard case coding and editing process) to not have been diagnosed with a reportable cancer, and individuals ineligible for study inclusion according to registry policy for contacting cases (e.g., individuals who had previously requested that the registry not contact them regarding participation in research studies). This resulted in a working sample of 446 cancer survivors.

Table 1 describes the eligible sample overall and by survey mode. Among all eligible sampled individuals, 56.5% were female, a majority were between ages 40 and 59 (66.6%), 6.7% were nonwhite, 15.5% were Hispanic, and 13.7% resided in rural counties. By cancer site, the eligible sample was 28.7% colorectal cancer, 14.4% multiple myeloma, 21.8% breast cancer, 21.1% prostate cancer, and 14.1% ovarian cancer. Local stage at diagnosis was most common (47.3%), followed by regional (25.6%), distant (21.8%), and in situ (5.4%). The samples within each of the experimental arms did not differ significantly in terms of any demographic (sex, age, race, ethnicity, geography) or cancer (time since diagnosis, cancer site, or stage at diagnosis) variables.

Two-hundred and nine of the 446 eligible individuals completed the survey, resulting in a response rate of 46.9%. Eleven individuals (2.5%) could not be contacted due to outdated contact information, 19 (4.3%) refused participation, and the remaining 207 (46.4%) did not respond. Figure 1 displays response outcomes at each stage of the contact protocol by experimental arm. Across all study arms combined, each subsequent contact yielded additional responses: 25.4% of responses were obtained after the first invitation packet, 31.1% after the reminder letter, 23.9% after the second/replacement invitation packet, and 19.6% after the final, phone call stage.

Table 2 displays response rates by experimental treatment, along with adjusted risk ratios and 95% confidence intervals for the outcome of survey response by each experimental treatment. For the comparison of survey mode, we found that the web survey response rate was 43.2%, compared to 50.4% for paper, but this difference was not statistically significant (RR = 0.88, 95% CI = 0.72, 1.07). The use of a brochure in the recruitment materials also did not significantly influence the proportion responding; 48.0% of those assigned to the brochure treatment responded compared to 45.8% of those not sent a brochure (RR = 1.03, 95% CI = 0.85, 1.25). Although not significant, the brochure appeared to have a more positive effect among the web arm (response rate was almost 5 percentage points higher) than the paper arm, in which response was unchanged.

Bivariate comparisons of the demographics of respondents to non-respondents for the full sample as well as by mode are displayed in Table 3. For the full sample, there was some variation between respondents and nonrespondents for several demographic variables. Individuals diagnosed within the year prior to selection had a higher response rate than those considered long-term survivors (51.6% vs. 40.6%, P = 0.022). Individuals aged under 40 were underrepresented among respondents (36.5% response rate) compared to older individuals; those aged 60 or older had the highest response rate of all age categories (58.7%; P = 0.028). The response rate was also significantly higher for non-Hispanics than Hispanics (49.2% compared to 33.8%, P = 0.020). There were no significant differences between respondents and nonrespondents in terms of sex, race, rurality, cancer site, or stage at diagnosis.

Among the paper arm, respondents and nonrespondents differed significantly only in terms of sex; 57.4% of females in the paper arm responded compared to only 41.2% of men (P = 0.016). However, this same trend was not observed for the web group, in which men and women responded at similar levels (42.3 and 44.3%, respectively). When comparing these response rates for females across modes, we found the response rate for females assigned to the paper survey arm was significantly higher than the response among females assigned to the web (P = 0.017). In the web arm, two variables showed significant differences between responders and nonresponders: race and ethnicity. Nonwhites and Hispanics were both underrepresented among respondents (P = 0.040 and P = 0.011 respectively).

Our multivariable assessment of the demographic representativeness of the responding samples and demographic-specific response rates by survey mode (Table 4) found few differences between modes. Adjusted risk ratios for response among the paper arm show the only significant predictors to be age 60 or above, with increased likelihood of response compared to age under 40 (RR: 1.78, 95% CI: 1.02, 3.06) and time since diagnosis (recently diagnosed RR: 1.36, 95% CI: 1.01, 1.84). The only variable significantly associated with response among the web arm in the adjusted model was Hispanic ethnicity (RR: 0.51, 95% CI: 0.27, 0.96). While we were unable to assess response outcomes by educational attainment because this information is not available in the registry database, we did collect educational attainment information in the questionnaire. The distribution of respondents by level of education was similar across study arms; 46.0% of paper respondents and 46.3% of web respondents reported having a bachelor’s degree or higher, 34.5% of paper and 28.4% of web respondents reported some college/associate’s degree, and 19.5% of paper and 25.3% of web respondents reported having a high school degree or less (P = 0.500).

In this randomized comparison of web and paper survey response outcomes in a study of cancer survivors ascertained through a central cancer registry, the overall proportion responding was slightly lower among the web arm. However, this difference in response rates was not significant. Considering that a recently-updated meta-analysis demonstrated that web surveys continue to yield response rates that are 12 percentage-points lower than other modes [39], the difference in response rates in this study (a seven percentage-point difference) was smaller than anticipated. This is a notable departure from longstanding trends showing web surveys obtaining much lower response rates than paper-based surveys [34‐38, 56] except in limited instances with specialized populations wherein Internet use may be more prevalent, including college students [57], physicians [58] [59], and volunteer samples recruited online [60].

Furthermore, unlike most prior research showing the demographic profile of web survey respondents is often much different and less representative of the target population than is found with paper surveys [61‐66], in this study the demographic representativeness of the responding sample members was mostly similar across survey modes. However, we did find that the oldest age group (65 or above) was more likely to respond than the youngest among the paper arm, but we did not observe this for web. This is consistent with prior research finding older individuals overrepresented among responders to a paper survey while web respondents are on average younger [63, 65] and that responders to web surveys are typically younger than those to a paper survey [64]. Due to the continuing relationship between age and web survey response patterns, using web alone to survey cancer survivors may not yet be advisable, especially since the larger population of cancer survivors is on average older than those included in this study, and only 44% of adults age 80 or above use the internet [67]. We also observed that Hispanics were less likely than non-Hispanics to respond the web survey, but this was not the case for the paper survey. This further suggests that adding a paper response option would be beneficial.

Our web survey was slightly different than most in that it was not email-administered. The use of a primarily postal mail-based contact protocol to encourage response may have been advantageous in helping to establish legitimacy and trust in the surveyor [40], but we did not test this directly. In the final phone call follow-up phase of this study, we offered nonrespondents in either study arm the option to respond over the telephone, but only 3 individuals completed the survey using this method. A similar approach that uses a paper response option for nonresponders to a web survey, the web-push design, has proven effective in samples of the general public as a way to collect a majority of responses online while also providing an option for those unable to respond using the internet [63, 64]. There is some evidence that this approach may even be more effective than paper-only in certain populations who are accustomed to receiving similar communications from the surveyor via email [68]. Further, in a study of Dutch childhood cancer survivors, various strategies of offering a paper-based alternative to a web survey produced similar response rates [69]. Thus, using a paper follow-up response option to a web survey administered by a cancer registry may be an effective approach to obtain responses from those reluctant to respond to a web survey.

Overall, we did not find evidence that sending a brochure describing the cancer registry encouraged significantly more people to respond. However, the results suggest it could have varying effects across survey modes; while response to the paper survey was unchanged with its introduction, response was slightly higher (but not significantly so) for the web survey, thereby reducing the gap in response between the two modes. This also could be related to establishing legitimacy and trust, which may be especially helpful when asking people to respond online. Responding online entailed manually typing an unknown web address and entering an access code, and could possibly have been viewed with more suspicion or hesitation. There has been mixed evidence regarding the effect of sending study-specific brochures on study participation. Some studies have found no significant effect of brochures on response rates [70‐72]. A recent analysis of multiple studies recruiting via the Utah Cancer Registry found that inclusion of a brochure describing details of the study in the recruitment packet decreased study cooperation [32]. However, study-specific brochures are very different in nature than the type used in this study, so it is unclear whether these past results are informative for cancer registry-specific informative brochure use. Due to the relatively small effect size of the brochure in this study, it would be worthwhile to retest this comparison in a larger sample size to further evaluate its effect across modes. It is worth noting that many registries require enclosure of such brochures in research recruitment mailings in order to explain how a person’s name was obtained. While our registry does not require a brochure, for registries that do, future testing may be made more informative by concentrating on variations in brochure design and contents rather than whether it is included or not.

In the comparison of individual demographics of respondents to nonrespondents, for the overall sample we found differences by time since diagnosis, age, and ethnicity. These differences are consistent with prior studies which have documented demographic factors that have influenced response to surveys or other studies administered via cancer registries include Hispanic ethnicity [73, 74], age [6, 7, 11, 21, 22, 32, 73, 75, 76], and time between diagnosis and recruitment [6, 11, 73, 77, 78]. Unlike prior studies, we did not observe overall differences according to sex [73, 74, 77], race [6, 7, 32, 73‐75], or cancer stage at diagnosis [6, 7, 73]. However, we did see females overrepresented among respondents to paper but not web, and Hispanics and nonwhites underrepresented among web respondents. While we were unable to fully assess response by educational attainment, we did find that web and paper respondents reported similar levels of education.

There are limitations worth noting for this study. First, we were unable to offer incentives for participation, which resulted in a lower response rate than similar studies that have been conducted out of the registry. We were also unable to assess whether educational attainment or other socioeconomic variables affected response across the study arms because registries do not collect this information. Additionally, because this was a pilot study, the sample size was relatively small, making it difficult to identify significant differences for small effect sizes or to draw conclusions about various demographic subgroups included in the study. We also did not evaluate the use of a paper response option delivered later in administration, as is done in most web-push designs.

Another limitation is that due to the focus of this study on early-onset diagnoses of particular cancer sites, and the oversampling of some subgroups, our sample is not representative of cancer survivors generally. Most notably, in this study of early-onset cancer survivors, only 16.8% of the eligible sample was aged 60 or older; in contrast, 61.3% of all cancers of any site diagnosed in the same time period in Utah were among individuals aged 60 or above. Additionally, with the inclusion of two female-specific cancer sites and an oversample of Hispanics, compared to the entire registry of cancer diagnoses from 2001 to 2016, our sample had a higher percentage of females (56.5% vs. 48.4%) and Hispanics (15.5% vs. 5.8%) than are included in the database. Therefore, the results may not be generalizable to other registry-based study samples. In prior research, we found that recruitment outcomes from samples obtained from this registry vary according to cancer and demographic variables [32], thus we expect response results would vary in studies of different cancer sites or age groups. Nevertheless, the differences observed across experimental arms and demographic subgroups offer informative evidence and warrant further experimentation with web surveying in other registry samples.

This study has found that collecting survey data via the internet is a feasible approach for cancer registries wishing to obtain responses from a representative group of individuals diagnosed with cancer, despite being limited to a postal mail and telephone-based contact approach. Although it may be advisable to utilize a paper follow-up response option for those who do not respond online (as is done in the standard web-push design), these results signal that registries may consider incorporating web surveys without much loss of overall response compared to the traditional paper-based approach. Future research should more fully assess the viability of a web-push design for obtaining survey data via cancer registries using larger samples that are inclusive of more cancer sites and older individuals. Additionally, further examination of how brochure enclosures influence response across survey modes is warranted.