Comparing web and mail responses in a mixed mode survey in college alcohol use research

Abstract

Objective

This exploratory study examined potential mode effects (web versus U.S. mail) in a mixed mode design survey of alcohol use at eight U.S. colleges.

Methods

Randomly selected students from eight U.S. colleges were invited to participate in a self-administered survey on their alcohol use in the spring of 2002. Data were collected initially by web survey (n = 2619) and non-responders to this mode were mailed a hardcopy survey (n = 628).

Results

College students who were male, living on-campus and under 21 years of age were significantly more likely to complete the initial web survey. Multivariate analyses revealed few substantive differences between survey modality and alcohol use measures.

Conclusions

The findings from this study provide preliminary evidence that web and mail surveys produce comparable estimates of alcohol use in a non-randomized mixed mode design. The results suggest that mixed mode survey designs could be effective at reaching certain college sub-populations and improving overall response rate while maintaining valid measurement of alcohol use. Web surveys are gaining popularity in survey research and more work is needed to examine whether these results can extend to web surveys generally or are specific to mixed mode designs.

Introduction

Web surveys are a viable mode of data collection on many college campuses because of the near universal use of the Internet among college students (Couper, 2000, Jones, 2003, Rainie, 2001). With the growing popularity of web surveys, it has become more important to compare this innovative mode of data collection to other, more traditional, survey methods. Therefore, the impact of collecting data among undergraduate students using web surveys versus traditional approaches such as mailed paper surveys has become a topic of college-based research (e.g. Carini et al., 2003, Pealer et al., 2001, Wygant and Lindorf, 1999).

Past research suggests that data collection modality can lead to substantially different answers to questions regarding alcohol and other drug use (e.g. Link and Mokdad, 2005, Turner et al., 1998, Wright et al., 1998). However, at least three randomized college-based studies suggest minimal differences in the reporting of alcohol use between web surveys and more traditional paper-based survey approaches (e.g., Bason, 2000, McCabe et al., 2002, Miller et al., 2002). For instance, Miller et al. (2002) randomly assigned 255 college students ages 18–29 to complete a survey in one of three conditions, including one paper-based and two web-based conditions. The survey contained several well-known alcohol-related measures such as the Alcohol Use Disorders Identification Test (AUDIT: Saunders, Aasland, Babor, de la Fuente, & Grant, 1993), Rutgers Alcohol Problem Index (RAPI: White & Labouvie, 1989), University of Rhode Island Change Assessment (URICA: Prochaska & DiClemente, 1986), Alcohol Dependence Scale (ADS: Skinner & Allen, 1982), and a quantity-frequency-peak index used to assess drinking rates. Miller et al. (2002) found there were no significant mean differences among the assessment conditions on any measures of alcohol use. The authors concluded that the web-based survey mode of data collection represented a suitable alternative to a paper-based approach because data integrity was not compromised. In another study, Bason (2000) randomly assigned 3000 students to complete the CORE survey (Presley, Meilman, & Cashin, 1996) between four modes of data collection (telephone, mail, web-based and interactive voice recognition) and examined substantive differences in prevalence estimates of alcohol and other drug use. Although the web mode produced the lowest response rate of the four survey modes, there were minimal substantive differences in prevalence rates of substance use between web respondents and other modes. Finally, McCabe et al. (2002) randomly assigned 7000 students to complete a survey via two distinct survey modes (web and mail). The web mode produced a sample that was more representative of the overall student population with respect to gender and there were no significant differences in alcohol use measures. In addition, there were no differences in the distribution of race, class year, academic credit hours, or age between the final samples obtained from the two survey modes.

Despite evidence that web surveys can produce comparable sample demographics and estimates of substance use as mailed paper surveys when college students are randomly assigned to survey modes (e.g. McCabe et al., 2002, Miller et al., 2002), there remains a need for comparative research that examines the impact of using a non-randomized mixed mode design on sample demographics and substance use estimates. Non-randomized mixed mode studies, including web and mailed paper surveys have been conducted at multiple universities, and very few substantive differences between survey mode have been found (e.g. Carini et al., 2003). However, such studies have not examined highly sensitive behaviors such as substance use.

There are several different types of web surveys and the current investigation focuses on a probability-based survey approach within a mixed mode design (Couper, 2000). The main objective of this study was to examine whether demographic characteristics and alcohol use data collected initially using a web survey as the first mode of data collection differed from a subsequent mailed paper survey as the second mode of data collection.