Development and process evaluation of a web-based responsible beverage service training program

Initial project development meetings were convened for a multidisciplinary team composed of individuals from Deschutes Research, Inc. (a company focused on the dissemination of evidence-based programs), InterVision Media (a Oregon-based technology media and Web intervention development programs), and Oregon Research Institute (a non-profit independent behavioral sciences research center) to generate ideas regarding the design, development, and evaluation of a Web-based RBS training program. Team members had experience and expertise in the development and evaluation of alcohol treatment programs, health behavior change programs, Web-based behavioral interventions, and the programming of complex interactive websites. In addition, the team had been previously involved in a Phase I SBIR research program that developed and tested a portion of the full RBS Web-based program that is described in this report. The team followed an incremental and iterative formative development process[9] spanning the steps between initial design and eventual program testing (see Figure 1). This process allowed for multiple opportunities for feedback from representative RBS trained servers.

Focus group procedures were informed by the team’s experience and by published guidelines[12]. These focus groups helped to identify the factors that would encourage participants to use the Web-based RBS program to achieve their work goals. Focus groups also provided early feedback about the type of content that needed to be included and how it would be best delivered.

For optimal adoption and accurate use, Web-based tools like the RBS program should embody established usability standards[13, 14]. A relatively small number of usability testers can provide extremely valuable data that inform revisions to a program[15, 16]. The typical procedure calls for each user to meet individually with a research staff member who acts as a facilitator. During the session the usability tester is asked to interact with functional portions of the program using a hands-on approach as he/she accomplishes certain assigned tasks while receiving minimal direction from the facilitator. During this time each tester is asked to describe their thoughts using a think-aloud technique derived from cognitive science[17] that has proven effective in the study of human-computer interactions[18, 19]. As noted by Hughes[20], the think-aloud technique provides “…direct, real-time observations [by] the user rather than self-reports such as surveys” (p.493). Think-aloud methods assess cognition concurrently with its occurrence, and thus may be better at describing the thoughts and attitudes of users rather than asking them for their retrospective self-reports[21]. At the conclusion of the session, testers were asked for their overall feedback including whether they would recommend the program to other servers, how interesting the program was, how much of the program information was new to them, whether important topics were omitted, how easy the program was to use, and how the program compared to in-person RBS training. Each usability test session was recorded and transcribed for review and analysis of themes and relevant statements.

Two rounds of usability tests were conducted. The first round involved seven alcohol servers and managers (six female and 1 male) of whom six described themselves as Caucasian, one as African American, and one as Hispanic/Latino (multiple categories were allowed). The second usability test consisted of 3 female alcohol servers and managers who described themselves as Caucasian and non-Hispanic/Latino. Usability testers were observed as they completed a series of tasks on the website including user registration, reviewing program information, completing section quizzes, and completing the online assessments.

All data associated with the study were maintained in a confidential manner. All transmission of data over the Internet was protected by our use of Secure Sockets Layer (SSL) – an industry standard encryption protocol. Study data were stored on secure servers located within a locked computer room that could only be accessed by approved study staff. In addition, each study participant was assigned a unique username and password which provided secure access to the program website. All study results were described only as aggregate data and never in a manner that could be used to identify any participant’s identity. Participants agreed to an online informed consent before they were able to interact with the Web-based intervention. The research protocol was approved by the Institutional Review Board of Oregon Research Institute.

The set of 40 attitude items was initially grouped into 7 subscales according to theme and similarity of rating scales. All items were scaled in a manner such that a higher rating reflected a more desirable or positive attitude. Subscales were further refined by examining their internal reliability (Cronbach's alpha; a) which resulted in dropping 2 scales thus yielding 5 subscales: Usefulness at Work (8 items; a = .65), Self-efficacy (8 items, a = .85), Risks (2 items, a = .66), Being Prepared (2 items, a = .87), and Establishment Policies (10 items, a = .92). A composite score for each subscale was assigned to each participant defined as the mean of ratings of subscale items. Baseline to Post-Test comparisons of attitudes used t-tests for independent samples for each subscale. Analysis of changes in attitudes from Baseline to Post-test was examined using paired t-tests.

Program exposure (use of the program) was the number of visits and overall time spent using the program[24].

Scores of the five attitude subscales were examined using paired t-tests for change from Baseline and Post-test. In each instance, statistically significant improvements in mean scale ratings were observed (see Table 3).