Describe survey design
Describe target population, sample frame. Is the sample a convenience sample? (In “open” surveys this is most likely.)
Mention whether the study has been approved by an IRB
Describe the informed consent process. Where were the participants told the length of time of the survey, which data were stored and where and for how long, who the investigator was, and the purpose of the study?
If any personal information was collected or stored, describe what mechanisms were used to protect unauthorized access
Development and testing
State how the survey was developed, including whether the usability and technical functionality of the electronic questionnaire had been tested before fielding the questionnaire
Open survey versus closed survey
An “open survey” is a survey open for each visitor of a site, while a closed survey is only open to a sample which the investigator knows (password-protected survey)
Indicate whether or not the initial contact with the potential participants was made on the Internet. (Investigators may also send out questionnaires by mail and allow for Web-based data entry.)
Advertising the survey
How/where was the survey announced or advertised? Some examples are offline media (newspapers), or online (mailing lists—If yes, which ones?) or banner ads (Where were these banner ads posted and what did they look like?). It is important to know the wording of the announcement as it will heavily influence who chooses to participate. Ideally the survey announcement should be published as an appendix
State the type of e‑survey (e.g. one posted on a Web site, or one sent out through e‑mail). If it is an e‑mail survey, were the responses entered manually into a database, or was there an automatic method for capturing responses?
Describe the Web site (for mailing list/newsgroup) in which the survey was posted. What is the Web site about, who is visiting it, what are visitors normally looking for? Discuss to what degree the content of the Web site could pre-select the sample or influence the results. For example, a survey about vaccination on an anti-immunization Web site will have different results from a Web survey conducted on a government Web site
Was it a mandatory survey to be filled in by every visitor who wanted to enter the Web site, or was it a voluntary survey?
Were any incentives offered (e.g. monetary, prizes, or non-monetary incentives such as an offer to provide the survey results)?
In what timeframe were the data collected?
Randomization of items or questionnaires
To prevent biases items can be randomized or alternated
Use adaptive questioning (certain items, or only conditionally displayed based on responses to other items) to reduce number and complexity of the questions
Number of items
What was the number of questionnaire items per page? The number of items is an important factor for the completion rate
Number of screens (pages)
Over how many pages was the questionnaire distributed? The number of items is an important factor for the completion rate
State whether respondents were able to review and change their answers (e.g. through a Back button or a Review step which displays a summary of the responses and asks the respondents if they are correct)
Unique site visitor
If you provide view rates or participation rates, you need to define how you determined a unique visitor. There are different techniques available, based on IP addresses or cookies or both
View rate (ratio of unique survey visitors/unique site visitors)
Requires counting unique visitors to the first page of the survey, divided by the number of unique site visitors (not page views!). It is not unusual to have view rates of less than 0.1 % if the survey is voluntary
Participation rate (ratio of unique visitors who agreed to participate/unique first survey page visitors)
Count the unique number of people who filled in the first survey page (or agreed to participate, for example by checking a checkbox), divided by visitors who visit the first page of the survey (or the informed consents page, if present). This can also be called “recruitment” rate
Completion rate (ratio of users who finished the survey/users who agreed to participate)
The number of people submitting the last questionnaire page, divided by the number of people who agreed to participate (or submitted the first survey page). This is only relevant if there is a separate “informed consent” page or if the survey goes over several pages. This is a measure for attrition. Note that “completion” can involve leaving questionnaire items blank. This is not a measure for how completely questionnaires were filled in. (If you need a measure for this, use the word “completeness rate”.)
Indicate whether cookies were used to assign a unique user identifier to each client computer. If so, mention the page on which the cookie was set and read, and how long the cookie was valid. Were duplicate entries avoided by preventing users access to the survey twice; or were duplicate database entries having the same user ID eliminated before analysis? In the latter case, which entries were kept for analysis (e.g. the first entry or the most recent)?
Indicate whether the IP address of the client computer was used to identify potential duplicate entries from the same user. If so, mention the period of time for which no two entries from the same IP address were allowed (e.g. 24 h). Were duplicate entries avoided by preventing users with the same IP address access to the survey twice or were duplicate database entries having the same IP address within a given period of time eliminated before analysis? If the latter, which entries were kept for analysis (e.g. the first entry or the most recent)?
Log file analysis
Indicate whether other techniques to analyze the log file for identification of multiple entries were used. If so, please describe
In “closed” (non-open) surveys, users need to login first and it is easier to prevent duplicate entries from the same user. Describe how this was done. For example, was the survey never displayed a second time once the user had filled it in, or was the username stored together with the survey results and later eliminated? If the latter, which entries were kept for analysis (e.g. the first entry or the most recent)?
Handling of incomplete questionnaires
Were only completed questionnaires analyzed? Were questionnaires which terminated early (where, for example, users did not go through all questionnaire pages) also analyzed?
Questionnaires submitted with an atypical time stamp
Some investigators may measure the time people needed to fill in a questionnaire and exclude questionnaires that were submitted too soon. Specify the timeframe that was used as a cut-off point, and describe how this point was determined
Indicate whether any methods such as weighting of items or propensity scores have been used to adjust for the non-representative sample; if so, please describe the methods