Should samples be weighted to decrease selection bias in online surveys during the COVID-19 pandemic? Data from seven datasets

Generally used for marketing purposes, online surveys have recently become a popular data-gathering tool in scientific research [1], mainly helpful during the COVID-19 pandemic. Besides protecting data collectors from infection, cost savings, simplicity of data collection, ease of processing findings, flexibility in questionnaire design, and the ability to contact respondents across national borders are all arguments in its favor [1]. However, the use of web surveys has triggered a heated debate regarding their scientific validity [2, 3].

The main argument against web surveys is the selection bias of the sample, which is not chosen at random, the target population being a convenience sample rather than a probability sample [1]. This non-probability method of selection is generally problematic, leading to an unequal probability of selection. Bias further occurs since specific characteristics (such as age, education, gender) are under- or over-represented in the gathered sample, thus impacting the reliability of the results [1]. Even a well-designed sampling plan would frequently result in the survey being completed by too many women and not enough men or by too many young people and not enough elderly individuals. Furthermore, all these factors might be linked to different health-related variables, attitudes, and behaviors that survey researchers are interested in [4].

Selection bias occurs in studies that use online surveys as it only reaches a subgroup of the target population [5]. Only literate people, those who have access to the internet, and those sufficiently interested in the topic can complete online surveys [5]. For example, when a subgroup is targeted (thus overrepresented, such as literate people or those with access to the internet), selection bias will generally increase as the target population becomes less diverse, resulting in biased findings [5, 6]. Sometimes, a survey about COVID-19 would only attract a specific subgroup of people interested in the topic. However, during infectious disease outbreaks, a quick online survey is necessary to reach a large number of people in a short time to collect the needed information [7]. Moreover, various types of problems and errors are encountered in the data collected online (information bias), leading to concerns about the quality and reliability of the resulting scientific information [1].

To overcome biases and improve the quality of online survey findings, many authors have adopted weighting methods [1, 4], such as rectifying imbalances between the survey sample and the population by applying these methods to adjust demographic characteristics (gender, age, ethnicity, educational background, and geographic area) [4]. Because some factors of interest may not always have a strong enough link with demographic weighting variables, weighting methods can only compensate for proportionality, not always representativeness [8]. Hence, the considerable debate about weighting methods and their effect on variance during analysis, as some researchers claim that weighting has little potential for eliminating biases in web surveys [9]. As variance is used to calculate confidence intervals and hypothesis tests, weighting data would raise the variance of estimates [10], leading to a loss of accuracy [10]. Nevertheless, researchers are often willing to accept inaccuracy to obtain unbiased estimates [10].

A direct comparison of unweighted and weighted samples has rarely been performed in the literature [11, 12]. From a practical perspective, comparing the two techniques is critical because they may provide different findings of the overall impact strength, outcome consistency across studies, and other variables’ effect on the association.

Two studies comparing weighted and unweighted estimates from online samples have revealed that demographic weighting decreased bias in some situations while it substantially increased it in others [11, 12]. Recent research using aggregated data to evaluate racial/ethnic inequities in COVID-19 mortality has found that weighted population distributions underestimated the excess burden of COVID-19 among African American and Latin individuals, compared with analyses conducted with an unweighted population [13].

Consequently, this work aims to compare weighted and unweighted association measures after adjustment over potential confounding, taking into account dataset properties such as the initial gap between the population and the selected sample, the sample size, and the variable types.

Our study compared weighted and unweighted samples of online surveys and assessed the extent to which weighting methods can adjust the web sampling to the reference sample and how it would affect the results. Our findings revealed a high variation of age and gender between weighted and unweighted samples within the same population; however, high similarities were found for dependent and independent variables in terms of relative difference measures.

The regression analysis results showed a high relative difference between weighting and unweighting methods in some datasets and for some variables, while a low difference was found for others; association measures would increase or decrease after weighting. These discrepancies could be explained by the large sample size and the high relative difference in gender, related to lower relative differences in association measures between weighted and unweighted methods. However, a high relative difference of age was associated with the high relative difference of association measure. These results indicate that proportions of the sociodemographic variables are adjusted after applying the weighting methods; however, it does not necessarily affect the association between variables.

The impact of weighting was limited in some datasets, while differences were found in others. The discrepancies between weighted and unweighted databases were significantly affected by the sample size, followed by age relative difference and gender relative difference. A possible explanation could be that when analyzing the use of weights to compensate for the distributions of different variables, some factors of interest may not always have a strong enough link with the demographic variables; thus, the weighting method could not correct any biases. Consequently, the impact of weighting depends on the variables of interest and how these variables are related to the sociodemographic variables. As a result, the decision to weight samples will be based on the study objective, design, and type of outcome.

Our work showed that the initial gap between the sample and the population, in addition to the sample size and the presence of a significant association between some sociodemographic variables and the dependent variable, could all impact the association measure, but in differential ways: correcting for age gap would improve association measures, but not gender gap correction. Similarly, other researchers had previously reported that weighting techniques can compensate for proportionality but not always representativeness because some factors of interest do not always have a strong enough link with the demographic weighting variables [16]. Thus, adjusting for proportionate overrepresentation and underrepresentation of specific respondent categories does not imply that the substantive responses of online access panel respondents are equal to those of the general population [16]. Oppositely, according to Bethlehem and Stoop, one or more qualitative auxiliary variables are required for the weighting method. Nevertheless, even if the target variable and the stratification variables have a strong relationship, the change in the target variable's values appears very low [16].

Our results showed that the larger the sample size, the lower the impact on the association measure; in other words, lower samples derived association measures are more affected if not weighted. This finding corroborates the principle that large sample sizes and high response rates positively influence the quality of estimates, according to the theoretical framework of probability sampling [17]. Similarly, a large-scale study that used 17 samples from online surveys found that bigger sample size (lower margin of sampling error around the estimate) is associated with a better level of precision [12]; large-scale online surveys have the advantage that specific subgroups can be identified [16]. The fundamental assumption is that people who engage in an online survey, whether elderly single women, less educated people, ethnic minorities, or other usually underrepresented groups, are equivalent to those who do not engage in online surveys [16], even though people who belong to these groups are hard to reach or unlikely to participate in surveys [16]. However, one study has addressed the erroneous idea that larger samples imply more valid replies [18], showing that larger samples do not always yield better estimates than smaller ones from non-probability samples, while a larger sample size can lead to greater accuracy only with probability samples [18]. Similarly, according to Bryman and Bell, precision cannot be guaranteed with a large sample size [19]. Thus, additional studies are required to further depict these findings.

Our findings reinforce the variability of results found in the literature about the application of weighting methods in scientific surveys. It is unclear whether or not online surveys can be made more representative [20]. While weighted samples are expected to be more representative than unweighted ones, this study could demonstrate that this is not always the case. As a result, one cannot simply assume that using the weighted method will always result in a more accurate estimate of the population studied. Also, it cannot be concluded that the unweighted technique will always yield more conservative sample homogeneity suggestions than the sample-weighted method, as demonstrated by previous findings showing that demographic weighting reduced bias in some cases and increased it in others [11, 12]. A study compared data from a self-administered online survey with the answers collected in a face-to-face interview and found that the results were not significantly affected by weights on age, gender, or education [8]. Another study compared two datasets collected online and showed that the impact of the weighting method on the results was very limited [1]. Other findings revealed that non-probability samples significantly differed from probability samples, particularly in terms of attitudes and behaviors, even after making them demographically similar to target groups [11, 21‐23].

The results of this analysis of online surveys indicate that weighting methods should be used cautiously, as weighting did not affect the results in some datasets, while it did in others. Weighting methods might yield unpredictable results, depending on variable gaps, sample size, and the association between sociodemographic characteristics used for adjustment and dependent variables. Further research is necessary to define situations in which weighting would be beneficial.