How users make judgements about the quality of online health information: a cross-sectional survey study

Different conceptualizations of information quality have been proposed in previous literature. The process-oriented model views information quality as a product of measurement where accuracy is guaranteed during the measurement process [18], while the system-oriented model defines information quality as various steps of information collection, retrieval, and display [19]. In comparison, the user-oriented model conceptualizes information quality as users’ subjective perception of whether information satisfies their information needs [20], and information quality judgement is the process by which users apply quality criteria to determine the overall quality of certain information [20]. This study adopts the user-oriented approach to investigate how users make subjective judgements on the quality of health information.

Several user-oriented models were found in previous literature. Bovee, Srivastava, and Mak (2003) proposed an intuitive model of information quality, proposing a three-layer model with integrity, accessibility, interpretability, and relevance in the first layer. Integrity included four criteria: accuracy, completeness, consistency, and existence. Datedness, usefulness, and other criteria were included as the second layer of the relevance criterion. Age and volatility were included within datedness as the third layer of the criteria [21]. Kahn, Strong, and Wang (2002) developed a two-by-two conceptual model that defined two dimensions of information quality: one dimension for product quality and service quality and the other for conforming to specifications and meeting or exceeding consumer expectations. They assigned multiple criteria to these quadrants, such as freedom from errors, conciseness, completeness, and consistency within the quadrant "product quality * conforms to specifications" and timeless and security within the quadrant "service quality * conforms to specifications" [17]. Stvilia et al. (2007) developed a three-category framework, including intrinsic, relational, and reputational categories. Within each category, several criteria were included (e.g., currency and complexity within the intrinsic category and accuracy and naturalness in the relational category) [22].

Sellitto and Burgess (2005) developed a weighted framework consisting of criteria with different weights [5]. Criteria "reputable" was assigned the highest weight, followed by "no advertising" and "creation”. Stvilia, Mon, and Yi (2009) extended Stivilia's model (2007) and suggested a model of consumer health information quality with five constructs: accuracy, completeness, authority, usefulness, and accessibility, each of which contained several subcriteria (e.g., credibility and reliability in the construct of accuracy) [23]. Subsequently, Kim and Oh (2009) developed a fine-grained list of criteria for information quality judgement, including clarity, rationality, novelty, and quickness [24], which were not included in the list of Stvilia et al. Sun et al. (2015) further extended the list by identifying additional criteria: solution feasibility, certification, politeness and humour [25]. Other lists of judgement criteria have also been found in the works of other scholars (e.g., see Choi & Shah, 2016 [26]; Emamjome et al., 2013 [27]). They share most of their criteria with the aforementioned lists.

The issue with these studies is that terms were used inconsistently within different lists of criteria. For instance, the consistency criterion from Stvilia's study was replaced by the term readability in Zhu et al.'s study with the same definition [28], which was further changed to representation interpretability in Ge et al.'s study [29]. Besides, currency was interchanged with timeliness and quickness in different studies [30]. A summary of the aforementioned criteria is presented in Table 1. Recently, Sun et al. (2019) conducted a systematic review of quality judgement of online health information and summarize all the criteria in previous literature into a list of twenty-five criteria [11]. This comprehensive list of criteria was used in this study.

Several factors have been found to influence the judgement of health information quality. Benotsch, Kalichman, and Weinhardt (2004) conducted a survey of HIV patients to determine how patients evaluated the quality of online health information for certain health websites [18]. Educational and income levels were found to be significantly related to the overall assessment of online health information quality in addition to health literacy (including knowledge) and other psychological factors. Another national survey of Americans’ health information seeking indicated that educational level affected users' ability to navigate within the online environment to seek health information [31], which affected their judgement of online health information.

Another set of studies focused on the factors that are related to particular criteria for the quality of health information, i.e., trustworthiness and credibility. Age, sex, and education were found to be significantly related to these criteria [32]. Three reviews summarized a list of possible influencing factors found in empirical studies: gender, education, health status, income, age, health literacy, race, and health beliefs. However, health status has been found to be significantly related to trustworthiness and credibility in certain studies but not in others [33‐37]. In addition, controversial results on health beliefs have been found (e.g., Atkinson et al. found that people with poor health status trusted online health information more, while Cotten et al. observed the opposite [33‐35, 38, 39]). Furthermore, health beliefs have been found to affect the intention to use health information rather than the direct judgement of quality [40]. Therefore, these two factors were left for further investigation, while gender, age, education level, and health literacy were retained in this study. In addition, the health literacy scale was replaced with the eHealth literary scale, which is more appropriate for the online environment. Race was not used in this study since the participants were all Chinese.

Although quite a few studies have focused on the overall judgement of health information quality or examined some of its dimensions/criteria while ignoring other criteria, no previous studies have investigated both how likely users are to use certain quality judgement criteria and how important they consider these criteria. In other words, it remains unclear whether the likelihood of using certain criteria is consistent with the perceived importance of these criteria among users with different demographic characteristics. This study investigated how these demographic variables and eHealth literacy affect these patterns and consistency.

The demographics of the participants were collected and categorized as follows. In terms of age, the participants were categorized into five groups from 18 years old to 60 years and above (18–29; 30–39; 40–49; 50–59; 60 and above). Educational levels were also categorized into five groups from junior high school or below to Ph.D. degree (junior high school or below; high school; undergraduate; master’s; Ph.D.). When gender was considered, two groups (female; male) were used to categorize the participants (no participant indicated gender as unclear or transgender).

In addition to the demographic variables, two sets of questions were used in this study. The first set was items to assess the intention of using a particular criterion in the judgement of online health information and the perceived importance of that criterion, adopted from Sun et al.'s review article [11]. For each criterion, the participant was asked to indicate whether he or she used it in evaluating the quality of online health information. If yes, the participants proceeded to indicate how important the criterion was using a five-point Likert scale. The second set was the eHealth Literacy scale (eHEALS) to assess consumers’ ability to engage in eHealth [43]. It was used to replace the traditional health literacy scales since the context of this study was online health information seeking and evaluation. This scale has been validated by various empirical studies and shows good psychometric properties. The distribution of the eHealth literacy scores among the participants was skewed (S = -0.446), with a median total score of 30 for the eHEALS scale (range: 8–40). Since there was no consensus on the cut-off score of the eHEALS scale to categorize people into high and low eHealth literacy, we used the median score to divide the participants into these two user groups.

All the measures used in the study were forwards and backwards translated following the COSMIN guidelines. Each type of translation involved two bilingual translators whose mother tongue was the original language: one was the domain expert, and the other was naive about the domain. The backwards translation was compared with the original questions, and discrepancies were resolved by discussion among the authors and translators.

Two kinds of statistical analyses were adopted in this study. To determine whether a particular criterion was used by a patient, chi-square analysis was conducted to compare the group differences in the frequency of criterion use since it was a binary variable. For the perceived importance of each criterion, nonparametric analysis (Kruskal–Wallis test and 1-way ANOVA for post hoc test) was used to compare the group differences (the normal distribution assumption was not guaranteed). It should be noted that in the analysis of importance, participants who indicated that they did not use a particular criterion were excluded since the measurement of the five-point Likert scale assumed that the participants used the criterion regardless of how important they considered it (even if it was considered unimportant, the value was 1 in the scale). Participants who did not use the criterion would introduce bias into the analysis interpretation of the results. The scores of the eHealth literacy scale items were summed to indicate participants’ level of literacy when they conducted online health information seeking and evaluation [44]. The data analysis was conducted using SPSS 26 software.

A total of 1500 patients were invited to participate in this study, of whom 1063 accepted the invitation and finished the survey (acceptance rate: 70.9%). Among them, nearly two-thirds were under forty years old, while less than 10% were over sixty. In terms of educational level, more than half of the participants held a college degree, and more than one-quarter of them had received higher degrees (master’s and Ph.D.). In comparison, less than 20% held a primary or high school degree. Details are shown in Table 2.

Regarding the eHealth literacy level of the participants, the distribution was skewed (S = -0.446) with a median total score of 30 for the eHEALS scale (range: 8–40). This is similar to recent survey results of eHealth literacy from other countries using the same scale [45, 46]. Since there was no consensus on the cut-off score of the eHEALS scale to categorize people into high and low eHealth literacy, we used the median score to divide the participants into these two user groups.

The statistical results of the frequency of use for the judgement criteria for health information quality among different user groups and the perceived importance are presented according to the demographic variables and eHealth literacy levels mentioned above. The details of the post hoc analysis can be found in the appendix.

Patients with high eHealth literacy were more inclined to use the criteria of transparency and usefulness than those with low eHealth literacy. However, there were no differences in the perceived importance of these two criteria between these two groups of patients. In contrast, for the criteria of familiarity, accessibility, aesthetics, comprehensiveness, and practicality, no significant differences were found in their frequencies of use between the two groups of participants, although they perceived them to have significantly different importance. The nonparametric statistical analysis (K-W) showed that patients with high eHealth literacy considered these criteria to be more important than did those with low eHealth literacy. For the criteria of identification, interactivity and balance, patients with high eHealth literacy considered these criteria to be more important and therefore were more inclined to use them than those with low eHealth literacy. No differences in the frequency of use and perceived importance were found for the remaining criteria between patients with high and low eHealth literacy. Details are shown in Table 3.

For the criteria of familiarity, identification, aesthetics, and anonymity, no significant differences were found in the likelihood of using these criteria in judging the quality of online health information among different age groups. However, these criteria were considered to have different levels of importance among these groups. The nonparametric statistical analysis (K-W) showed that for the criteria of familiarity, identification, and aesthetics, patients from 18 to 29 years old considered them less important than did those from 30 to 39, 40 to 49, and 50–59 years old. In comparison, the difference between any other two age groups was not significant. In terms of the criterion of anonymity, only patients from 18 to 29 years old considered it less important than those from 40 to 49 years old did. Surprisingly, patients over 60 years old were least likely to use this criterion. Patients from 30 to 39 years old were found to have the second-lowest likelihood of adopting this criterion. In addition, patients from 18 to 29 years old were less likely than those aged 40 to 59 years old to apply this criterion to the quality judgement of online health information. For the other criteria, no significant differences were found in either the likelihood of use or the perceived importance among patients in different age groups. Details are shown in Table 4.

A similar situation was found among patients with different educational levels. For the criteria of expertise, familiarity, aesthetics, and security, patients with different educational levels were found to have no significant differences in their inclination to use these criteria. However, they assigned different importance to these criteria. The expertise criterion was considered more important by patients with a master's degree than by patients with either a bachelor's or a high school degree. Similarly, the security criterion was considered more important by patients with a master's degree than by those with either a bachelor's or a high school degree. In contrast, familiarity was considered more important by patients with either a bachelor's or high school degree than by those with a master's degree. Furthermore, patients with high school degrees considered familiarity even more important than those with a bachelor's degree. For the criteria of objectivity, identification, and accuracy, no significant differences in their perceived importance were found among patients with different educational levels. However, their likelihood of being used differed significantly among these patient groups. For the objectivity criterion, patients with primary school degrees were more inclined to use this criterion than those with higher degrees when making a quality judgement about online health information. Surprisingly, patients with a Ph.D. degree were more inclined to ignore this criterion than those with either a bachelor's or a master's degree. It should be noted that patients with a primary school degree were less likely to use this criterion for quality judgement than patients with any other educational level. In sharp contrast, they considered this criterion to be more important than those with a bachelor's degree. Details are shown in Table 5.

For the criteria of trustworthiness, relevance, believability, readability, practicality, completeness, anonymity, and security, female patients considered them to be more important than male patients did. However, female patients were not more inclined to use these criteria than male patients in judging the quality of online health information. No significant differences in the likelihood of using the remaining criteria and their perceived importance were found between these two groups of patients. Details are shown in Table 6.

The inconsistency between the likelihood of use and perceived importance indicates that patients may use different sets of quality judgement criteria in different ways. The criteria of popularity, currency, and navigability are those in which no significant difference was found in either the likelihood of use or perceived importance among the patients grouped by all the demographic variables and eHealth literacy. This suggests that these criteria may be independent of patients’ demographic and eHealth literacy backgrounds. Patients share similar attitudes regarding their importance and how to use them. The underlying reason could be that applying these criteria to the quality judgement of online health information does not involve patients' knowledge, experience, and other cognitive features, which are influenced by their age, education, health literacy, and gender. Another possible reason could be that there are universal prototypes for popularity, currency, and navigability among all patients and even the entire population. Therefore, demographic variables may not affect whether these criteria are used and how important they are perceived to be. Unfortunately, no previous studies were found in this area, and further investigation is required to test these possible explanations.

In contrast, the perceived importance of criteria does not always guarantee the inclination to use these criteria. For example, patients with a master's degree assigned more weight to the expertise criterion than those with either a high school or bachelor's degree. However, in practice, they did not show a preference for this particular criterion over the other two groups of patients. The reason could be that the more education patients receive, the more attention they give to features that are closely related to the information content. Hence, the perception of expertise and security conveyed by the information is more attractive to well-educated patients. In comparison, for the criterion of aesthetics, the opposite was observed. The same reasoning could be applied to the perceived importance of the criterion of familiarity among different age groups. As patients get older, they may have fewer cognitive resources and may rely on more superficial features of online health information, such as the heuristic of familiarity, to assess its overall quality.

It should be noted that for the criterion of comprehensiveness, although patients with a primary school degree considered it more important than those with a bachelor’s degree, they were actually more likely to ignore it when making a quality assessment of online health information. It is common sense that if one group of people considers particular criteria more important than another group, they tend to use these criteria more frequently. This counterintuitive finding may be due to the irrational reasoning and decision-making of these patients during a crisis (i.e., the COVID-19 pandemic and infodemic). During the infodemic, patients are likely to be influenced by the risk situation and emotion [47, 48], which further biases their rational thinking process and affects their quality judgement. Therefore, they may not naturally rely on the criteria they consider important [49], which contributes to the inconsistency.

The findings of this study provide guidelines on how online health information should be presented to facilitate users' quality judgement, which further helps them use information appropriately.

First, online health information services require fine-grained customization to facilitate the evaluation of users with different demographic characteristics. As patients can be grouped into fine-grained user groups by combining these demographic variables, this study provides practical guidelines on prioritizing detailed user groups during the customization of online health information services. There are different priority sequences for these user groups to facilitate their quality judgement.

Second, the health literacy of patients should be given more attention when attempting to motivate patients to find higher-quality health information. As one of the main drivers of health behavioural change [50], this may influence patients’ willingness to follow certain online suggestions and perform particular types of protective behaviours, even if these suggestions are of low quality. Therefore, it is necessary for scholars to develop more efficient interventions to increase patients’ health literacy.

Several limitations deserve attention when interpreting the results of this study. First, only one of the leading hospitals of the coastal province with a population of forty million was chosen as the research context for this study. Hence, patients from other hospitals in this coastal province were not included in this study, nor were patients from different provinces of the country. However, this hospital was listed in the top 100 hospitals of China and was the top hospital with more than two million visits annually. Second, the number of patients who did not use a particular criterion in their judgement was small, which may bias the statistical analysis and validity of the results. Third, income level and other demographic variables were not explored in this study. The participants from this study considered their income sensitive and private information and refused to provide this information even though they were ensured that the information would be confidential. Fourth, this study did not take other confounding variables into consideration, which might influence the interpretation of the results. This could be further explored in future studies. Fifth, only patients from China were included in this study. It is possible that samples from other countries may exhibit different patterns, which could be further investigated in the future.

Future research could be conducted in the following two directions. One direction would be to further explore the reasons for the inconsistency between the likelihood of using specific quality judgement criteria for online health information and their perceived importance. Although several possible explanations were proposed in this study, the underlying mechanisms remain unclear. The other direction is to further investigate the influence of other variables (e.g., different types of health information needs and people’s resilience) [8, 51‐53] on the use patterns of quality judgement criteria to provide a more comprehensive understanding of this topic.