Depressive symptoms and web user experience

Perception and evaluation of websites

The perception of a website is primarily constituted by the users’ appraisal of three core website characteristics: content, usability, and aesthetics (e.g., Cober et al., 2003; Tarasewich, Daniel & Griffin, 2001; Thielsch, Blotenberg & Jaron, 2014). For content and usability, standardized definitions are available: In ISO standard 9241–151 (ISO, 2006), the International Organization for Standardization defines content as ‘a set of content objects’ on a web user interface. A content object is an ‘interactive or non-interactive object containing information represented by text, image, video, sound or other types of media’ (ISO, 2006, p. 3). The definition of website usability is often based on ISO 9241-11 as the ‘extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use’ (ISO, 1998, p. 2). However, a differentiation between subjective and objective usability is vital (see Hornbæk, 2006): while a website may be experienced as unusable from a subjective user’s point of view (e.g., caused by misunderstandings or missing functions), it still could perform well on an objective usability measure (e.g., based on fast loading speed or a good search function). A common definition is presently not available for the third main characteristic of a typical website: the design in terms of aesthetics. In research, aesthetics is often described as an immediate pleasurable subjective experience (Leder et al., 2004; Moshagen & Thielsch, 2010; Reber, Schwarz & Winkielman, 2004). Aesthetic perceptions and responses occur immediately at first sight; thus, the visual aesthetics of a website is processed very quickly, often within a split second (e.g., Bölte et al., 2017; Lindgaard et al., 2006; Thielsch & Hirschfeld, 2012).

Those three main characteristics interact in the perception of a website and the use process. Several studies investigate pre- and post-use differences in user evaluations, an excellent overview can be found in Lee & Koubek (2012). They proposed a model in which usability and aesthetics are strongly connected in pre-use evaluations, and relatively weakly connected after use. In the present study, we focus on post-use evaluations of content, usability and aesthetics. In addition, we investigate an outcome variable with special importance for practice: the overall appraisal of a website in terms of the intention to visit a given website again. The three main website characteristics contribute to this overall appraisal of a website (e.g., Kang & Kim, 2006; Lee & Koubek, 2012; Moshagen & Thielsch, 2010). Previously, using structural modeling of the data in two different studies, Thielsch and colleagues (2014) found that content has the largest influence on the user’s intention to revisit a website, while only a small effect was found for aesthetics, and subjective usability did not exhibit a significant influence. However, two aspects were not investigated in these studies: (a) objective usability and (b) user characteristics.

User characteristics are part of widespread user experience models, such as the CUE model (“Components of User Experience”, Thüring & Mahlke, 2007), as antecedents of an interaction experience with a website or a technical system. Thus, according to the CUE model, user characteristics are indirectly responsible for user experience outcomes such as an overall impression, and the revisitation of a website. However, there are only few UX studies systematically investigating user characteristics: for example, the effects of age (e.g., Sonderegger, Schmutz & Sauer, 2016; Thielsch, 2008), gender (e.g., Bardzell & Churchill, 2011; Tuch, Bargas-Avila & Opwis, 2010) or personality (e.g., Bosnjak, Galesic & Tuten, 2007; Burnett & Ditsikas, 2006; Thielsch, 2008). Only a few studies have been published with a focus on the web user experience of individuals with mental disorders. Due to the high prevalence of depression in the population, we aim to specifically investigate web users with different levels of depression severity.

Depressive disorders

Depressive disorders typically involve mood impairment and rank among the most frequent mental disorders throughout the world. Comparing samples from the USA, Canada, Germany, Netherlands, Mexico, Brazil, and Turkey, Andrade and colleagues (2000) found 30-day prevalence rates from 2% up to 5%, 12-month prevalence rates between 4% and 11%, as well as lifetime prevalence rates between 7% and 19%. Despite an increasing use of antidepressants over the last two decades, evidence that prevalence rates decline over time could not yet be found (Patten et al., 2016). Thus, a certain number of participants can be expected to be affected by symptoms of depression even in a common website test using a convenience sample or a usual panel sample.

Major Depressive Disorder (MDD) is part of the depressive disorders and describes a fully developed mood disorder. In addition to depressed or irritable mood, MDD is also characterized by decreased interest or pleasure in most activities, significant changes in weight or appetite, sleep, and activity, as well as psychomotor agitation or retardation, fatigue or loss of energy, feelings of guilt or worthlessness, impaired concentration, and suicidal ideation (American Psychiatric Association, 2013). In addition to these various symptoms, Blanco et al. (2014) present evidence for high comorbidity rates: the prevalence of at least one comorbid psychiatric disorder was 76% among patients with MDD. In addition to MDD, 54% were diagnosed with a substance use disorder, 31% with an anxiety disorder, 16% revealed another mood disorder, 28% a personality and 1% a psychotic disorder.

To illustrate the suffering of the people affected, Murray et al. (2012) report an increase in disability-adjusted life years (DALYs as the sum of years of life lost (YLLs) and years lived with disability (YLDs); see Murray & Lopez, 1996) which made major depressive disorder climb to place 11 in causing the most DALYs of 291 diseases in 2010 (compared to place 15 in 1990). More specifically, MDD causes impairment in self-care, mobility, cognition, social functioning and role functioning (Kessler et al., 2003). In addition to the suffering, direct and indirect costs to society are caused as well: mood disorders impair work performance. Kessler et al. (2006) estimate 27.2 annually lost workdays per ill worker with an associated annual capital loss of $4426.

Summing up, investigating depressive symptoms and their effects in everyday life is not only justified by the frequency of its occurrence, but also by the far-reaching consequences they entail. Affecting mood, cognition and behaviour, depressive symptoms are suspected to globally alter peoples’ perceptions—consequently also with regard to web user experience. In the evaluation of website-features, depressive mood, an overall loss of interest, difficulties to concentrate on the specific content (see diagnostic criteria, American Psychiatric Association, 2013 and Harvey, 2007) as well as biased evaluation patterns sensu Beck (i.e., Beck et al., 1979) might serve as specific mediatory processes. Altered web search-performance might also be driven by impaired cognitive performance and by psychomotor anomalies that are likewise listed among the diagnostic criteria (see American Psychiatric Association, 2013).

E-Mental-Health, depression and user experience

In Germany, 67% (12-month period) respectively 44% (lifetime) of individuals who suffer from a depressive disorder remain untreated—and this proportion may be relatively low by international standards (Mack et al., 2014). Furthermore, a large proportion of affected people that do receive treatment, do so with a delay which is possibly aggravating their problems (Wang et al., 2007; Wittchen et al., 2011).

The use of Internet-based treatment (“E-health” or more specifically “E-mental-health”), such as online-platforms offering information and/or aiming at reducing depressive symptoms, seems to be an important approach to improve the supply situation of people with depression. It is aimed at overcoming limitations of traditional treatment services by increasing treatment availability and reducing barriers of access (Ebert et al., 2015). Whether online-based interventions also entail burdens due to features of Internet-use, such as a lack of personal contact, is a relatively new research topic. Knowles and colleagues (2014) performed a review of eight qualitative studies on computerized depression (and anxiety) treatments: they revealed that advantages of such therapies, such as more privacy and higher experienced control, could also be perceived as limitations compared to classical face-to-face interventions. Additional studies confirmed this conclusion, demonstrating that patients with depressive disorders had mixed experiences with computerized treatments (Gega, Smith & Reynolds, 2013; Knowles et al., 2015). Yet Internet-based treatments of depression were comparable in terms of drop-out (Christensen, Griffiths & Farrer, 2009), and proved to be effective and lead to general satisfaction of participants (Davies, Morriss & Glazebrook, 2014; Richards & Richardson, 2012). Nevertheless, the research on user experiences in depressed web-users as a whole is still in its infancy.

Few studies have investigated the usability of specific interventions (e.g., Kasckow et al., 2014; Tiburcio et al., 2016) or user experience of specific online platforms and websites designed for people suffering from depressive disorders (Morris, Schueller & Picard, 2015; Prusti et al., 2012). The lack of literature emphasizes the need for a differentiated investigation of the interaction between depression and user experience, respectively between web users with depressive symptoms and information websites.

Research question and hypotheses

In this paper, we investigate the impact of depression on subjective as well as objective measures of web user experience. Subjective measures refer to an evaluation of three main website characteristics (i.e., content, subjective usability, and aesthetics), while objective measures capture performance rates in online search and memory tasks (i.e., objective usability). In both cases, we expect impaired results due to depressive symptoms: worse evaluations of all three subjective main website characteristics as well as poorer outcome in objective performance-tasks. The hypothesized model (Fig. 1) was established by integrating depression severity as the user characteristic of interest in the model of web user evaluation proposed by Thielsch and colleagues (2014). Depression is seen as a disease located on a continuum from healthy to sick, thus treated as continuously variable. As argued above, depression is expected to decrease subjective and possibly even objective measures of content, usability and aesthetics, which again potentially increase the intention to revisit. Based on previous research (Thielsch, Blotenberg & Jaron, 2014) the overall appraisal of a website in terms of the intention to revisit should be highly influenced by content perceptions, while the impact of perceived usability or aesthetics is expected to be weaker. Yet as prior research in this context has not investigated objective usability, we aim to further explore this matter.

The specific study hypotheses are:

1. (depression → content): Depression negatively influences the subjective appraisal of website content.

2. (depression → subjective usability): Depression negatively influences the subjective appraisal of website usability.

3. (depression → objective usability): Depression negatively influences objective measures of website usability.

4. (depression → aesthetics): Depression negatively influences the subjective appraisal of website aesthetics.

5. (content → intention to revisit): Subjective appraisal of website content positively influences the intention to revisit.

6. (subjective usability → intention to revisit): Subjective appraisal of website usability marginally positively influences the intention to revisit.

7. (objective usability → intention to revisit): Objective measures of website usability marginally positively influence the intention to revisit.

8. (aesthetics → intention to revisit): Subjective appraisal of website aesthetics marginally positively influences the intention to revisit.

A two-study approach was used to test the robustness of the findings, as Study 2 aims at replicating and extending the findings deriving from Study 1.

Method

Results

We first tested whether the variables of interest were significantly related to demographic factors: gender did not reveal significant differences regarding depression (t = .35, p = .73), website features (content: t =  − .76, p = .45; usability: t =  − .31, p = .76; aesthetics: t =  − 1.16, p = .25) or the intention to revisit (t =  − 1.88, p = .60). Likewise, the level of education did not portray significant gaps with regard to depression (despite a significant main effect, F(4, 324) = 2.56, p = .04, post hoc comparisons using Bonferroni correction did not reveal significant differences between the five educational groups, ranging from no school-leaving qualification to higher education entrance qualification), website characteristics (content: F(4, 324) = 1.88, p = .11; usability: F(4, 324) = 1.02, p = .40; aesthetics: F(4, 324) = .78, p = .54) or the intention to revisit (F(4, 324) = 2.06, p = .09). Finally, age did not correlate with ratings of depression (r =  − .03, p = .58), content (r =  − .05, p = .36) or aesthetics (r =  − .06, p = .26), while it showed a significant relationship with usability (r =  − .11, p = .04) and the intention to revisit (r = .15, p < .01).

Table 1 shows means, standard deviations, and intercorrelations between all variables under study. As objective usability was not significantly correlated with depression severity or intention to revisit, it was excluded from further analyses. The remaining variables are correlated as expected: depression shows significant negative relationships with all three website features (content, usability, aesthetics) measured via questionnaires (−.15 ≤ r’s ≤ − .13; all p’s <.05). Again, according to expectations, the intention to revisit is significantly positively correlated with all three dimensions of web user experience (.39 ≤ r’s ≤.69; all p’s <.01).

Using structural equation modeling, the hypothetical relationships were investigated simultaneously (see Fig. 2; Table 2). All item loadings of the parameters integrated in structural equation modeling were >0.5 (Table 1). The pathway estimates (β) are reported in a standardized format. As hypothesized, depression was significantly negatively associated with content, usability and aesthetics (−.15 ≤ β’s ≤ − .13; all p’s <.05). However, with regard to the website features, only content revealed a significant positive association with the intention to revisit (β = .67, p < .01).

Overall, the model fitted the data satisfactorily. The chi-square test as a measure of exact fit shows the preferable non-significant result (chi square = 3.267, df = 1, p = .071). The same applies for the additional measures of approximate fit: the RMSEA (=.083) lies between .08 and .10, thus providing a mediocre fit (MacCallum, Browne & Sugawara, 1996). This seems defendable when analysing a model with various parameters leading to restrained fit with regard to the RMSEA. CFI (=.996) and TLI (=.963) exceed the cut-off 0.95 and therefore indicate a good fit (Hu & Bentler, 1999). As for the SRMR, values less than .05 are obtained by well fitting models (in this study SRMR = .017; Byrne, 1998).

Discussion

Applying the hypothetical model, we find that depression significantly influences all three dimensions of website user experience measured by questionnaire (confirming H1, H2a, H3). In line with previous research results (Thielsch, Blotenberg & Jaron, 2014), significant path coefficients leading from website features to the intention to revisit them can be reported for content only (confirming H4), while the remaining UX features did not significantly influence this intention (rejecting H5, H6).

Furthermore, we did not find significant associations with an objective measurement of usability (rejecting H2b). There are several possible explanations: (a) There is no measurable impairment due to depressive symptoms in online performance at all; (b) such a decline is only to expect in severe depression (which is not focused in the present study); (c) the possibility that, even as the tasks used in this study were typical for Internet use, they are too easy to capture impaired performance caused by depression. The differences between individuals low on depression severity compared to those high on depression severity are apparent in the significant path coefficient leading from depression to subjective usability, but possibly could be compensated by the individuals affected when it comes to fulfilling the tasks capturing objective usability. In the latter case (actual impairment could not be detected), the operationalization of objective usability should be revised to once again try to capture impaired performance. Additionally, as data from only one website were collected in the whole sample, a replication of the study appeared to be necessary to test the robustness of the findings.

Method

Results

As in Study 1, we first searched for significant relationships between variables of interest and demographic factors. Again, gender did not reveal significant group differences regarding website features (content A: t =  − .12, p = .90, B: t =  − .44, p = .66; usability A: t = 1.06, p = .29, B: t =  − .22, p = .82; aesthetics A: t =  − .09, p = .93, B: t =  − .57, p = .57) or the intention to revisit (A: t =  − .79, p = .43, B: t =  − 1.23, p = .22), but was significant for depression (t = 3.12, p < .01; female participants: M = 17.45, SD = 6.19; male: M = 15.57, SD = 5.61).

Furthermore, the level of education did not differ for depression (F(4, 387) = .91, p = .46), website characteristics (content A: F(4, 387) = 1.05, p = .38, B: F(4, 387) = 1.43, p = .22; usability A: F(4, 387) = 1.04, p = .39, B: F(4, 387) = .94, p = .44; aesthetics A: despite a significant main effect, F(4, 387) = 2.61, p = .04, post hoc comparisons using Bonferroni correction did not reveal significant differences between the five educational groups ranging from no school-leaving qualification to higher education entrance qualification, B: F(4, 387) = 2.17, p = .07) nor the intention to revisit (A: F(4, 387) = .94, p = .44, B: F(4, 387) = .31, p = .87).

Finally, age did not correlate with ratings of content (A: r =  − .003, p = .96, B: r =  − .02, p = .71), usability (A: r = .01, p = .82, B: r = .10, p > .05), aesthetics (A: r =  − .04, p = .45, B: r =  − .05, p = .33), but partly with the intention to revisit (A: r = .12, p = .02, B: r = .08, p = .13) and with depression (r =  − .20, p < 0.01). As male participants (M = 47.28; SD = 12.34) were significantly older (t =  − 4.69, p < .01) than female (M = 41.44; SD = 11.82) this relationship between age and depression might partly be mediated by gender (gender → depression: β =  − .11, p = .03; gender → age: β = .23, p < .01; age → depression: β =  − .18, p = .01).

Table 3 (referring to website A: “Physical training”, with information on health exercises) and Table 4 (referring to website B: “MedOnline”, with health information) show means, standard deviations, and intercorrelations between all variables under study. Again, the total score referring to the search and memory tasks (objective measure of usability) did not significantly correlate with depression severity (A: r =  − .03; p = .62; B: r =  − .06, p = .27)³ and correspondingly was excluded from further analyses again. However, we did find significant correlations between objective usability and the intention to revisit in both cases (A: r = .22, p < .01; B: r = .19, p < .01), possibly indicating an improvement in operationalization.

Comparing the ratings (see Tables 3 and 4 for M and SD) for website A (actually existing) and B (mock-up website), there were no significant differences concerning the ratings of content (t =  − 1.66, p = .10), nor the reported intention to revisit (t =  − 1.79, p = .08). We found significant differences with regard to subjective usability (t = 4.71, p < .01) and aesthetics (t =  − 5.76, p < .01): while the mock-up website was evaluated to be more usable compared to the really existing website, it was perceived as less aesthetic.

While the findings regarding website A (correlations and paths; see Fig. 3, Tables 5 and 6) resemble the findings in Study 1 (depression and UX: −.15 ≤ β’s ≤ − .12, all p’s <.05), the findings referring to website B slightly differ: the pathways leading from depression to content and usability do not emerge as significant (−.09 ≤ β’s ≤ − .08, .09 ≤ p’s ≤.16). However, the impact on aesthetics remains significant in A (β =  − .15, p < .01) and also in B (β =  − .10, p = .04). Likewise, the path leading from content to intention to revisit is significant in both cases (A: β = .69, p < .01; B: β = .67, p < .01).

On the whole, the hypothetical model fitted the data. As the chi-square test is considered very strict in modeling large sample sizes (and undesirably does become significant in B), alternative measures of approximate fit were added. As shown in Table 6, good fit is indicated for A (CFI = 1.000, TLI = 0.998 and RMSEA = 0.022, SRMR = 0.008), while the model fit (cf. 2.2) in B appears to be worse mostly with regard to RMSEA (CFI = 0.993, TLI = 0.929 and RMSEA = 0.123, SRMR = 0.020). Correspondingly, reducing the model (B) by the non-significant paths (cf. Fig. 3 and Table 5) leads to a strong improvement (Chi-square = 9.531, df = 4, p = .05) meeting a cut-off value close to .06 (RMSEA = 0.059) displaying good fit by the abridged model (Hu & Bentler, 1999).

Discussion

Comparing the models regarding both websites in this study, we find again shared evidence for the pathway between depression and aesthetics (confirming H3), as well as for the one leading from content to the intention to revisit (confirming H4). The importance of the remaining paths we expect to be relevant due to theoretical considerations (from depression to content and from depression to usability) differs between websites: while the evaluation of the actually existing website (A) involves a biased perception of content (confirming H1) and usability (confirming H2a) due to depression, the mock-up site (B) partly prevented such a distortion and the path coefficient (rejecting H1 and H2a) did not reach significance.

As we now find a significant correlational association of objective usability and the intention to revisit in both cases (A and B), we can assume that the operationalization of this website feature is improved. However, we still do not find significant correlational relationships between depression and performance as the measure of objective usability (rejecting H2b). The low medium score on the objective usability tasks indicates that items were not too easy to answer. Thus, objective usability seemed to be not notably impaired by depression severity, at least in the range analysed in the present studies.

Practical implications

As the use of online (self-)help appears as a promising approach in altering depressive symptoms and avoiding barriers of traditional offline-treatment (e.g., Knowles et al., 2014; Richards & Richardson, 2012), the present studies’ results may help to prevent possible difficulties in designing such online environments. Content, usability and aesthetics will have to withstand a possibly biased evaluation given by people suffering from depression. Importantly, as we investigated the variables in general population, we might underestimate the strength of the (indeed small) effect—that may be more pronounced in a more severe depressed sample and that can only be observed in its entirety in a clinical sample.

As content appears to be crucial in deciding whether a website is visited again (which will be necessary to bring about a change), the first goal should be to improve comprehensibility, credibility and perceived benefit of a website. This could be achieved by extensively investigating the users’ expectations, fully responding to those, or possibly link together series of websites fulfilling the requirements as a whole. Furthermore, a website with controlled usability leading to superior subjective ratings (Study 2, mock-up website) antagonized the negative influence of depression on perceived usability. This attempt should be analysed more thoroughly to define helpful improvement suggestions for designing e-health websites. In the present study, the mock-up website was optimized in terms of an easy navigation and concise lists of relevant topics on first-level subpages.

Finally, our data show that depression is not necessarily influencing objective performance scores. Thus, in website evaluations with users it is important to take objective and as well subjective measures into account, as they seem to differ. Additional checklists or expert-based investigations of a website in question (see e.g., Hasan & Abuelrub, 2011; Prusti et al., 2012) might help to estimate the amount of bias in subjective evaluations of depressive users.

Limitations and future research

Some limitations should be considered while interpreting the results of our studies, most of them directly point at avenues for future research.

Firstly, one has to keep in mind that we tested in the general population applying a web-based screening approach. Thus, there was no individual diagnostic procedure enclosing measures such as interviews. There is a potential bias in self-administered scales. Yet web-based survey methods are highly feasible for research as such bias is reduced, among other things due to participants perception of high anonymity and thus a lower social desirability bias (e.g., Crutzen & Göritz, 2010; Kreuter, Presser & Tourangeau, 2009; Pealer et al., 2001). Furthermore, investigating the general population is a basic step before testing in clinical samples and potentially burdening patients with extensive experimental setups (e.g., Krampen, Schui & Wiesenhütter, 2008). Yet a necessary and interesting next step would be to replicate the findings testing in clinical samples of people affected by MDD. An effect that can be observed in a non-clinical sample, where the phenomena of interest are less pronounced, is expected to be of relevance and more pronounced in a clinical sample.

Secondly, we found effects of demographics on outcome measures, in particular of age on usability (in Study 1) and on intention to revisit (in Study 2). The influence of age found in Study 1 is in line with prior research (Sonderegger, Schmutz & Sauer, 2016) and illustrates the general relevance of such user characteristics in user experience studies.

Thirdly, even though we extended the tested websites and replicated results in Study 2, the restricted range of stimuli, all dealing with topics related to e-health, does not allow drawing generally valid conclusions. Hence, further investigation of preferably diverging websites is needed. In addition, a huge amount of mobile phone depression apps is available (see Huguet et al., 2016; Shen et al., 2015), presenting another highly interesting field for user experience research.

Fourth, as we do not find substantial relationships between depression and objective usability, further investigation is needed to establish whether the effect might exist in severe depression or for other aspects of objective usability—or objective usability is not impaired by depression at all.

Finally, we focused on information websites and thus used the intention to revisit as important dependent variable. In clinical practice, online interventions are of high interest (Andersson & Cuijpers, 2009). In such studies one main variable of interest would be drop-out. Intentions are positively correlated to actual behavioural outcomes (e.g., Prestwich, Perugini & Hurling, 2008) and a positive user experience facilitates actual revisiting (Van ’t Riet, Crutzen & De Vries, 2010). Future research on health interventions online will benefit from a detailed analysis of web user experience, intentions, and actual revisiting (respectively drop-out). Thus, the present two-part study might be a valuable step towards the investigation of web user experience in the light of mental problems. In further studies, the exact mechanisms and pathways should be examined, to determine how depression as a symptom cluster exactly alters the web user experience (for example via sleep or concentration problems).