Establishing the HLS-Q12 short version of the European Health Literacy Survey Questionnaire: latent trait analyses applying Rasch modelling and confirmatory factor analysis

Sørensen et al. [11, 15] developed the HLS-EU-Q47 items to reflect their conceptual model, and suggested a 4-point rating scale with response categories ranging from very easy (1) to very difficult (4). In the present study, the rating scale was reversed to make a higher score indicate a higher proficiency at the latent trait. A ‘don’t know’ category, which was later recoded to missing data, was added to record such spontaneous utterings from respondents during the telephone interviews.

The translation procedure, that involved forward and back translation, is thoroughly described in Finbråten et al. [20]. Cognitive interviews were used to explore item interpretation and clarify any ambiguities.

The null hypothesis of a locally independent scale is weakened and might be rejected in favour of the alternative hypothesis, suggesting a multidimensional scale, in the following situations: When the proportion of individuals with significantly different person-location estimates on a pair of related subscales exceeds 5% (or the lower bound of the binominal 95% confidence interval (CI) exceeds 5%) [40‐43]; when the subtest analysis in RUMM points to low common subscale variance and high unique subscale variance (i.e. low fractal index A < 0.8 and high fractal index c close to 0.5, respectively) [44]; when a multidimensional Rasch model obtains a better data-model fit than a one-dimensional Rasch model (i.e. significantly lower deviance or -2log-likelihoods using a likelihood-ratio test for nested models and AIC for non-nested models) [45]; and/or when absolute values of Rasch-model residual correlations among items exceed 0.3 [46].

Likelihood-ratio tests (LRT) were used to compare data-model fit for nested models, such as the one-, two-, three- and 12-dimensional Rasch models, where the first implies parallel subscales (correlations between all theoretically defined subscales were fixed at 1) and the latter implies freely correlated subscales. The LRT statistic “difference in deviance” is asymptotically χ2 distributed with degrees of freedom (df) equal to the difference in the number of estimated model parameters comparing two nested models [47]. Akaike’s information criterion (AIC) [48] was used to compare the data-model fit for non-nested models, such as the various short versions. Similar to Allen and Wilson [49], both deviance and AIC are reported when applying the consecutive approach to the HLS-EU-Q47 (calculated by adding the values for each of the three subscales). Lower values of deviance and AIC indicate better data-model fit. In addition to deviance and AIC, total item chi-square with p-values was used to assess the data-model fit of the short versions.

Items were interpreted as under−/over-discriminating relative to the Rasch model when the infit MNSQ were significantly different from the expected value of 1 (indicating perfect fit to model) or, more precisely, when the infit value was above/below the 95% CI and the corresponding t value was > 1.96 [45, 50].

In addition to infit, chi-square probability was used to assess item fit for the various short versions. Items were considered as misfitting at high chi-square values and probability values lower than a Bonferroni-adjusted 5% [51]. Item difficulty was described using item-location estimates.

Applying the consecutive approach to the HLS-EU-Q47 and the one-dimensional approach to the short versions, differential item functioning (DIF) was explored using analysis of variance (ANOVA) [40] for the dichotomized person factors of gender, age and level of highest completed education: age was dichotomised as above or below the sample mean (47 years), and highest level of completed education was split into “compulsory and upper secondary school” versus “university level”.

The ordering of the response categories was examined applying the one-dimensional and the consecutive approaches. Significantly different thresholds in “correct” order [52, 53] were used as evidence for ordered response categories. The targeting of the (sub)scales was evaluated by comparing the distribution of the item threshold estimates to the distribution of the person estimates [51]. For more detailed descriptions of the analyses performed, see for example, Andrich [30], Hagquist [53], Hagquist, Bruce and Gustavsson [54] or Finbråten et al. [20].

The null hypothesis of a one-dimensional scale is weakened when overall fit indexes for multifactorial CFA are superior to a one-factorial CFA [55]. Owing to categorical raw data, we defined the variables as ordinal in the “lsf” file and estimated the polychoric item correlations and their asymptotic covariance matrix using the PRELIS application to LISREL [56]. As target values for goodness-of-fit (GOF) indexes is based on maximum likelihood estimation (ML), we used robust ML to obtain the Satorra-Bentler scaled chi-square GOF index (SB scaled χ²) [57, 58]. The following overall GOF indexes for model comparison based on the null model were used: SB scaled χ², standardized root mean square residual (SRMR), root mean square error of approximation (RMSEA), comparative fit index (CFI) and non-normed fit index (NNFI, or Tucker-Lewis index [TLI]). Schumacker and Lomax [59] recommend SRMR < .05, RMSEA ≤ .05, and CFI and NNFI ≥ .95 as GOF index target values, whereas Hu and Bentler [60] claim that SRMR and RMSEA values close to or below 0.8 and 0.6, respectively, indicate sufficient overall fit. The AIC and the Bayesian information criterion (BIC) were also taken into account. AIC and GOF indexes were used to compare the various short versions and to compare the one-factor to the three-factor approaches for the various short versions. Item communalities (squared factor loadings for completely standardized solution) describe the shared variance among the items that is accounted for by the latent variable, HL [61].

Person separation index (PSI), person separation reliability (PSR) and the H coefficient [62] (obtained from RUMM, ConQuest and calculated manually based on communalities from CFA) were used as measures of reliability. Values exceeding 0.85 (0.65) are sufficient if conclusions are to be drawn at the individual (group) level [63].

Overall analyses included those for dimensionality, data-model fit, response dependence and reliability. The initial analyses of dimensionality indicated that neither the entire HLS-EU-Q47 nor the three health domains, HC, DP and HP, were sufficiently unidimensional (the respective proportions of significant t-tests were 21, 9, 13 and 10% [not reported in Table 2]).

Using LRT, the observed data were more likely under a 12-dimensional Rasch model than a one-, two- or three-dimensional Rasch model (Fig. 1). As expected, the data-model fit improved significantly in each step from the one-dimensional through the 12-dimensional model. For example, the drop in deviance from the three- to the 12-dimensional model was ΔD = 3878 ≈ χ² [Δdf = Δestimated parameters (ep) = 72], p < 0.01, critical value = 103. The three-dimensional model had a lower AIC and a better fit than the consecutive approach of the three health domains, HC, DP and HP (ΔAIC = 1129). The correlations between HC and DP, HC and HP, and DP and HP were r = 0.81, 0.71 and 0.83, respectively.

Applying the 12-dimensional approach to the HLS-EU-Q47, we observed relatively low subscale reliability indexes, most likely owing to few items per scale—on average 3.9 items per subscale (see PSR values in Fig. 1). Consequently, we observed acceptable reliability indexes for the one-dimensional approach and for the subscales of the two- and the three-dimensional approaches.

Analyses at the item level included analyses of item fit, item discrimination, DIF and ordering of response categories. Table 2 shows the item fit when applying the one-, three- and 12-dimensional approaches.

Under-discriminating items were observed using the one- (items 12, 29, 38, 45 and 47), three- (items 11, 12, 29, 30 and 38) and 12-dimensional (items 12, 18, 19, 38 and 43) approaches; items 12 and 38 under-discriminated in all analysis approaches. Over-discriminating items were also observed when the one- (items 16, 19, 20 and 40), three- (items 8 and 40) and 12- dimensional (item 16) approaches were applied. Using the consecutive approach, several items displayed DIF (Table 2).

Unordered response categories were observed for items 8, 15 and 21 when applying the one-dimensional approach, as well as when analysing the HC, DP and HC health domains consecutively (Table 2).

According to the t-tests, none of the short versions could be deemed sufficiently unidimensional (the proportions of significant t-tests were above 5% and the lower binominal 95% CI proportion was 0.07 for both the HLS-Q12 and the HL-SF12, whereas the lower binominal 95% CI-proportion was 0.09 for the HLS-EU-Q16 (Table 3).

Nevertheless, the fractal indexes indicated that the HLS-Q12 and HL-SF12 had relatively high amounts of common variance (A = 0.90 and 0.91, respectively), high subscale correlation (r = 0.80 and 0.87, respectively), and relatively high unique subscale variance (c = 0.50 and 0.39, respectively).

Using LRT, the data fit better to the three-dimensional models compared to the one-dimensional models of the various short versions (three-dimensional models had significantly lower deviance, except for the HL-SF12, where the difference was insignificant). The GOF indexes obtained via CFA were also better for the three-factor models as compared to the one-factor models of the various short versions (Table 4).

Comparing the short versions, the HLS-Q12 obtained best fit to the Rasch model (had the lowest total item chi-square value and the lowest AIC; Table 3) whereas the HL-SF12 obtained best fit when performing CFA (had the lowest likelihood estimates and the best GOF indexes). The HLS-Q12 also had acceptable GOF indexes (Table 4). The HLS-EU-Q16 displayed the weakest overall fit indexes (highest total item χ² value and highest log likelihood estimates).

All the one-dimensional short versions had acceptable reliability values (PSI and PSR values above 0.75 and the H coefficient was above 0.82). The HLS-EU-Q16 had the highest reliability indexes (PSI = 0.830, PSR = 0.826 and H = 0.882). The reliability indexes were slightly higher for the HLS-Q12 than for the HL-SF12 (Tables 3 and 4). Applying the three-dimensional approach, the PSR and H values of the correlated subscales were below the recommended values (Tables 3 and 4). The HLS-Q12 was the best-targeted, with a mean person location of 0.759. The mean person locations for the HL-SF12 and HLS-EU-Q16 were 0.816 and 1.020, respectively (not shown in the table).

Compared with the HL-SF12 and the HLS-EU-Q16, the HLS-Q12 showed best fit at the item level as there were no occurrences of misfitting items, items displaying DIF with regard to available person factors or items with unordered response categories. For the HL-SF12, items were within-item biased (items 15, 26 and 30 displayed DIF), and one item with unordered response categories was observed (item 15). Problems with regard to DIF (items 5, 11, 13, 21 and 37) and unordered response categories (items 8 and 21) were also observed for the HLS-EU-Q16. In addition, item 11 under-discriminated and showed significant misfit (chi-square probability = 0.001; Table 5). A high level of unique variance was observed for the items of the various short versions (relatively low communality values).

Investigating the item-location estimates of the HLS-Q12, items 28 (judge if the information on health risks in the media is reliable) and 10 (judge the advantages and disadvantages of different treatment options) had the highest location estimates (1.068 and 0.892, respectively) and were thus the hardest to endorse. Items 23 (understand why you need health screenings) and 32 (find information on healthy activities such as exercise, healthy food and nutrition) were the easiest to endorse (item-location estimates of − 1.158 and − 1.135, respectively).

A 12-dimensional model described the HLS-EU-Q47 data best. This result is perfectly in line with prior research [20, 21]. Hence, applying the HLS-EU-Q47 recommends a complex 12-multidimensional approach returning 12 subscale-scores for each individual. This is of little practical use, especially as proficiency cannot be compared across the different subscales owing to relative points of zero. Owing to few items in each subscale, few of the 12 subscales were sufficiently reliable.

Contrary to common practice when using the HLS-EU-Q47, we cannot recommend reporting either total or health domain subscale scores. The three-dimensional approach, reflecting the three health domains, obtained better data-model fit than the one-dimensional approaches. The three health domain subscales each returned sufficiently large reliability indexes, but lower indexes than those reported by the HLS-EU Consortium [11], Nakayama et al. [18] and Duong et al. [19].

Prior research on people with chronic disease (type 2 diabetes) [20] indicates that the HC subscale brings notable multidimensionality into the data. It is very interesting that this not was supported in our recent analyses of a national sample, because it indicates that the “health care” subscale actually work differently for patients with chronic disease than in the population as a whole. This seems reasonable, as people with chronic disease have more experiences with using health care facilities than the general population.

Like previous studies [20, 21] we observed items violating local independence, items displaying DIF, and poorly fitting items. Poorly fitting items which over- or under-discriminate tap into constructs other than the latent trait [64]. Over-discriminating items measure “too much of something else” that is positively correlated with the latent trait and are therefore viewed as less problematic than under-discriminating items. Contrary to Finbråten et al. [20], who found that few items displayed DIF in people with chronic disease (type 2 diabetes), we found several items displaying DIF for age and education, which means that people with higher versus lower levels of education, as well as younger versus older people, probably perceive these items differently. Consequently, comparisons of HL across age groups or educational levels would be invalid [54]. The increased number of items displaying DIF in our study might be expected as the sample of people with type 2 diabetes in Finbråten et al. [20] mainly consisted of elderly persons.

Based on the PCA and t-test procedures in RUMM, none of the short versions stood out as sufficiently unidimensional. However, considering the lower binominal 95% CI proportions of 0.07, the two 12 item versions HLS-Q12 and HL-SF12 could be considered sufficiently unidimensional [54]. Hagell [41] claims that the width of the binominal 95% CI is influenced by sample size. When investigating dimensionality in a larger sample, binominal 95% CI-proportions might represent acceptable values. Moreover, unidimensionality is a relative matter and depends on the level of precision. Evaluation of unidimensionality should also include an assessment of the purpose of the measurement, together with theoretical and practical considerations [30, 41].

Comparing the one- and three-dimensional approaches to the short versions, the three-dimensional approach obtained lowest deviance and sufficient values for the GOF indexes. Like some previous studies [32, 65] we found that Rasch modelling and CFA returned similar results. Thus, a three-dimensional approach could be recommended for the short versions. However, when applying a three-dimensional approach the three subscales obtained low reliability indexes. The difference in deviance between the one- and three-dimensional approaches was significant; but the differences were rather small for the two 12 item versions.

A one-dimensional approach to the HLS-Q12 and HL-SF12 could probably be defensible as the results point to minor violations of unidimensionality. Further, the one-dimensional approach yields higher reliability indexes than each of the subscales of the three-dimensional approach. Providing one HL score for each individual, the one-dimensional approach to the HLS-Q12 and the HL-SF12 is practical from a clinical point of view. In addition, the HLS-Q12 and HL-SF12 are in line with the conceptual model of Sørensen et al. [16], but dimensionality should be further explored in future studies.

Comparing the psychometric properties of the various short versions, the HLS-Q12 obtained best fit to the Rasch model, whereas the HL-SF12 obtained best fit through CFA. The HL-SF12 was found to have several weaknesses, including items with unordered response categories (item 15) and several items displaying DIF (items 15, 26 and 30). Owing to a larger number of items, the HLS-EU-Q16 stood out as more reliable than the other short versions [63]. However, the HLS-EU-Q16 could not be considered sufficiently unidimensional, it had the highest AIC, and did not yield acceptable GOF indexes. In addition, problems regarding item misfit, DIF and unordered response categories were observed.

Altogether, the HLS-Q12 has better psychometric properties than the HL-SF12 and the HLS-EU-Q16. The HLS-Q12 can be deemed the best-targeted scale and free from under-discriminating items and DIF. The HLS-Q12 reflects the conceptual framework, and it is well balanced because it consists of one item from each of the 12 dimensions. The HLS-Q12 version could be well suited for use in HL screenings at both the individual and community levels. Nevertheless, in future studies, we recommend extending the number of response categories from four to six to increase the reliability of the scale [66]. However, item 38 represents a specific concern because this item under-discriminated when applying one-, three- and 12-dimensional approaches to the entire HLS-EU-Q47. In contrast, when we applied a one-dimensional approach to the HLS-Q12, the item showed good model fit. In future validation of the HLS-Q12, this item should be further investigated.