Measuring health literacy among low literate people: an exploratory feasibility study with the HLS-EU questionnaire

To develop a modified version of HLS-EU-Q16, one researcher (HS) examined HLS-EU-Q16 on sentence structure and language. Level 'B1' as defined in the Common European Framework of Reference for Languages was used as a reference, because documents of this level are supposed to be understood by 95% of the population speaking that language [30]. This preliminary analysis was carried out in an intuitive manner, using http://​www.​zoekeenvoudigewo​orden.​nl to look up words on B1-level and http://​www.​accessibility.​nl/​kennisbank/​tools/​leesniveau-tool to verify sentence structures. Subsequently, an expert organization on accessible communication (non-profit organization 'Wablieft vzw') was tasked to independently develop an alternative version of the questionnaire, called HLS-EU-Q16-EZ. Questions assessed as too complex (use of jargon or difficult words, subordinate clauses) according to expert organization’s opinion, were rewritten in lay-language and everyday speech. Moreover, the expert organization was instructed to only modify if necessary and, if adjustments were made, to rewrite them with a maximum resemblance to the original formulation.

Purposive sampling was used to select participants for the study. Inclusion criteria were: being aged between 18 and 70 years, a low level of education (preferably primary school, highest being high school because of the compulsory schooling in Belgium until the age of 18) and low income (defined as a gross family income in the previous year ≤16,965.47 euro for the applicant, raised with 3140.77 per person living on the same address; or receiving a specific reimbursement designated for people who are considered to be in a ‘vulnerable’ position). To illustrate, low income requirements in 2014 were set at maximum 16,743.70 euro in the previous year for the applicant (and 3099.72 per additional person), while an average gross income then amounted 46,197 euro [31‐33]. To participate, being able to read and understand Dutch was a language requirement, although it was not necessary to be a native speaker. Exclusion criteria were: having a psychiatric and/or neurodegenerative disorders (e.g., Alzheimer’s, Parkinson, ...); (being formally recognized as) having physical, intellectual or sensory impairment(s); or not being able to interact with the researcher. To recruit people eligible for participation, 11 organizations working with people in a vulnerable position were contacted, which is a methodology similar to the ones used in previous research with vulnerable groups [34, 35]. The number of eligible participants was difficult to determine beforehand: recruitment was scheduled on informal gatherings with fluctuating number of people showing up. Saturation in participants’ opinions was pursuit, rather than a predetermined sample size, as a consequence of challenging recruitment.

To determine the suitability of HLS-EU-Q16, cognitive interviews about the questionnaire were performed by one researcher using the Verbal Probing technique [36]. The suitability of the questionnaire was operationalized as participants’ opinion about: the length of the questionnaire and questions; comprehension of questions and language; clarity of instructions; the response process; the layout of the questionnaire and any other remarks not directly addressed by the researcher.

Participants randomly received either HLS-EU-Q16 or HLS-EU-Q16-EZ, dividing them in two groups. To avoid stigmatization, everybody who was present at the meeting when the interviews were scheduled to be held, could complete the questionnaires that had been randomly distributed to the group, but only those meeting the inclusion criteria were interviewed directly afterwards. The purpose of the study was explained, participants were assured of the anonymity of the study and of the fact that they would not be judged, which stimulated them to give their honest opinion about the questionnaire. Information was gathered through a semi-structured interview, carried out by the same researcher. Interviews were audio-taped to facilitate analysis afterwards. Interviews were ended because of data saturation.

To construct a questionnaire on B1-level, at least 12 items would have needed to be revised based on vocabulary and/or sentence structure (Q1, Q4 – Q13, Q16).

Difficulties in completing the questionnaires (HLS-EU-Q16 = A; HLS-EU-Q16-EZ = B) as reported by participants in the interviews, are summarized in Table 2. Words that were unclear in A-version were ‘preventive’ (Dutch ‘preventief’) (Q10), ‘mental wellbeing’ (Dutch ‘geestelijk welzijn’) (Q13), ‘to judge’ (Dutch ‘beoordelen’) (Q5 and Q11) and ‘media’ (Q12).

Non-responses were analyzed quantitatively. In Table 3 missing responses are illustrated. The sum of non-responses generated in the group completing the HLS-EU-Q16 was higher (n = 17) than the number of total non-responses for those who filled out the HLS-EU-Q16-EZ (n = 10). Incomplete questionnaires were generated by 7 of 13 participants, of which four filled out the HLS-EU-Q16; one participant completed only half of HLS-EU-Q16.

When referring to the initial framework of HLS-EU questionnaire [3], questions often reported to be less comprehensive were those about ‘disease prevention’ (domain) or ‘appraisal’ of information (competency). Indifferent of domain or competency, all questions referring to a potential role of the media in health care seemed hard to relate to. Referring to one’s general practitioner as a source of health information was an apparent and frequently mentioned response. This statement was most often accompanied by the comment that respondents had difficulties deciding how to respond. Indirectly, these findings revealed the trust in general practitioners and consequently their crucial position in health care, particularly for vulnerable groups (people with low levels of income and education) [39‐41].

Some difficulties persisted even when HSL-EU-Q16-EZ (the modified questionnaire) was presented to participants. Because data collection with HLS-EU-Q16 was not carried out face-to-face, as opposed to HLS-EU-Q47 in which Computer Assisted Personal Interviewing (CAPI) was used, people of low literacy groups could benefit ​from a minor adjusted HLS-EU-Q16 [21]. To resolve both comprehension and decision process problems, some questions would benefit from an additional explanation. Providing extra information in short, explanatory sentences can bypass problems concerning the meaning of words. Moreover, it may rule out confusion amongst respondents by giving more context and making it easier for people of low literacy groups to respond. The suggestion for contextual information is made based on reports of confounding questions: some questions were perceived as too similar, leaving respondents confused about their objective. To situate questions, providing information with regard to the ‘competencies’ to which questions are referring to (for instance, distinguishing questions about ‘appraising’ from those about ‘applying’) could facilitate interpretation. In conclusion, it is advisable to add information similar to the information that would be provided by the interviewer when collecting data based on CAPI (HLS-EU-Q47). Although occurring rarely, questions can truly be irrelevant because of very specific situations: for instance, someone who needs to rely on health care professionals’ (or informal caregivers’) decision because of their medical condition, might find some questions irrelevant; someone with no or limited access to 'the media'. In these rare cases, people would benefit from an additional category to report accordingly; adversely, it increases the likelihood of non-responses. When the questionnaire would be used with forced answering (electronically), benefits and disadvantages of an extra category should be outweighed.

The following changes might improve the understanding of the questions: basic instructions can allude on how questions should be interpreted, guidance on answering options can be provided in case people would feel to only ‘rely on their general practitioner or other (non-)medically trained people’; questions can be broken down into different components, so it can be avoided that people find the portrayed situation in a part of the question irrelevant to them; the meaning of terminology might be explained through an example, for instance of a preventive examination, or questions might be put into a different, more clarifying order (per domain or competency surveyed) [42]. These suggestions should be implemented cautiously, avoiding narrowing the questionnaire to specific health-related topics. However, some additional information might increase comprehensibility and potentially response rate.

In this research, the focus was on HLS-EU-Q16 (in Dutch). Because HLS-EU questionnaires are available in different languages, but not all of them were tested on ‘cultural applicability’, it would be helpful to determine feasibility of HLS-EU questionnaire beforehand, preferably by surveying or interviewing the intended users [7, 38, 43]. Evaluating the questionnaire’s suitability and eventually readjusting it, might be particularly useful if questionnaires are intended to be used in a paper-pen version, without assistance from a researcher [44]. Because assessing feasibility is subjective, adjustments made in correspondence with respondents’ feedback are specific for those interviewees and therefore not generalizable.

A strength of this study is the insight that was gained with regard to the cognitive process of respondents when completing HLS-EU-Q16. Moreover, this study was specifically targeting a population considered to be vulnerable because of low levels of education and income. To the best of our knowledge, this was the first study targeting this population to examine feasibility of a questionnaire. Similar studies have been carried out in a peadiatric setting or targeting vulnerable groups, such as refugees. [43, 45]. The research of Wångdahl et al. examining HL in refugees, indicated sufficient language proficiency – preferably respondents’ mother tongue – and modification of questions or answering options to be contributing to questionnaires’ accessibility. In their research, HLS-EU-Q16 (in Swedish) was modified based on data collected through cognitive interviews [43].

The main limitations of this study are related to the vulnerable character of the population. Recruitment was challenging: an unusual, yet inclusive approach was used to reach eligible participants in order to avoid stigmatization. Moreover, reaching people of younger age meeting the education requirement seemed even more challenging, which was to be expected given the compulsory education until the age of 18 years old in Belgium. Despite efforts to reach this population, it still resulted in relatively small sample size, but as testing suitability is a qualitative approach, it is not based on representative samples. Data saturation was reached when 13 participants were interviewed, a sample regarded as sufficient [46].

Although a test-retest design would have allowed to generate interesting data, reproducibility was not feasible. Participants seemed to find it difficult to distinguish between answering the 16 questions and answering on the feasibility of these questions (interviewer’s asking about how the questionnaire was perceived often resulted in respondents reporting about their answers). Asking them to participate in a second, similar survey would probably lead to more confusion.

Interviews were conducted using the Verbal Probing technique: the researcher attempted to minimize the use of ‘leading’-probes to reduce bias. However, as the purpose of this interview was to analyze comprehension of questions, an example or clarifying sentence was provided, with the intent to get insight in the difficulties reported by participants and potential solutions, but never from the start. Ideally, the interview would have been conducted during the completion of the questionnaire. However, the time between completing the questionnaire and the interview was minimalized to 1 h maximum. A final limitation is that this research was carried out with paper-based questionnaires only. The findings presented in this study are not necessarily transferable to the same instrument in a web-based version. Working with digital surveys requires a specific set of skills: these were not taken into account in this research.

Based on reported difficulties, an altered HLS-EU-Q16 can be developed to determine HL in people with low literacy. Based on the findings of this research, some people with potential limited HL might benefit from some questions being formulated more clearly, for instance by adapting wording or by providing contextual information to facilitate interpretation [42]. Moreover, a complementary, modified questionnaire could be provided or it might be useful to determine HL through interviews as opposed to surveying with pen and paper [47].