Background
Care services for older adults are not primarily aimed at improving health, but rather at compensating health declines to preserve more general quality of life (QoL) aspects such as independence, daily functioning and social participation. Concerns have been expressed that benefits of these care services are underestimated in economic evaluations when health-related QoL measures such as the EQ-5D are used [
1‐
5]. Valuable aspects of care services for older adults may consequently be overlooked in policy making processes informed by these evaluations.
To address these concerns researchers recently developed the ICEpop CAPability measure for Older people (ICECAP-O) [
6] and the Adult Social Care Outcomes Toolkit (ASCOT) [
7], which include a broader set of QoL aspects. These measures were developed for the purpose of evaluating health and social care services and can be used in economic evaluations. Because of the inclusion of QoL aspects beyond health and the involvement of older adults in the development of the measures [
3,
6,
7], it is likely that the ICECAP-O and ASCOT more adequately reflect the objectives of care services for older adults and the perspective of older adults than the EQ-5D [
8].
When evaluating care services for older adults, it is important that the measures used validly assess the QoL of older adults and are feasible to use in samples of older populations. The measure should be comprehensive and adequately reflect the patient perspective (i.e.
content validity [
9,
10]), and the items should be comprehensible and acceptable; the target population should be able and willing to give responses to the questions (i.e.
feasibility). Poor content validity and feasibility increase the risk of misclassification, biased results, missing responses and irritation, boredom or intellectual withdrawal of respondents [
11‐
13].
The coverage of relevant QoL aspects and the operationalization of these aspects determine the content validity and feasibility of QoL measures. Aspects should be operationalized in such a way that older adults understand what is meant and that the items and response options correspond to the reality of their daily life. Previously, researchers that investigated survey response processes have shown that answers to questions are prone to a variety of response issues and that responses not always correspond with the developers’ intentions [
12,
14‐
21]. Response issues are a result of the interplay between the respondents’ motivation and capacities, the interview setting and attributes of the measurement instrument [
14,
17,
20,
22] and can arise during several stages of the response process, as presented in a model by Tourangeau [
17]. These issues may threat the validity and feasibility of the measures [
20,
22,
23]. For example, commonly reported problems concern interpretation difficulties due to double-barrelled or ambiguous questions, and response tendencies such as positive responding and acquiescence. In the case of QoL of older adults, adaptation and comparison mechanisms pose a challenge to measurement and may cause unexpected ways of responding [
12,
24‐
28].
Qualitative methods are increasingly used to include the target populations’ perspective on the content validity and feasibility of measures [
9,
12,
18,
21,
23,
29,
30]. Cognitive interviewing techniques such as the think-aloud technique provide insight in the response process. Threats to content validity and feasibility of measures can be revealed when the nature of response issues and the meaning of responses are identified and better understood [
9,
18,
21,
22,
29,
31].
Little is known yet about the extent to which response issues emerge using the ICECAP-O or ASCOT, and whether this differs from the EQ-5D. Although the items were in general interpreted as intended, some struggles with the ASCOT and the adult version of the ICECAP (ICECAP-A) were reported [
7,
32,
33]. Compared to the EQ-5D, more subjective and diverse interpretations of the terms in the ICECAP-A were found [
32]. A few studies showed that the ICECAP-A and ASCOT included aspects that were considered important, but according to respondents lacked coverage of health aspects as compared to the EQ-5D [
2,
7,
33]. No previous study has compared how older adults interpret and respond to the ICECAP-O, ASCOT and EQ-5D.
The objective of this study was to explore the content validity and feasibility of Dutch translations of the EQ-5D-3L, ICECAP-O and ASCOT from the perspective of older adults, by identifying response issues and comparisons of coverage and comprehensibility of domains.
Results
Respondents
The variation in characteristics of the ten respondents is shown in Table
2. All respondents reported at least two health issues, varying from hearing loss to cardiovascular diseases and diabetes. Two of them indicated to have some problems with memory, attention and thinking.
Table 2
Characteristics of the respondents
Mrs. N. | 90 | Alone | Amsterdam | Middle | 4 | Fair | Fair |
Mr. O. | 77 | With partner | West-Friesland | High | 2 | Fair | Unknown |
Mr. Q. | 100 | With daughter | Amsterdam | Middle | 6 | Good | Fair |
Mrs. S. | 88 | With partner | West-Friesland | Low | 5 | Fair | Fair |
Mr. U. | 75 | With partner | West-Friesland | High | 4 | Poor | Good |
Mr. W. | 67 | Alone | Amsterdam | Low | 1 | Good | Good |
Mrs. X. | 88 | Alone | Amsterdam | Low | 6 | Fair | Fair |
Mrs. Y. | 75 | With partner | West-Friesland | Middle | 2 | Good | Very good |
Mrs. Z. | 91 | Alone | West-Friesland | Middle | 4 | Poor | Good |
Mrs. A. | 91 | With partner | Amsterdam | Middle | 6 | Poor | Poor |
One respondent, Mr.Q. (pseudonyms are used), a 100-year old man, lacked the strength to complete all three measures. Mr. Q. only completed the ASCOT and his daughter helped him to elaborate on his response choices.
Response issues
Table
3 summarizes the response issues experienced by respondents for each item of the EQ-5D-3L, ICECAP-O and ASCOT in a matrix format. The issues that occured most widely across a variety of respondents or that pose a serious threat to the validity or feasibility will be illustrated below (definitions of response issues are shown in Table
1).
Table 3
Matrix of issues experienced by respondents versus items of the EQ-5D, ICECAP-O and ASCOT
Comprehension phase |
Understanding the words |
- | Odd wording | | | | | | | | | | | | / | | | | | | | |
- | Difficult wording | | | | | | | | | | | | | | | | | | /// | / |
Interpreting intended scope and meaning |
- | Difficult interpretation of item | | | | | | | | //// | | / | // | | | | | / | | ////// | // |
- | Wrong interpretation of item | | | | / | | | | /// | | | | | | | | | | | |
- | Narrow interpretation of item | | | ///// | ////// | ////// | /////// | /////// | | | //// | | | | ////// | | | /////// | | |
Selecting and reporting phase |
Mapping |
- | Different answers for different aspects | / | / | | | | / | | | | // | | | | | | | | | |
- | Response option partly applicable | | | | | | | | | | | | // | | | /// | | | | |
- | Irrelevant response option | | | | | | / | | | | | | | | // | //// | | | | |
- | Missing intermediate response options | // | | | | | | | | / | | / | | | | | | | | |
- | Similar response options | | | | | | | | | / | | | / | | / | | // | | | |
- | Disagreement with order of response options | | | | | | | | | | / | | | /// | | / | | | | |
Editing |
- | Positive responding | // | | /// | // | // | //// | / | | // | //// | /// | / | | / | // | / | | | |
EQ-5D-3L
Narrow interpretation of item
Three of the five questions from the EQ-5D-3L were more narrowly interpreted by one or more of the respondents than intended by the developers. Respondents focussed on one aspect of the double-barrelled items ‘Pain/discomfort’ and ‘Anxiety/depression’ and on one type of activities for the ‘Usual activities’ item. Where ‘Usual activities’ according to the developers encompasses social function, respondents mentioned domestic tasks and doing groceries while responding to this item. For example, Mrs. Z. (71) explained her choice [‘I have some problems with performing my usual activities’] with: “I am not able to do many things, that is why I’ve got somebody who takes care of my household activities”.
Mapping issues
Two respondents perceived a gap between response options on the ‘Mobility’ item, and considered the response options as a rough estimate, as illustrated by the comment of Mr. O. (76): “This is either black or white, in between are many possibilities”.
Positive answering
For all EQ-5D-3L items except ‘Selfcare’, it occurred that respondents picked a more positive response option than what an outsider would expect. For example, Mr. W. (67), who suffers from daily pain due to rheumatoid arthritis, chose ‘I have moderate pain or discomfort’ on the ‘Pain/Discomfort’ item because he compared his situation to that of others:
Of course I’ve got pain, but there are thousands who have similar complaints or even worse. Pain, I don’t know any better. I get up and go to sleep with pain. It’s just as part of my life as getting coffee. I have pain but don’t feel it anymore. There are worse things in the world. When I see children in a wheelchair I always think I should not complain.
ICECAP-O
Narrow interpretation of item
Some of the ICECAP-O items were more narrowly interpreted by the respondents than the developers intended. This was most pronounced for the ‘Attachment’ and ‘Security’ items. Rather than choosing the response level that overall fitted their situation best, respondents tended to concentrate on one aspect of a domain. When answering the ‘Attachment’ question, respondents focused on friendship and did not mention love or intimacy. On the ‘Security’ item (thinking about the future without concern), respondents focused either on financial insecurities or on worries about their or their partner’s health.
Difficult/wrong interpretation of item
The ‘Role’ item (doing things that make you feel valued) was difficult to understand. Four respondents explicitly stated that they didn’t understand how to interpret this item. Three others provided an answer but their explanation revealed that they thought about limitations in daily functioning rather than having a purpose that is valued. Mrs. N. (89) for instance, a woman living alone who much enjoys social outings, picked the answer ‘I am able to do many of the things that make me feel valued’ and said: “I am able to do many household tasks myself”.
Mapping issues
Compared to the EQ-5D-3L, there were a few more issues with the mapping of personal situations on the available response options. For example, Mr. O. (67) indicated that he missed an intermediate response option for ‘Enjoyment’, and Mrs. X. (87) considered the last two response options of ‘Enjoyment’ as similar. Mrs. Y (74), a cheerful woman who provides informal care to her husband, ticked the most positive answer of the ‘Control’ domain [‘I am able to be completely independent’]. However, she expressed that her situation was not the most ideal in her eyes. It was an undesirable consequence of her husband’s Alzheimer’s; she rather wanted to share decisions and activities with her husband.
Positive answering
Positive answering occurred most often on the ‘Attachment’ and ‘Control’ items. Mrs. N. (89) talked again about household tasks while picking the most positive answer on the ‘Control’ item [‘I am able to be completely independent’]: and states “I can do everything myself, I am completely independent, absolutely” while she earlier explained that she receives help for chores and financial tasks from numerous people. She conveyed earlier in the interview: “I don’t do anything in the household myself”.
ASCOT
Narrow interpretation of item
Two items were more narrowly interpreted than intended across various respondents: ‘Personal safety’ and ‘Accommodation cleanliness and comfort’. While answering the ‘Personal safety’ domain, respondents focused mainly on fear of crime, not thinking about fear of abuse, falling or physical harm. Mrs. A. (91), a woman who could not get outdoors due to the steep stairs and who during the conversation expressed fear of her dominant and aggressive husband, chose the first response option [‘I feel as safe as I want’] “Because I always lock the door. We do this since two strange men showed up in our house. When I don’t know who’s at the door, I open the door with the door chain set in place”. Concerning the ‘Accommodation’ item, respondents more often focused on cleanliness than on comfort of the home.
Difficult interpretation of item
Four items were hard to understand for several respondents. Some of them did not know which activities or aspects they ought to include or exclude in their answer on the ‘Control over daily life' and ‘Occupation’ items. Further, the ‘Dignity’ items were most poorly understood; respondents indicated they did not get what was meant by the questions and that they didn’t see a connection between having help and the way they feel and think about their selves. Mr. O. (76) decided to skip both ‘Dignity’ items:
Which of these statements best describes how having help to do things makes you think and feel about yourself? I think that’s a difficult sentence. Please explain, because I don’t get what they mean. I completely don’t understand this question. Would you mind if I skip this one? If you wouldn’t be here I would place a big cross on the question. I don’t mind the difficult phrase, but I just don’t get it. […] I never think and feel about myself, so what do they mean?
Mapping issues
Occasionally, respondents found it difficult to identify the difference between the response options. For example, Mr. W. (67) asked while responding to the Occupation item: “What is the difference between the first two response options? This means the same to me”.
On two other items, some respondents disagreed with the order of the response options. Three respondents, Mr. U (75), Mr. W. (67) and Mrs. Y. (74), did not perceive the first response of the ‘Food and Drink’ item [‘I get all the food and drink that I want’] as the most ideal option, as they considered that to be an unhealthy or compulsive situation.
Respondents also indicated that for two items only half of the composite response option fitted their situation. For instance, Mrs. Z. (71) chose the last response option on the Social participation item [‘I have little social contact with people and feel socially isolated’] although she indicated that she had little contact with other people, but did not feel socially isolated.
Positive answering
Positive answering occurred among various respondents on five items. For example, while reading the ‘Social participation’ item, Mr. Q. (100) told that he finds it very difficult that all his friends are deceased and that there is hardly anyone of his age left. Only sporadically some of his daughter’s acquaintances from church make a visit to their home. His daughter (67) confirmed that her father misses social contacts. Nonetheless, they both dismissed the two most negative response options (describing a situation with some/little social contact) because according to them these options did not fit.
Overall comparison of the measures
Completing each interview, respondents compared the three measures based on coverage of QoL aspects (indicator of content validity) and comprehensibility of the items (indicator of feasibility).
Coverage of QoL aspects
The respondents indicated that almost all questions of the three measures were important, and often preferred the measure which content most closely reflected their situation and daily life issues. For example, Mr. U. (74) preferred the ICECAP-O, as he spends much time reflecting on philosophical questions, and the domains covered by the ICECAP-O “promote living with your heart”. Mrs. Z. (71) on the other hand struggled with many physical problems and being downhearted at times, and preferred the EQ-5D-3L which health-related questions were relevant to her daily life. Mrs. Y. (74), who has no major physical concerns herself but takes care of her husband with Alzheimer, preferred the ICECAP-O and ASCOT above the EQ-5D-3L: “These cover a somewhat broader field”. As most of the respondents did have some health issues, the EQ-5D-3L domains were in general considered relevant, although the number of three response options gave a more “rough indication” than the other two measures with four response options.
While the respondents appreciated the domains of the ICECAP-O and ASCOT as they covered ‘valuable topics’, not all ICECAP-O and ASCOT domains were considered relevant. Mr. W. (67) was one of the respondents who expressed not to think about the future, therefore for him the question about ‘Security’ of the ICECAP-O was not important.
A common opinion of the respondents was that the measures did not result in a comprehensive picture of their QoL. According to the older adults, only a proper personal conversation would convey the relevant topics and details of their lives. Nonetheless, the respondents mentioned only one topic important for their QoL that was not covered by one of the measures; the concerns or delight about the wellbeing of family members, especially of (grand) children. Often the family is a source of happiness or reason for worries, and apparently the respondents did not feel that this impact on their QoL was sufficiently covered by the domains ‘Attachment’ in the ICECAP-O or ‘Social participation’ in the ASCOT.
Comprehensibility of the items
Most consistently, the respondents indicated the EQ-5D-3L was most easy to answer, as the questions were most specific and only three response options were available, making them more distinguishable. Mr. O (76), for example, said about the ICECAP-O: “This questionnaire is very open, all directions are possible, that is difficult for me. I think the questions should be more specific, I find this a bit too vague”. Overall, the measure that was most relevant to the personal situation of the respondent was also the one perceived as most easy to answer.
Discussion
This paper aimed to compare the validity and the feasibility of Dutch translations of the EQ-5D-3L, ICECAP-O and ASCOT from the perspective of older adults by identifying response issues and comparisons of coverage and comprehensibility of domains. The respondents in this study stated that their responses to the measures did not give such a comprehensive picture of their QoL as a proper personal conversation would, and specifically mentioned one domain missing in all three measures: the well-being of their family. Of the three measures, the older adults preferred, both in terms of coverage and comprehensibility of the domains, the measure that most closely reflects their daily life. Since many older adults face at least some health problems, the EQ-5D-3L is likely to be relevant to a large part of this population. Because the items in the EQ-5D-3L corresponded to the reality of their daily life, the EQ-5D-3L was also the measure most easy to complete. However, just like in other qualitative studies [
2,
33] it was recognized that the ICECAP-O and ASCOT have a broader scope and include valuable topics.
We identified response issues for all items of the three measures, related to mapping of responses, interpretation of items or positive responding. These issues are likely to occur when the EQ-5D-3L, ICECAP-O or ASCOT are administered in populations of older adults, and pose potential threats to the validity and feasibility of the measures.
Mapping issues resulted in some extent to misclassification, but we expect that the effect on the validity of the responses is not large, as respondents in general chose a response alternative that most closely reflected their situation. The least amount of mapping issues were identified for the EQ-5D-3L, probably because respondents perceived the response options of the EQ-5D-3L as clearly distinguishable. The ICECAP-O and ASCOT, as well as a new five level version of the EQ-5D [
46], include more response options, intended to increase the sensitivity of the measures. The results of our study suggest that more response options may be accompanied with an increase in occurrence of mapping issues.
Nonetheless, we expect that interpretation issues and positive responding pose a larger threat to the validity and feasibility of the measures. Where the EQ-5D-3L questions were interpreted most of the time as intended, the ICECAP-O and ASCOT both included questions that were poorly or wrongly understood, most apparent the ‘Role’ item of the ICECAP-O and the ‘Dignity’ items of the ASCOT. Our respondents chose to skip these items or just picked one of the answers without fully understanding what it meant. Moreover, all three measures include two or three items that were considered ambiguous, either due to double-barrelled questions or broad concepts that cover more than one domain in life. Vagueness and ambiguity can lead respondents to interpret items in variable ways [
17]. This difference in interpretation would become even more troublesome when it occurs within the same person, for example when measuring QoL
changes in economic evaluations. The impact of this response issue on the validity of responses warrants further research, especially on how it affects the measurement of changes.
Positive answering seems to appear regardless of the item or measure. This is not surprising, as older persons are notorious for giving ‘rosy’ reports of themselves and their living situation, especially to global questions [
26]. These rosy reports may reflect the results of adaptation and downward comparison processes or other self-presentation and coping mechanisms [
24‐
26,
28,
47]. Positive responding due to these mechanisms is not necessarily considered misclassification bias, for example when it reflects a real adaptation in one’s self-evaluation of QoL [
47], but is a threat to validity when it occurs due to being reluctant to come across as overly negative or critical. It is important to investigate further how positive responding affects the measurement of changes.
Adaptation issues and struggles with abstract items were also found in a study that used a think-aloud approach to assess the ICECAP-A [
32]. Compared to our study, a pilot study of a previous version of the ASCOT identified less response issues [
7], while in another study some additional concerns about the ICECAP-A and EQ-5D-3L were expressed by research professionals [
33].
Our study was the first that used a qualitative approach to compare the content validity and feasibility of the EQ-5D-3L, ICECAP-O and ASCOT in older adults. A strength of this study is that the perspective of older adults themselves was central in the assessment of the measures. Furthermore, the same methodology was used to evaluate all three measures, facilitating a comparison.
The findings of our study should be interpreted while taking the following limitations into account. First, our sampling strategy based on the maximum variation principle was aimed at identifying as many different issues as possible and limits inferences about how common the issues are in a wider population [
18]. Furthermore, our sample size was quite small. The frequency of issues in Table
3 should therefore not be generalized. Although the identification of other issues in other older adults or situations cannot be excluded, we are rather confident about the comprehensiveness of the identified issues, since the number of new identified issues decreased considerably per interview. However, the sample size was too small to identify patterns between response issues and characteristics of older adults. It is likely that response issues occur more frequently among older adults with more communication difficulties, illiteracy, cognitive disorders or among those with a higher need to please others. Another limitation of the study is that the think-aloud process and the presence of the interviewer may have affected the way in which respondents answer questions, specifically the amount of attention paid to questions and the occurrence of positive responding [
22,
48‐
50]. Finally, some of the identified issues may have occurred due to the translation of the measures into Dutch. Nonetheless, the translations were developed carefully with the intention of conceptual equivalence and entailed forward and backwards translations, as recommended.
Conclusions
To conclude, response issues are likely to occur when the EQ-5D-3L, ICECAP-O and ASCOT are used for economic evaluations in older adults. Researchers should be aware of these issues and aim to minimize their occurrence, for example by using face-to-face interviews in studies with older adults, in which interviewers have the availability to explain items more extensively. Several response issues warrant further research, especially on the way these issues affect the measurement of changes in older adults. None of the instruments provided a comprehensive picture of the QoL of older adults in itself, but our respondents preferred the measure that most closely reflected the reality of their daily life. For now, researchers that use these measures in older adults should be aware of the response issues that may occur, when interpreting the outcomes of their study.
Competing interests
The authors declare that they have no competing interests.
Authors’ contributions
KvL conducted the interviews, coded the data and drafted the manuscript. initially coded the data. MM, AJ and MW reviewed the data analysis. MM, AJ, MW and JB were involved in drafting the manuscript and interpretation of the results. All authors participated in the design of the study, contributed to revisions of the manuscript and read and approved the final manuscript.