“When I saw walking I just kind of took it as wheeling”: interpretations of mobility-related items in generic, preference-based health state instruments in the context of spinal cord injury

Generic, preference-based health state instruments play an important role in informing priorities regarding available treatment options [1]. Preference-based health state instruments consist of two components: a ‘descriptive system’ that asks respondents to describe their health by responding to questions in a standardized questionnaire (resulting in health state descriptions) and a ‘valuation system’ that provides preference weights (also known as index scores) that represent the values that respondents, usually from the general population, attach to living in each health state (health state values).

Instruments used to measure health-related quality of life (HRQoL) can be classified as either generic or specific to a particular disease, condition, or population otherwise defined (e.g., older adults). Disease-specific instruments are more sensitive to small changes in HRQoL for the target group because they focus on aspects that are known to be relevant to that group. In contrast, generic instruments are intended to be applicable to a wide range of conditions [2]. Information gained by using generic and disease-specific instruments is valuable when conducting economic evaluation to inform decision processes concerning competing demands for scarce health care resources [1, 3‐6]. However, only generic instruments allow for comparisons of interventions across different diseases and patient groups.

Instruments used to measure preference-based HRQoL are often referred to as utility instruments, health state utility instruments, preference-based HRQoL instruments or health state instruments. The variety of terms used suggests an underlying lack of conceptual clarity of the definition and theory of ‘quality of life’ [7]. In this study, generic preference-based health state instruments are referred to as ‘health state instruments’. The items and dimensions contained within health state instruments differ in their conceptual coverage of physical, social and mental health. Throughout this paper, ‘item’ is referred to as “a linguistic statement generally consisting of a stem plus a number of ordered response levels” [8], whereas ‘dimension’ is used to describe the conceptual idea that is represented by one or several items, e.g., ‘mobility’ or ‘usual activities’ [8].

What most health state instruments have in common is that a small number of items (often only one) represent each dimension. This is in contrast to classic measurement theory in psychometrics, which recommends measuring each dimension by several items in order to improve reliability and validity [9]. Asking respondents to describe their health by choosing response options in a questionnaire is considerably different from measuring physical parameters like a person’s body temperature. While a clinical thermometer directly transforms body temperature to a displayed number, the application of a questionnaire includes at least two phases of interpretation. Firstly, the respondent needs to interpret the questions in the questionnaire in order to choose a response option. Secondly, the recipient of the data needs to interpret the results. Knowing how respondents interpret items is crucial to ensure that an instrument measures what it is supposed to measure [10]. Diverging interpretations can occur if (i) respondents interpret items in a way that diverges from the interpretation originally intended by the instrument developer, or (ii) if the item interpretation by a subgroup of respondents (e.g., wheelchair users) diverges systematically from interpretations made by the individuals outside of that subgroup. Either situation is problematic for any questionnaire that is used to capture health status (or, more broadly, quality of life) because the validity of the data is compromised. There is a sizeable literature on variation of health state values between different groups (e.g., age groups, educational groups, countries, and patients vs. general population) [11, 12], and variation in health state values observed when the same individuals describe their health using different health state instruments [8, 13‐16]. Systematic differences in item interpretation across subgroups may well be a factor in explaining observed variation in health state descriptions and values.

Although the instruments used to measure HRQoL are intended for quantitative use, qualitative research is a highly appropriate means of exploring what is driving the differences in health state descriptions and values [17, 18]. As health state instruments are often used to capture impairments, it is problematic if groups that have significant levels of impairment cannot adequately describe their health state. A focus group study by Whitehurst and colleagues found that individuals living with spinal cord injury (SCI) is a subgroup that faces challenges in describing their health state [19]. These findings supported other studies–outside the context of preference-based instruments–that suggest mobility-related items in standardized, generic HRQoL questionnaires pose challenges for individuals living with SCI [20‐22]. Item interpretation has been widely studied in the field of psychometrics and the need for respondents to interpret an item before they can choose an appropriate response option is a well-established phase within theoretical models that describe the response process [11, 23‐25]. However, there have been relatively few studies focusing on health state instruments [10, 26]. The presence of diverging item interpretations and its impact on the validity of health state descriptions and values is an under-researched empirical question in health economics. The aim of this study is to further explore and describe difficulties faced by respondents living with SCI when answering mobility-related items included in health state instruments.

Fifteen individuals participated in three focus groups (five per focus group). Four themes emerged from the analysis (see Table 2): (i) wording of mobility, (ii) reference to aids and appliances, (iii) lack of suitable response options, and (iv) reframing of items. The first three themes describe item features that cause difficulties when respondents attempt to describe their health state; the fourth theme summarizes comments about a response strategy respondents proposed when facing inaccessible items.

Results are presented by theme rather than by instrument because the primary interest of this study is to describe and highlight participants’ general perceptions to mobility-related items (rather than draw comparisons across instruments). Additional information is reported in Appendix 2, which provides a systematized overview of the mobility-related content from the focus group transcripts (i.e., the themes of the final coding framework and the respective mobility-related quotes). Focus group abbreviations (i.e., FG1, FG2 and FG3) and instrument names indicate the respective instrument and focus group transcript for each quote. For example, ‘(FG2, 15D)’ indicates that the quote is in reference to the 15D, made during the second focus group.

This study explored diverging item interpretations in the descriptive systems of health state instruments, using the example of individuals with SCI and mobility-related items. Study participants consistently reported difficulties in finding relevant response options for mobility-related items. Consequently, some participants interpreted items differently from the originally intended meaning. Although this allowed them to describe their level of mobility ‘correctly’, from their own perspective, such diverging item interpretations pose a direct threat to the validity of health state instruments.

Narrow wording was a central topic in the focus group discussions. Previous findings have questioned the suitability of the word ‘walking’ for the SCI population, although not within the context of health state instruments. For example, the word ‘walking’ has been considered unacceptable and even perceived as offensive and inappropriate in an investigation of the SF-36 and the Quality of Well-Being scale in an SCI population [21]. Similarly, respondents in a qualitative study reported problems in interpreting the physical functioning items of the SF-36 due to narrow wording [10]. Dijkers concluded that the wording used to capture physical functioning in the SF-36v2 is suitable only if the aim is to capture impairment – but not if the level of functioning or participation is to be measured [22]. Despite the important role of wording for item interpretation (and, therefore, for the validity of health state instruments), it has so far received little attention in the health economics literature.

A number of study participants proposed reframing as a response strategy to provide a more ‘correct’ description of their level of mobility, i.e., choosing to replace “walking” with “wheeling”. Within the context of health state instruments, this means that the value assigned to a particular reframed health state description is incorrect, because participants in the respective valuation study did not value the reframed health state. This is one example of how reframing can pose a threat to the validity of health state instruments. A related issue is that the recipient of the data is unlikely to know if reframing took place, and for whom it took place [10]. When making use of data from questionnaires, a tacit assumption is that the respondents are relatively homogeneous in their item interpretations, or at least that any variance in item interpretation can be handled as normally distributed measurement error. This assumption allows the recipient of the data to disregard respondent variation in interpretation of the scales since it is assumed that the errors will cancel out, on average, given a large sample size. However, if specific groups of respondents systematically deviate in their interpretation of the scales, the results for these groups will be biased, regardless of the sample size. Examining mobility within the context of SCI has highlighted that reframing is problematic and it is highly likely that reframing occurs in other patient groups, e.g., individuals with motor neurone disease [34], and for items/dimensions other than mobility.

A second recurring issue in the focus groups was the way aids and appliances are referred to in health state instruments. Some participants wanted the wheelchair to be mentioned explicitly, while others would have liked to have instructions on if and how aids should be taken into account. These reactions are in line with findings from a study involving patients with motor neurone disease responding to the 3-level EQ-5D (EQ-5D-3L) [34]. Individuals using aids and appliances (e.g., hearing aids, eyeglasses and wheelchairs) or medications (e.g., painkillers) that influence the symptoms and/or functionings described in the descriptive systems could face comparable challenges if no explicit instruction to aid interpretation is provided; as shown in Table 1, there is no consensus among current instruments. Asada developed a classification that differentiates between medical technologies (e.g., medication), nonhuman aids (e.g., wheelchair), human assistance (e.g., help of another person when walking) and accommodating environmental factors (e.g., a barrier-free physical environment), and suggests to include medical technologies and nonhuman aids only in the assessment of health [35]. To some extent, this position is reflected in the current state of play because none of the health state instruments included in this study (15D, AQoL-8D, EQ-5D-5L, HUI, QWB-SA or SF-6D) refer to environmental factors, such as accessibility and the availability of rehabilitation.

Modifying items is one approach to try to make items more accessible to a broader range of clinical contexts. There have been several attempts in the literature to modify items of non-preference-based HRQoL instruments, primarily focusing on the physical functioning dimension of the SF-36 [21, 36, 37]. However, modifying health state instruments is problematic for several reasons. Firstly, comparisons across different patient groups are no longer valid if an instrument includes modified items or instructions for a particular subgroup of respondents. Secondly, psychometric evidence for the original instrument cannot be considered valid evidence for the modified version of the instrument. Thirdly, as alluded to above, health state values representing preferences for health states of the original descriptive system should not be applied to health state descriptions from the modified descriptive system.

In recent years there has been a call for more patient involvement in instrument development, as most health state instruments were developed with little input from individuals representative of the intended respondents [10, 38, 39]. The current study contributes to this call by exploring variation in item interpretation from the perspective of individuals living with SCI. An example of the value of such an approach was the participants’ suggestion of a broader conceptualization of mobility, one that is not solely restricted to an individual’s ability to walk. Ideally, exploring item interpretations would have been part of the development process of the respective descriptive systems. The findings of this study have implications beyond the measurement of preference-based health status because mobility-related items are ubiquitous in generic HRQoL instruments. More specifically, the findings are valuable to researchers during the process of selecting a suitable instrument, when reevaluating existing instruments, or when developing new instruments. The findings could also help explain how variations between the descriptive systems of different preference-based instruments may contribute to poor agreement between their derived health state values [19, 40].

Further qualitative research in this area should explore whether similar inferences (accessibility, reframing, etc.) can be drawn for items and contexts beyond mobility and SCI. As a case in point, a number of studies have shown that individuals with mental health conditions have difficulties describing their health [41‐45]. Secondly, important conceptual and normative questions should be revisited in order to judge the relative merits of different health state instruments. For example, with the EQ-5D-5L becoming established as a widely-used health state instrument, should it include explicit instruction as to whether respondents should be considering medical technologies, nonhuman aids and/or human assistance when completing the instrument; and what are the implications of maintaining the status quo?

This study identified four themes that provide insight into the issues individuals living with significant mobility impairment may face when responding to mobility-related items in health state instruments. The observed problems–such as the lack of suitable response options and the reframing of items–pose considerable threats to the validity of health state descriptions, health state values and, ultimately, cost-effectiveness estimates in the evaluation of treatments and interventions. The extent to which other respondent groups experience similar difficulties in interpreting items, whether mobility-related or otherwise, remains unknown.