Original ArticleStructural equation modeling–based effect-size indices were used to evaluate and interpret the impact of response shift effects
Introduction
Patient-reported outcomes (PROs) have become increasingly important, both in clinical research and practice. PROs may include measures of subjective well-being, functional status, symptoms, or health-related quality of life (HRQL). The patient perspective on health provides insight into the effects of treatment and disease that is imperative for understanding health outcomes. PROs thus present important measures for evaluating the effectiveness of treatments and changes in disease trajectory, especially in chronic disease [1] and palliative care [2].
The investigation and interpretation of change in PROs can be hampered because different types of change may occur. Differences in the scores of PROs are usually taken to indicate change in the construct that the PROs aim to measure. However, these differences can also occur because patients change the meaning of their self-evaluation. Sprangers and Schwartz [3] proposed a theoretical model for change in the meaning of self-evaluations, referred to as a response shift. They distinguish three different types of response shift: recalibration refers to a change in respondents' internal standards of measurements; reprioritization refers to a change in respondents' values regarding the relative importance of subdomains; and reconceptualization refers to a change in the meaning of the target construct. To illustrate, when a patient is being asked to fill in a questionnaire about quality of life, he or she may indicate to be limited in social functioning (SF) very often before treatment and some of the time after treatment. The change in these responses can be interpreted as a reduction in SF and indicative of a reduction in quality of life. However, the observed change may also occur because the patient has recalibrated what very often means, for example, the response very often may refer to many more times after treatment than it did before treatment. With reprioritization response shift, the observed change may occur because the relative importance of SF to the patient's quality of life increased. Finally, with reconceptualization response shift, the meaning of a patient's response may have changed, for example, a patient may interpret SF as work related before treatment and as family related after treatment.
As the occurrence of response shift may impact the assessment of change, the detection of possible response shift effects is important for the interpretation of change in PROs. One of the methods that can be used to investigate the occurrence of response shift is Oort's structural equation modeling (SEM) approach [4]. Advantages of the SEM approach are that it enables the operationalization and detection of the different types of response shift and that it can be used to investigate change in the construct of interest (e.g., HRQL) while taking possible response shifts into account. We note, however, that SEM is a group level analysis and will only detect response shifts that affect a substantial part of the sample.
Although clinicians and researchers acknowledge the occurrence of response shift, little is known about the magnitude and clinical significance of those effects [5]. The detection of response shift is usually guided by tests of statistical significance. Although statistical tests can be used to determine whether occurrences of response shift are statistically significant, they cannot be taken to imply that the result is also clinically significant (i.e., meaningful). Statistical significance tests protect us from interpreting effects as being real when they could in fact result from random error fluctuations. However, statistical significance tests do not protect us from interpreting small but trivial effects as being meaningful. Therefore, assessing the meaningfulness of change in PROs has been an important research focus [6], [7] as it is imperative for translating results to patients, clinicians, or health practitioners. However, there is no universally accepted approach to determine the meaningfulness of change in PROs [8].
One of the approaches that can be used to determine the clinical significance of change in PROs is to calculate distribution-based effect-size indices. Distribution-based effect sizes are calculated by comparing the change in outcome to a measure of the variability (e.g., a standard deviation). The resulting effect sizes are thus standardized measures of the relative size of effects. They facilitate comparison of effects from different studies, particularly when outcomes are measured on unfamiliar or arbitrary scales [9]. In addition, previous research has shown that distribution-based indices often lead to similar conclusions as when the clinical significance of effects is directly linked to patients' or clinicians' perspectives on the importance of change, that is, the so called anchor-based indices of effects [10], [11], [12]. Furthermore, the interpretation of effect-size indices as indicating small, medium, or large effects is possible using general rules of thumb (e.g., Ref. [13]). Therefore, distribution-based effect-size indices can be used to convey information about the clinical meaningfulness of results.
The aim of this article is to explain the calculation of effect-size indices within the SEM framework for the investigation and interpretation of change. In addition, we explain how this enables the evaluation and interpretation of the impact of response shift on the assessment of change. Specifically, we use SEM to decompose observed change into change because of response shift and change because of the construct of interest (i.e., true change). Subsequently, we illustrate the calculation and interpretation of various effect-size indices, that is, the standardized mean difference (SMD), standardized response mean (SRM), probability of benefit (PB), probability of net benefit (PNB), and number needed to treat to benefit (NNTB), for each component of the decomposition. This enables the evaluation of the contributions of response shift and true change to the overall assessment of change in the observed variables. To illustrate, we will use SEM to investigate change in data from 170 cancer patients, whose HRQL was assessed before surgery and 3 months after surgery. We aim to show that distribution-based effect-size indices can contribute to the clinical interpretability of change in PROs.
Section snippets
Calculation of effect-size indices of change
Later we explain the calculation and interpretation of different effect-size indices of change using pretest and post-test comparison as an example. A more detailed explanation of the (statistical) derivations of these effect-size indices and their inter-relationships is offered in an online Technical Appendix (see on the journal's Web site at www.elsevier.com).
Illustrative example
To illustrate the calculation and interpretation of effect-size indices of change, we used HRQL data from 170 newly diagnosed cancer patients. Patients' HRQL was assessed before surgery (pretest) and 3 months after surgery (post-test). The sample included 29 lung cancer patients undergoing either lobectomy or pneumectomy, 43 pancreatic cancer patients undergoing pylorus-preserving pancreaticoduodenectomy, 46 esophageal cancer patients undergoing either transhiatal or transthoracic resection,
Discussion
In this article, we have shown how to calculate effect-size indices of change using SEM. We used SEM for the decomposition of change, where observed change (e.g., change in the subscales of a HRQL questionnaire) is decomposed into change because of recalibration, reprioritization and reconceptualization, and true change in the underlying construct (e.g., HRQL). Calculation of effect-size indices for each of the different elements of the decomposition enables the evaluation and interpretation of
Acknowledgments
This research was supported by the Dutch Cancer Society (KWF grant 2011-4985). F.J. Oort and M.G.E. Verdam participated in the Research Priority Area Yield of the University of Amsterdam. We thank M.R.M. Visser from the Academic Medical Center of the University of Amsterdam for making the data that was used in this study available for secondary analysis.
References (34)
- et al.
Recommended methods for determining responsiveness and minimally important differences for patient-reported outcomes
J Clin Epidemiol
(2008) - et al.
Integrating response shift into health-related quality of life research: a theoretical model
Soc Sci Med
(1999) - et al.
Clinical significance of patient-reported data: another step toward consensus
J Clin Epidemiol
(2005) - et al.
What is clinically meaningful change on the Functional Assessment of Cancer Therapy-Lung (FACT-L) Questionnaire? Results from Eastern Cooperative Oncology Group (ECOG) Study 5592
J Clin Epidemiol
(2002) - et al.
A combination of distribution- and anchor-based approaches determined minimally important differences (MIDs) for four endpoints in a breast cancer scale
J Clin Epidemiol
(2004) - et al.
Understanding the minimum clinically important difference: a review of concepts and methods
Spine J
(2007) - et al.
Size of treatment effects and their importance to clinical research and practice
Biol Psychiatry
(2006) - et al.
The Multidimensional Fatigue Inventory (MFI): psychometric qualities of an instrument to assess fatigue
J Psychosom Res
(1995) - et al.
Methods to explain the clinical significance of health status measures
Mayo Clinic Proc
(2002) Differences in what quality-of-life instruments measure
J Natl Cancer Inst Monogr
(2007)
Using structural equation modeling to detect response shifts and true change
Qual Life Res
The clinical significance of adaptation to changing health: a meta-analysis of response shift
Qual Life Res
Interpretation of patient-reported outcomes
Stat Methods Med Res
Estimating clinically significant differences in quality of life outcomes
Qual Life Res
Comparison of distribution- and anchor-based approaches to infer change in health-related quality of life of prostate cancer survivors
Health Res Educ Trust
Statistical power analysis for the behavioral sciences
Cited by (19)
Validation of the Australian Pelvic Floor Questionnaire in Chinese pregnant and postpartum women
2020, European Journal of Obstetrics and Gynecology and Reproductive BiologyCitation Excerpt :Longitudinal follow-up of 316 participants was used to assess clinical changes in pelvic floor symptoms associated with pregnancy and childbirth. Effect size (ES, mean change of score/standard deviation of baseline score) and standardized response mean (SRM, mean change of score/standard deviation) [21] were used to demonstrate the degree of responsiveness. Statistical Package for Social Science version 20.0 was applied in this study.
Using structural equation modeling to detect response shift in quality of life in patients with Alzheimer's disease
2019, International PsychogeriatricsResponse shift results of quantitative research using patient-reported outcome measures: a descriptive systematic review
2024, Quality of Life ResearchValidation of online delivery of the Australian Pelvic Floor Questionnaire in an Irish obstetric population
2023, International Urogynecology JournalResponse Shift After Cognitive Behavioral Therapy Targeting Severe Fatigue: Explorative Analysis of Three Randomized Controlled Trials
2023, International Journal of Behavioral Medicine
Conflict of interest: The authors have no conflict of interest to declare.