Original Article
Structural equation modeling–based effect-size indices were used to evaluate and interpret the impact of response shift effects

https://doi.org/10.1016/j.jclinepi.2017.02.012Get rights and content

Abstract

Objectives

The investigation of response shift in patient-reported outcomes (PROs) is important in both clinical practice and research. Insight into the presence and strength of response shift effects is necessary for a valid interpretation of change.

Study Design and Setting

When response shift is investigated through structural equation modeling (SEM), observed change can be decomposed into change because of recalibration response shift, change because of reprioritization and/or reconceptualization response shift, and change because of change in the construct of interest. Subsequently, calculating effect-size indices of change enables evaluation and interpretation of the clinical significance of these different types of change.

Results

Change was investigated in health-related quality of life data from 170 cancer patients, assessed before surgery and 3 months after surgery. Results indicated that patients deteriorated on general physical health and general fitness and improved on general mental health. The decomposition of change showed that the impact of response shift on the assessment of change was small.

Conclusion

SEM can be used to enable the evaluation and interpretation of the impact of response shift effects on the assessment of change, particularly through calculation of effect-size indices. Insight into the occurrence and clinical significance of possible response shift effects will help to better understand changes in PROs.

Introduction

Patient-reported outcomes (PROs) have become increasingly important, both in clinical research and practice. PROs may include measures of subjective well-being, functional status, symptoms, or health-related quality of life (HRQL). The patient perspective on health provides insight into the effects of treatment and disease that is imperative for understanding health outcomes. PROs thus present important measures for evaluating the effectiveness of treatments and changes in disease trajectory, especially in chronic disease [1] and palliative care [2].

The investigation and interpretation of change in PROs can be hampered because different types of change may occur. Differences in the scores of PROs are usually taken to indicate change in the construct that the PROs aim to measure. However, these differences can also occur because patients change the meaning of their self-evaluation. Sprangers and Schwartz [3] proposed a theoretical model for change in the meaning of self-evaluations, referred to as a response shift. They distinguish three different types of response shift: recalibration refers to a change in respondents' internal standards of measurements; reprioritization refers to a change in respondents' values regarding the relative importance of subdomains; and reconceptualization refers to a change in the meaning of the target construct. To illustrate, when a patient is being asked to fill in a questionnaire about quality of life, he or she may indicate to be limited in social functioning (SF) very often before treatment and some of the time after treatment. The change in these responses can be interpreted as a reduction in SF and indicative of a reduction in quality of life. However, the observed change may also occur because the patient has recalibrated what very often means, for example, the response very often may refer to many more times after treatment than it did before treatment. With reprioritization response shift, the observed change may occur because the relative importance of SF to the patient's quality of life increased. Finally, with reconceptualization response shift, the meaning of a patient's response may have changed, for example, a patient may interpret SF as work related before treatment and as family related after treatment.

As the occurrence of response shift may impact the assessment of change, the detection of possible response shift effects is important for the interpretation of change in PROs. One of the methods that can be used to investigate the occurrence of response shift is Oort's structural equation modeling (SEM) approach [4]. Advantages of the SEM approach are that it enables the operationalization and detection of the different types of response shift and that it can be used to investigate change in the construct of interest (e.g., HRQL) while taking possible response shifts into account. We note, however, that SEM is a group level analysis and will only detect response shifts that affect a substantial part of the sample.

Although clinicians and researchers acknowledge the occurrence of response shift, little is known about the magnitude and clinical significance of those effects [5]. The detection of response shift is usually guided by tests of statistical significance. Although statistical tests can be used to determine whether occurrences of response shift are statistically significant, they cannot be taken to imply that the result is also clinically significant (i.e., meaningful). Statistical significance tests protect us from interpreting effects as being real when they could in fact result from random error fluctuations. However, statistical significance tests do not protect us from interpreting small but trivial effects as being meaningful. Therefore, assessing the meaningfulness of change in PROs has been an important research focus [6], [7] as it is imperative for translating results to patients, clinicians, or health practitioners. However, there is no universally accepted approach to determine the meaningfulness of change in PROs [8].

One of the approaches that can be used to determine the clinical significance of change in PROs is to calculate distribution-based effect-size indices. Distribution-based effect sizes are calculated by comparing the change in outcome to a measure of the variability (e.g., a standard deviation). The resulting effect sizes are thus standardized measures of the relative size of effects. They facilitate comparison of effects from different studies, particularly when outcomes are measured on unfamiliar or arbitrary scales [9]. In addition, previous research has shown that distribution-based indices often lead to similar conclusions as when the clinical significance of effects is directly linked to patients' or clinicians' perspectives on the importance of change, that is, the so called anchor-based indices of effects [10], [11], [12]. Furthermore, the interpretation of effect-size indices as indicating small, medium, or large effects is possible using general rules of thumb (e.g., Ref. [13]). Therefore, distribution-based effect-size indices can be used to convey information about the clinical meaningfulness of results.

The aim of this article is to explain the calculation of effect-size indices within the SEM framework for the investigation and interpretation of change. In addition, we explain how this enables the evaluation and interpretation of the impact of response shift on the assessment of change. Specifically, we use SEM to decompose observed change into change because of response shift and change because of the construct of interest (i.e., true change). Subsequently, we illustrate the calculation and interpretation of various effect-size indices, that is, the standardized mean difference (SMD), standardized response mean (SRM), probability of benefit (PB), probability of net benefit (PNB), and number needed to treat to benefit (NNTB), for each component of the decomposition. This enables the evaluation of the contributions of response shift and true change to the overall assessment of change in the observed variables. To illustrate, we will use SEM to investigate change in data from 170 cancer patients, whose HRQL was assessed before surgery and 3 months after surgery. We aim to show that distribution-based effect-size indices can contribute to the clinical interpretability of change in PROs.

Section snippets

Calculation of effect-size indices of change

Later we explain the calculation and interpretation of different effect-size indices of change using pretest and post-test comparison as an example. A more detailed explanation of the (statistical) derivations of these effect-size indices and their inter-relationships is offered in an online Technical Appendix (see on the journal's Web site at www.elsevier.com).

Illustrative example

To illustrate the calculation and interpretation of effect-size indices of change, we used HRQL data from 170 newly diagnosed cancer patients. Patients' HRQL was assessed before surgery (pretest) and 3 months after surgery (post-test). The sample included 29 lung cancer patients undergoing either lobectomy or pneumectomy, 43 pancreatic cancer patients undergoing pylorus-preserving pancreaticoduodenectomy, 46 esophageal cancer patients undergoing either transhiatal or transthoracic resection,

Discussion

In this article, we have shown how to calculate effect-size indices of change using SEM. We used SEM for the decomposition of change, where observed change (e.g., change in the subscales of a HRQL questionnaire) is decomposed into change because of recalibration, reprioritization and reconceptualization, and true change in the underlying construct (e.g., HRQL). Calculation of effect-size indices for each of the different elements of the decomposition enables the evaluation and interpretation of

Acknowledgments

This research was supported by the Dutch Cancer Society (KWF grant 2011-4985). F.J. Oort and M.G.E. Verdam participated in the Research Priority Area Yield of the University of Amsterdam. We thank M.R.M. Visser from the Academic Medical Center of the University of Amsterdam for making the data that was used in this study available for secondary analysis.

References (34)

  • F.J. Oort

    Using structural equation modeling to detect response shifts and true change

    Qual Life Res

    (2005)
  • C.E. Schwartz et al.

    The clinical significance of adaptation to changing health: a meta-analysis of response shift

    Qual Life Res

    (2006)
  • J.C. Cappelleri et al.

    Interpretation of patient-reported outcomes

    Stat Methods Med Res

    (2014)
  • K.W. Wyrwich et al.

    Estimating clinically significant differences in quality of life outcomes

    Qual Life Res

    (2005)
  • Coe, R. (2002). It's the effect size, stupid: what effect size is and why it is important. Paper presented at the...
  • R. Jayadevappa et al.

    Comparison of distribution- and anchor-based approaches to infer change in health-related quality of life of prostate cancer survivors

    Health Res Educ Trust

    (2012)
  • J. Cohen

    Statistical power analysis for the behavioral sciences

    (1988)
  • Cited by (19)

    • Validation of the Australian Pelvic Floor Questionnaire in Chinese pregnant and postpartum women

      2020, European Journal of Obstetrics and Gynecology and Reproductive Biology
      Citation Excerpt :

      Longitudinal follow-up of 316 participants was used to assess clinical changes in pelvic floor symptoms associated with pregnancy and childbirth. Effect size (ES, mean change of score/standard deviation of baseline score) and standardized response mean (SRM, mean change of score/standard deviation) [21] were used to demonstrate the degree of responsiveness. Statistical Package for Social Science version 20.0 was applied in this study.

    View all citing articles on Scopus

    Conflict of interest: The authors have no conflict of interest to declare.

    View full text