Distribution- and anchor-based methods to determine the minimally important difference on patient-reported outcome questionnaires in oncology: a structured review

The use of patient-reported outcomes (PRO), including health-related quality of life (HRQOL), in cancer clinical trials has substantially increased over the years [1]. PROs are critical to fully understand overall treatment effectiveness and to establish the benefit of a given experimental drug over the standard of care in a particular cancer population [2, 3]. Thus, assessment and analysis of PRO data must be carried out in compliance with a rigorous and appropriate methodology to ensure robust interpretation of the results [4].

The interpretation of PRO scores and their clinical importance is a major challenge, in terms of both clinically relevant score differences between two measurement times and two treatment arms [5]. A statistically significant result may not be clinically relevant, as it should also reflect changes or differences that are meaningful for the patient, i.e., they should take into account a minimally important difference (MID). The MID was defined by Jaeschke et al. as “the smallest change in an outcome that a patient would identify as important” [6]. Hence, the determination of the MID is crucial in order to assess the effectiveness of health care interventions, as well as for sample size calculation when HRQOL is the primary or co-primary endpoint in clinical trials.

Different methods have been proposed to determine the MID. These methods are generally grouped into two categories, namely anchor-based and distribution-based approaches [7, 8]. The anchor-based approaches use an external indicator, called an “anchor”, and differences can be determined either cross-sectionally (differences between clinically-defined groups at one time point) or longitudinally (change in the scores of a single group over time). The anchor can be either an objective measure (e.g., Karnofsky or ECOG performance status) or a subjective measure, generally reflecting the patient’s point of view, which is of interest (for example, the patient rating of change). Distribution-based approaches are based on statistical criteria from the PRO scores. These approaches include fractions of the standard deviation (SD) of PRO scores, the effect size [9], and the standard error of measurement (SEM) [10] as estimates for the MID. Distribution-based approaches have the advantage of simplicity of use, since they do not require an external criterion. However, they produce similar MID results for both deterioration and improvement. This simplifies the interpretation but may be questionable, since a larger MID is often observed for deterioration than for improvement [11].

Some recommendations have been proposed regarding the best method to apply, depending on the design of the study. For instance, analysis must rely primarily on relevant patient-based and clinical anchors [12]. Moreover, both distribution- and anchor-based approaches remain the most commonly used methods to determine the MID [13]. However, robust and reliable determination of the MID remain challenging. In fact, due to the longitudinal design often used in MID analyses, a potential response shift effect may bias the results. The impact of the response shift effect on the longitudinal analysis of PRO is well established and has been widely studied [14]. However, studies investigating the impact of response shift effect on MID determination remain sparse [15]. Another important possible limitation of studies aiming to determine the MID is the sample size. Indeed, most studies aiming to explore the MID on a given PRO questionnaire use data from an existing cohort or randomized clinical trial. Thus, the volume of available data may not be sufficient to provide a reliable MID, in particular due to the number of possible categories for the anchor.

To date, longitudinal studies in oncology generally used the thresholds proposed by Osoba et al. in 1998 [5] for HRQOL for the interpretation of results, i.e. an MID of 5 or 10 points. However, these thresholds were obtained only on data collected with the European Organisation for Research and Treatment of Cancer (EORTC) QLQ-C30 cancer-specific questionnaire. A more recent meta-analysis proposed specific thresholds for each HRQOL scale of the EORTC QLQ-C30, and for each direction, i.e. improvement or deterioration [16]. Other studies proposed MID for specific cancer site questionnaires, such as the EORTC QLQ-BN20 for brain cancer [17], but few studies use these specific thresholds to interpret HRQOL results.

In this context, the objective of this structured review was to assess the most common practices used by the distribution and anchor-based approaches to determine the MID for PRO questionnaires in oncology, as well as the characteristics and the possible limitations relative to each approach.

The initial search identified a total of 64 studies (Fig. 1). After screening of the title and abstract by the 2 reviewers, 15 studies were excluded because they were not relevant to the subject (N = 9), were not original papers (N = 3) or reported computer adaptive testing (N = 3). After reading the full text of the remaining 49 articles, three additional articles were excluded as they explored cut-off scores and not MID. Thus, a total of 46 studies were finally included in this review [13, 17, 19‐62].

The objective of this structured review was to assess the most common practices used by the distribution and anchor based approaches to determine the MID for PRO questionnaires in oncology, and to present the variability of criteria used as well as possible limitations relative to each approach. We limited our research to year 2000 because we think a review of papers published since almost two decades would be reasonable and enough to conduct this review. Eligible studies included original articles aiming to determine the MID of self-administered questionnaires in cancer, using distribution- and/or anchor based approaches.

Using both the distribution and anchor-based approaches, as was the case in the majority of studies (80%), makes it possible to compare results for consistency, to highlight the strengths and weaknesses of each method, and to retain the most appropriate MID value or range to apply in further studies [12].

For the distribution-based approach, several criteria were reported at different assessment times. As already highlighted in previous reviews [63], the most frequently reported criterion was 0.5 SD at baseline, reported in 90% of studies using the distribution-based method. Despite the simplicity and the widespread use of this approach in the determination of the MID, no distinction can be made between improvement and deterioration.

Regarding the anchor-based approach, most studies used a longitudinal design (92.9%). Various anchors were applied, and were either patient- or physician-reported measures, as well as clinical or biological measures with clinical relevance. The most commonly used anchor was a PRO score or item, derived from the questionnaire of interest or from another questionnaire (41.9%). When this anchor was derived from the questionnaire for which the MID was being determined, then the MID on the corresponding dimension could not be assessed. This is the case for example when the overall HRQOL score is used as an anchor to estimate the MID on the QLQ-C30. Moreover, this requires to fix a threshold to qualify the clinically meaningful change on one dimension of the studied questionnaire. Finally, the choice of this kind of anchor could be questionable since the property of external criteria for the anchor is not entirely respected.

Another frequently used patient-centered anchor (used in 20.9% of the studies using the anchor-based approach) is the patient’s overall rating of change. This anchor reflects the patient’s perception of change, but it needs to be planned in the design of the study.

A large proportion of studies also used physician-reported measures such as performance status or MMSE score. These anchors could be considered as objective evaluations of the patient’s health status. However, they may not be appropriate for the assessment of the MID on all HRQOL dimensions. Given that performance status reflects the physical condition of the patient, it is generally correlated to the physical dimensions of HRQOL. Similarly, the MMSE is mostly correlated with the cognitive dimension of HRQOL. Thus, these anchors preclude assessment of the MID for more physiological or emotional dimensions of HRQOL [17]. This also means that several anchors are needed to accurately assess the MID and to check the robustness and complementarity of the results obtained using different anchors [12]. In our review, half the studies (53.5%) using an anchor-based approach used only one anchor. For the majority of the studies, the correlation between the anchor and scores of the studied questionnaire has been checked. The threshold of 0.3 was the most used criteria to detect a moderate correlation. On the other side, an important frequency of studies (46.9%) from those who checked correlation did not reported the criteria to identify a moderate correlation. Generally, checking correlation is important to know to what extent the anchor used is linked to HRQOL measure. Hence, the correlation between the anchor and the PRO scores must be assessed and only dimensions that are significantly correlated with the anchor (correlation coefficient |r| > 0.3) should be analyzed. In this review, 76.2% of the studies using an anchor-based approach verified this correlation.

The majority of the studies with a longitudinal design (58.7%) used a time interval between 1 and 6 months between two consecutive assessments. However, wide variation was observed between studies. A standard period remains to be determined, and further research is warranted to determine a suitable time window. This point is particularly important when the patient’s overall rating of change is used as an anchor, since long periods between assessments could induce a recall bias.

The majority of studies (71.7%) used one or two measurement times to determine the MID, but the stability over time was rarely investigated. For example, a change in PRO score of 5 points could be insignificant for the patients at the time of diagnosis, whereas it might be highly relevant after surgery. Therefore, it is strongly recommended to assess the MID with more than two measurement times [62]. This change in the patients’ perception of HRQOL change over time could reflect the occurrence of a response shift effect [64]. In our review, only one study investigated the impact of the occurrence of response shift on the MID determination using the patient’s overall rating of change as the anchor. However, the response shift effect could differentially impact on the results of the MID depending on the anchor used. Future studies are warranted to investigate this possible risk of bias and take it into account in the MID determination [15].

Several terminologies were used to identify the MID. However, using a standardized term and acronym referring to the MID should be investigated in future studies to avoid variability in terminology and to obtain more accurately the maximum number of articles needed for future analysis.

A frequent limitation was the small sample size in the anchor categories. If the number of patients in each category of anchor is not sufficient, then the resulting MID cannot be reliable and the robustness in this case is thus questionable. Only one study determined the sample size specifically for the MID analysis. This sample size calculation must be systematically performed for MID determination, even if the analysis is not the primary objective of the study, for example when data from a randomized clinical trial are used. Calculating an appropriate sample size per category will ensure the robustness of the results. Furthermore, the occurrence of missing data can also bias the MID analysis. Therefore, it is also important to determine the profile of missing data, and consider imputing missing data using the appropriate method.

Using only one electronic database (PubMed searches) was the main limitation of this work. Unfortunately, due to lack of resource, we could not use other databases to perform this review. A risk of bias could thus be observed since other interesting papers may not be captured in this database. Hopefully, a manual research conducted to the same papers obtained via our algorithm in Pubmed.

This review must be expanded in future studies to address all methods that have been used to determine the MID either if they are including or not including in distribution-based or anchor-based methods (i.e. minimal detectable change, Receiver operating characteristic (ROC) curve, Item Response Theory, etc.).

In light of these results, greater attention should be paid to the methodology in future studies investigating the MID of a given PRO questionnaire, in order to ensure reliable results. This will also make it possible to use the MID for sample size determination when designing clinical trials with HRQOL or PRO as a primary endpoint, as well as for facilitating interpretation of the results. In the context of clinical trials in oncology, the MID is rarely used to interpret results in a clinically meaningful way. In a recent review of phase III trials in non-small cell lung cancer including a PRO endpoint, only 20% of studies interpreted the results in light of the MID [65]. The time-to-HRQOL-deterioration is a recently proposed method to analyze longitudinal HRQOL data [66]. One advantage of this method is to incorporate the MID in the definition of the event to qualify the deterioration. This guarantees the clinical significance of the results, but the choice of the MID is crucial since it has a direct impact on the results of the analysis.

Further research is mandatory to improve the quality of the methodology used to determine the MID in HRQOL questionnaires used in oncology. In particular, the choice of an appropriate anchor(s) when using the anchor-based approach, or appropriate criteria when using the distribution-based approach is essential. The sample size should also be taken into account to produce reliable results. This could increase the use of these specific thresholds in future studies.