Item response theory-based measure of global disability in multiple sclerosis derived from the Performance Scales and related items

There is an acute need for a reliable and valid quantitative outcome measure of “global disability” in multiple sclerosis (MS) from the patient’s perspective. The North American Research Committee on Multiple Sclerosis (NARCOMS) registry, a volunteer registry that represents approximately 10% of the U.S. MS population affords a unique opportunity to develop and validate such a measure. Since 1998, the NARCOMS registry has employed the Performances Scales (PS) to assess perceived disability in adults living with MS [1]. Single-item PS were originally developed for eight domains of function (mobility, hand function, vision, fatigue, cognition, bladder/bowel, sensory, and spasticity) [1]. To increase content validity [2],[3], three more measures were added in 2001 to assess disability associated with pain [4], depression [5], and tremor/coordination [6]. Responses are recorded on a 6-point ordinal scale (0 normal, 1 minimal, 2 mild, 3 moderate, 4 severe, and 5 total disability) except for the mobility PS which is scored from 0 to 6.

The measurement properties of each individual PS are generally excellent. Non-response rates are low (1%-5%) and test-retest reliability high (median, 0.82, range 0.65-0.91) [1]. Evidence of criterion and construct validity is strong for virtually all the scales (Additional file 1) [4]-[9]. Although the PS have generally been analyzed separately, several authors have added raw scores on the eight original PS to form an ordinal summary index of disability, referred to as Performance Scales Sum (PSS-8) [1],[7],[10]-[12]. The PSS-8, has shown favorable properties in terms of internal consistency (Cronbach’s alpha, 0.78), test-retest reliability (intraclass coefficient of correlation, 0.89), and criterion validity (correlation with EDDS, 0.62-0.64; correlation with MSFC, 0.58) [1],[7]. Growing evidence also supports its discriminant and incremental validity [1],[10]. Furthermore only minimal response shift has been detected when administering the PS repetitively over a 1-year period [11].

An important unresolved question about the PS is that of whether a single sum score, such as the PSS-8, adequately reflects underlying global disability or whether two or more summary scores are needed to validly capture information on disability domains assessed by the PS. For instance, recent factor analysis of 7 of the original 8 PS (vision scale excluded, PSS-7) suggested that a better representation of a patient’s disability might be obtained with two separate scores─one combining the mobility, spasticity and bladder/bowel PS, and the other combining the hand function, fatigue, sensory, and cognition PS [11].

We and others have described how categorical factor analysis and bifactor analysis could help uncover the fine structure of disability in MS [13],[14]. In this article, we apply these methods and related techniques put forward by the Patient-Reported Outcome Measurement Information System (PROMIS) network [15] to the evaluation of the measurement structure of the 11 PS in a large cross-sectional sample of NARCOMS registrants. We explain how results from these analyses informed the Item Response Theory (IRT) calibration of the PS on a single scale of self-assessed global disability. Finally we compare the construct validity of three summary scores of global disability derived using assumptions and calculation methods of varying practicality and accuracy. To appeal to a wide readership, methodological details are presented in Additional file 2.

Of the 12,563 persons who filled a NARCOMS intake questionnaire between 2003 and 2011, 7,851registrants with self-reported diagnosis of MS completed all 11 PS. Nearly 80% of participants were women; 93% were white, and 53% completed their intake questionnaire in 2007 or later. Mean age was 46 years (SD, 11.1); mean age at diagnosis 39 years (SD, 10), and mean disease duration 15 years (SD, 11.3). Two-third of respondents were on disease-modifying therapy; 51% were unemployed.

Except if stated otherwise, all the results below were obtained from the validation sample after revision of PS response options as described in Additional file 2 (i.e., after a first round of analysis indicated that response options should be reduced from 7 to 6 for the mobility PS and from 6 to 5 for all the other PS except the fatigue PS). PSS-11 scores, (i.e., traditional raw summed scores) ranged from 0 to 43 out of a revised maximum total of 46.

This study supports the notion that information on self-assessed disability elicited by the PS is adequately captured by a single score of global disability. However, several limitations of the standard PSS-11 score were uncovered in this role in terms of excessive number of response categories for 10 items (Additional file 2), underestimation of global disability in the lower part of the scale and overestimation of global disability in the upper part (Figure 4), residual dependency among items from the physical and mental domains of disability (Figure 2), and equal weighing of items (Figure 3). In comparison, the IRT summed score derived from the PSS-11 was shown to be a superior summary measure of patient-assessed global disability in MS with construct validity greater than that of the PSS-11 and similar to that of the harder-to-calculate IRT pattern score.

In this study, we developed a table (Additional file 3) to easily obtain a patient’s IRT summed score of global disability from their raw PSS-11 score. We showed that this IRT estimate has a reliability of measurement (0.87) that is appropriate for both comparisons between groups and individual-level monitoring. To facilitate intuitive interpretation of where a patient stands relative to the distribution of scores in the NARCOMS reference sample, we transformed raw IRT scores into scaled scores with mean 50 and SD 15 in this sample. Finally, we created a diagram that provides graphical information about how each PS contributes to respondents’ perception of their level of overall disability (Figure 3).

We encountered several challenges in data analysis which directed us toward solutions that were less than perfect from a pure measurement perspective. Some experts stress the importance of focusing efforts on well-defined, narrow and strictly unidimensional constructs in order to meet Rasch requirements for fundamental measurement [31],[32]. These experts would probably point out that the current PS do not fully cover the breadth of the dimensions of physical disability and, especially, mental disability─they would likely recommend to write and test new PS in order to create two clearly-distinct, and better-defined, unidimensional measures of physical and mental disability. Instead, we adopted the position of those experts who emphasize clinical appropriateness at the cost of small measurement bias [33]-[36]: i.e., we relied on what the bifactor model enables and gave priority to incorporating in a single summary measure the domains of disability most commonly affected by MS. Analyses suggested that two underlying dimensions of disability could be identified in the data, but that these dimensions were ill-defined and highly correlated. Little reliable information on what we loosely called “physical” and “mental” disability was left in the PS after having extracted information on global disability (this information was provided by the minority of patients affected to markedly different degrees by physical and mental disability). Scores of residual physical disability and residual mental disability were significantly associated with the same patient characteristics. Furthermore, for both scores of residual disability the patterns of associations with patient characteristics was similar to that observed for scores of global disability. With one exception, only the strength of some associations differed slightly depending on the disability score examined. This exception pertained to high PDDS scores, which were positively correlated with high scores of global disability and high scores of residual physical disability, but not with high scores of mental disability. This finding is likely, at least in part to be a reflection of the bias of the upper portion of the PDDS scale toward mobility disability and physical disability in general.

It is crucial for NARCOMS to make full use of the voluminous amount of self-assessed disability data collected during the past decade. For majority of respondents only the 8 original PS are available. One appealing property of IRT is that it will theoretically be possible to use our final IRT model to generate comparable summed and pattern scores of global disability from different subsets of the 11 PS, including the 8 original PS [37]. Future work will be needed to establish how reliable these scores are [35]. Equally important will be the need to determine the longitudinal validity of the IRT scales. Because the IRT framework allows one to model the effects of multiple sources of bias, it will be essential to conduct validation studies not only with IRT score estimates and conventional statistical methods, but also using latent variable methods such as multidimensional IRT modeling and IRT modeling for longitudinal data.

In summary, our work suggests that `global disability’ due to MS is a statistically validated construct that may be readily assessed with the simple and quick-to-administer patient-rated PS. Although, this work was concerned with methodological issues involved in deriving such a score of global disability, it has implications for our understanding of the disease. We showed that self-assessed disability in MS can be conceptualized as a multifaceted construct encompassing elements of both physical and mental disability. Although all the elements considered were closely correlated, the balance between them varied among patients with identical global disability score. This result, derived based on statistical grounds, challenges the traditional view implicit in accepted disability scales (e.g. EDSS), that `physical’ disability is of primary import in MS.

EC designed the study, conceived the statistical methodology, analyzed the data, and prepared the manuscript. IK contributed to the conception of the paper, helped in result interpretation, and provided substantive revisions. GRC helped conceive the research idea, directed data collection, advised on specific statistical issues, and contributed to the final writing of the paper. All authors read and approved the final manuscript. EC had final responsibility for the decision to submit the manuscript for publication.