The AMC Linear Disability Score project in a population requiring residential care: psychometric properties

It is now widely accepted that examining quality of life is an important aspect in the treatment and evaluation of many conditions. Functional status is seen as an important determinant of quality of life. A wide variety of instruments have been developed to quantify functional status [1]. These instruments tend to have a fixed length and all items are administered to the whole group of patients under scrutiny. However, currently interest is moving towards the more flexible framework offered by item banks. An item bank is a collection of items, for which the measurement properties of each item are known [2, 3]. When using an item bank, it is not essential for all respondents to be examined using all items. This enables the burden of testing to be considerably reduced for both patients and researchers. It is even possible to select the 'best' items for individual patients using computerised adaptive testing algorithms [4]. Furthermore, results from studies using different selections of items from an item bank can be directly compared. Item banks, measuring concepts such as quality of life [2, 5], the impact of headaches [6] or functional status [7, 8], have been developed.

The AMC Linear Disability Score (ALDS) project item bank was developed to quantify functional status [7, 9]. The ALDS item bank covers a large number of activities, which are suitable for assessing respondents with a very wide range of functional status and many types of chronic condition. The item bank is particularly suitable for use in the Netherlands. The ALDS items were obtained from a systematic review of generic and disease specific functional health instruments [1]. Five psychometric aspects of the ALDS item bank need to be considered before it can be implemented. These are: (a) there needs to be enough variation in the response categories used for each item [9]; (b) estimates of the item response theory model parameters should not depend on patient characteristics such as age or gender [10, 11]; (c) estimates of the item response theory model parameters, which are stable across different subsets of items from the instrument and based on a sufficiently large sample [12] of respondents, should be available [9]; (d) an examination of the extent to which the ALDS items represent a single construct; and (e) testing whether a simpler item response theory model is suitable for the set of items.

This paper examines these five aspects of the ALDS item bank using the responses given by residents of supported housing schemes, residential care and nursing homes in and around Amsterdam, the Netherlands. This, mainly elderly, population has been chosen because they generally experience some level of functional restriction and consume a large amount of health care services.

Of the 160 items included in the item bank, one was removed because it was worded differently in two different item sets. Of the 159 remaining items, 77 were removed from the item bank. This process is described in Table 1. In step (a), 28 items were excluded from further analysis because fewer than 10% or more than 90% of responses were in the category 'cannot'. In step (b), 26 items were removed because they had significantly different estimates of the item difficulty parameter (β _i ) for for males and females and/or for younger and older respondents. Of these 26 items, 19 had different measurement characteristics for females and for males, 5 items had different measurement characteristics for those aged under 85 and for those aged 85 or over, and 2 items had different measurement characteristics for both males and females and for older and younger respondents. In step (c), 23 items had an item fit statistic p-value of less than 0.01. In addition, 3 items were excluded from further analysis because the value of the item difficulty parameter (β _i ) was significantly different between two item sets of items. Hence, 79 psychometrically sound items remained in the item bank. A short description of the content of the 79 items in the final version of the calibrated item bank, together with estimates of the dispersion (α) and difficulty (β) parameters and their standard errors, are given in Tables 2a and 2b. Following step (c) of the analysis, the anchors between the sets of items contained between 13 and 23 items. In addition, there was no evidence to suggest that estimates of θ do not follow a Normal distribution (Kolmogorov-Smirnov test, p-value = 0.637). In step (d), the full information factor analysis indicated that, for three of the four anchors between the item sets, there was only one latent root of the correlation matrix larger than 1. In the fourth item set, a second latent root was marginally above 1. The percentage of the variance explained by the first factor varied between 67% and 72%. The values of Cronbach's alpha coefficient for the four anchors were between 0.86 and 0.93. The confirmatory factor analysis carried out on the whole data set indicated that 70% of the variance was explained by the first factor. Cronbach's alpha coefficient for the whole data set equalled 0.98. In step (e), the one-parameter logistic item response theory model was fitted to the 79 items remaining after step (c). This model fitted the data significantly less well than the two-parameter model (p-value < 0.0001). For 3 items, the item fit statistic had p-value < 0.01. After removal of these items, the two-parameter model was still significantly better than the one-parameter model (p-value < 0.0001).

In this study, the psychometric properties of the item bank have been examined using a sample of 555 respondents and an incomplete calibration design. Each item was presented to between 262 and 293 respondents. These figures are above the minimum, of 200 respondents, regarded as necessary to implement the models used in this paper [12]. It could be argued that it would have been desirable for all respondents to be presented with all items, but this would have placed an unacceptable burden on the, often frail, population in this study. Incomplete calibration designs are regularly implemented in the development and maintenance of item banks used in educational testing [4, 14] and have gained some recognition in health related applications [15]. Developments in psychometric theory mean that it is now possible to perform the same types of analysis on data resulting from incomplete designs, as is performed on data from complete calibration designs [22, 23, 25]. The number of items in the anchors following the analysis, indicate that the design was still amply linked [9].

In this study, 81 (51%) of 160 items were removed from the item bank because they did not conform to the psychometric standards required of the item bank. This is a much higher level than would be expected in the calibration of an item bank for use in educational measurement. However, when the results are examined more carefully, 28 items were removed because they were too difficult or too easy for the population in this study. In addition, 26 items were removed because they had different item parameters for different groups of respondents. These problems would have been identified much earlier in an educational item bank. Hence, only 26 (25%) of 106 items were removed due to item misfit. The number of items retained in the item bank may have been higher if a more flexible model, based on, for example, non-parametric smoothing techniques had been used [26]. However, this type of model is less suitable as a base for implementing modern testing algorithms, such as computerised adaptive testing. In addition, it is possible that more items could be made available if the items demonstrating differential item functioning were included in the item bank with different item location parameters (β _i ) for males and females or for younger and older respondents. This may seem complicated, but is straightforward in the framework of a computerised item bank.

This paper has concentrated on the two-parameter logistic item response theory model. However, the one-parameter logistic item response theory model was also fitted to the 79 items remaining in the item bank. This model fitted the data significantly less well than the two-parameter model, even after 3 items demonstrating misfit at the item level were removed. This confirms the choice of the two-parameter model. This model was chosen because it allows the probability of responding in the category 'can' to be modelled more flexibly than when the one-parameter logistic model is used. This enables a more realistic model for the data to be built than when the more restrictive approach associated with the one-parameter model is chosen [18].

This paper has examined the calibration of the ALDS item bank in a population requiring residential care. It has been shown that the item bank has sound psychometric properties and could form a stable base for a wide range of applications. However, it is possible that the items will have different measurement characteristics for patients requiring treatment for specific chronic conditions or in other countries. Hence, it is important that the ALDS item bank is tested carefully before it is used to assess the functional status in other groups of respondents or in other countries.

Now that the measurement properties of the ALDS item bank have been examined carefully, the item bank can be used as a foundation for quantifying functional status. If modern algorithms, such as computerised adaptive testing, are implemented, it will be possible to obtain accurate measurements, whilst keeping the burden of testing on respondents and interviewers to a minimum. Items can be selected for use in further research, for allocation individuals to appropriate care settings and for calculating institutional funding based on the actual care load. It is hoped that the ALDS item bank will play an important part in the implementation of computerised adaptive testing of functional status.

RH and RL were supported by a grant from the Anton Meelmeijer fonds, a charity supporting innovative research in the Academic Medical Center, Amsterdam, The Netherlands.

RL conceived the study and supervised the data collection. RH prepared the first draft and carried out the analyses. RL, MV and RJdH critically reviewed the manuscript. RH prepared the final version.