The Academic Medical Center Linear Disability Score (ALDS) item bank: item response theory analysis in a mixed patient population

Functional status is now seen as an important determinant of patients' quality of life and a wide variety of instruments have been developed [1]. Instruments for quantifying functional status tend to have a fixed length and administer all items to the whole group of patients under scrutiny. Currently, interest is moving towards the more flexible framework offered by item banks in conjunction with item response theory. An item bank is a collection of items, for which the measurement properties of each item are known [2, 3]. Since item response theory centres on the measurement properties of individual items, rather than the instrument as a whole, it is not essential for all respondents to be examined using all items when using an item bank. It is even possible to select the 'best' items for individual patients using computerised adaptive testing algorithms [4]. This can reduce the burden of testing considerably for both patients and researchers. Furthermore, results from studies using different selections of items can be directly compared. Item banks measuring concepts such as quality of life [2, 5], the impact of headaches [6], fatigue [7, 8] and functional status [9‐12] have been described.

Before an item bank can be implemented, it is essential to calibrate it. During the calibration process, the measurement properties of the individual items and the item bank as a whole are investigated. In contrast to the procedures used when developing fixed length instruments, it is not essential to present all items to all respondents in the calibration sample. It is often more efficient to offer targeted sets of items to particular groups within the sample. The items in common between any two sets of items are known as anchors. This kind of design is known as an incomplete, anchored calibration design and allows all items and patients to be calibrated on the same scale [13]. These designs have been used widely in preparing item banks for educational testing and has had some recognition in the development of medical instruments [14, 15]. Developments in psychometric theory mean that it is now possible to perform the same types of analysis on data resulting from incomplete designs, as on complete designs [16‐18]. The consequences of the use of this kind of design in the development of the ALDS item bank have been discussed previously [14, 19]. If the primary aim of a study is to estimate the parameters of the two-parameter logistic item response theory model, as in this paper, little statistical information can be obtained from patients, whose functional status is much higher or lower than the difficulty of the items, with which they are presented [14, 19]. The respondents described in this paper were chosen to maximise the statistical information on, and hence minimise the standard errors of the estimates of, the parameters of the item response theory model. For this reason, they may not be representative of the populations described.

The Academic Medical Center Linear Disability Score (ALDS) item bank was developed to quantify functional status in terms of the ability to perform activities of daily life. The ALDS item bank covers a large number of activities, which are suitable for assessing respondents with a very wide range of functional status and many types of chronic conditions. The items were obtained from a systematic review of generic and disease specific functional health instruments [1]. The methodology used to develop the ALDS item bank, including the use of incomplete calibration designs, has been described in depth [14]. Other papers have examined technical [20] and practical [21] aspects of methods to deal with missing item responses and the use of a 'not applicable' response category for the items. It has been shown that some of the ALDS items may have different measurement characteristics for males and females and for younger and older respondents [22]. The first results showed that the ALDS item bank had acceptable psychometric properties in a residential care population [9].

This study expands the results described in previous papers by examining the measurement properties of a selection of ALDS items, judged to be clinically relevant for a range of medical specialities, in a mixed population. The sample of respondents consisted of: residents of supported housing, residential care or nursing homes; patients attending an outpatients clinic for the treatment of chronic pain; hospitalized stroke patients; or attending an outpatients clinic for Parkinson's disease. These groups of patients were chosen because they have a broad range of chronic conditions and levels of functional impairment.

Of the 115 items that were regarded as suitable for inclusion in the ALDS item bank, 38 were removed from and 77 were retained in the item bank. In step (a), a total of 24 items were removed from further analysis. Two items had been presented to fewer than 200 respondents, 1 item had fewer than 10% of responses in the category 'can carry out' and 21 items had more than 90% of responses in the category 'can carry out'. In step (b), a total of 11 items were removed from further analysis. Four items had different measurement characteristics for younger and older patients. Seven items had different measurement characteristics for male and female patients. In step (c), 3 items were removed from further analysis because their item fit statistic had a p-value less than 0.01. The item parameters (α and β) are given, with their standard errors, in Table 2. The probability that respondents with a range of levels of functional status can perform the items is illustrated in Figure 2. A histogram of the values of the difficulty parameters (β _i ) is given in Figure 3. It can be seen that the items cover the whole range of functioning, although there are more 'easy' than 'difficult' items.

In step (d), the values of Cronbach's coefficient alpha varied between 0.92 and 0.97 for the six sets of items and was equal to 0.98 for the whole data set. The values for each set of items are given in Table 1. The data sets had between 1 and 3 latent roots larger than 1 and the whole data set had 5 latent roots larger than 1. In general, there was one very large latent root and a number marginally greater than one. The percentage of variance explained by a single factor varied between 64% and 78% for the item sets and was equal to 77% for the whole data set.

In this study, an item response theory analysis of the ALDS item bank has been examined using an incomplete calibration design and a sample of 1002 respondents from: supported housing schemes, residential care or nursing homes (551); an outpatients' clinic for patients with chronic pain (235); following a stroke (127); and an outpatients clinic for Parkinson's disease (89). Each item in the analysis was presented to between 297 and 705 respondents. This is well above the minimum, of 200 respondents, regarded as necessary to implement the two-parameter logistic item response theory model [17].

The resulting item bank contains 77 items representing a wide range of levels of functional status. Although there are a number of items, which have very similar item parameters or content, there is no need to reduce the number of items further. Since estimates of respondents' functionals status are comparable, even when different sets of items are used to score them, researchers can choose items, which are particularly relevant to their 'population'. In this way, accurate estimates can be obtained, whilst minimising the burden of testing on both researchers and participants in clinical studies.

Before the item response theory based analysis was carried out, 55 of 170 items included in the data collection design were removed from the item bank because they were judged to be unsuitable for inclusion in the ALDS item bank. The insight required to judge that some of the items were unsuitable for the ALDS item bank could only been obtained once the items had been presented to a wide range of respondents. In the future, when developing an entirely new item bank, it may wise to conduct a broad pilot study before embarking on the full calibration study.

Previous results have shown that a proportion of items in the ALDS item bank have different measurement properties for men and women and for younger and older patients [9, 22, 29]. These results have been confirmed in this paper. Ideally, potential differences between the measurement characteristics of the items for different patient populations, for different groups of raters and for the interview and self-report versions of the ALDS item bank should also be examined in the same way as the differences between age and gender based groups. However, this was not possible for two reasons. Firstly, the groups of respondents with Parkinson's disease or acute stroke were too small to perform this analysis satisfactorily. Secondly, the levels of functioning in the respondents with chronic pain were much higher than those of the respondents living in residential care. This means that it was not possible to compare the groups at similar levels of functional status. Thirdly, all of patients in any given group were rated in the same way. Hence, it is not possible to separate differences caused by groups of raters and those caused by characteristics of the patient groups.

The respondents described in this paper were chosen to maximise the statistical information on, and hence minimise the standard errors of the estimates of, the parameters of the item response theory model. For this reason, they may not be representative groups from the populations described. This is particularly true for the residents of supported housing, residential care or nursing homes and for the stroke patients.

This does not have any consequences for the interpretation and implementation of the estimates of the parameters of the item response theory model [14] or the item response theory based factor analysis, but means that the values of Cronbach's alpha should be confirmed in future studies. In addition, the results for patients after a stroke and with Parkinson's disease need to be confirmed due to the small sample sizes used. Furthermore, in future studies it will be essential to examine whether the 77 items presented in this paper have the same measurement characteristics if they are presented to patients in an interview by nurses or by doctors or if patients respond to the items in a self-report situation.

The results presented in this article are different to those presented in a previous article examining the data from the residents of supported housing schemes, residential care or nursing homes [9]. There are two main reasons for this. Firstly, the selections of items included in the analysis were different. Secondly, the data described in this paper were collected from a mixed population of respondents. Previous research has commented on the differences between the one-parameter and two-parameter logistic item response theory models. In this paper, the two-parameter logistic item response theory model has been used. This model was chosen because previous results have shown that the one-parameter logistic model is unsuitable for this type of data [9].

The results in this paper have shown that the ALDS item bank has promising measurement characteristics for a mixed patient population. The authors feel that the item bank can be used as a reliable indicator of functional health status in residents of supported housing, residential care or nursing homes, patients with chronic pain, acute stroke or Parkinson's disease. This paper has examined a mixed patient population, so the authors expect that the item bank will have good measurement characteristics for a wide range of other populations. However, care should be taken when using the ALDS item bank in other populations until these results have been confirmed.

Although this examination of the ALDS item bank has concentrated on six sets of items, future applications of the item bank are not bound to these sets of items. If these results are confirmed in future studies, the ALDS item bank will form a good foundation for a computerised adaptive testing procedure [4]. It would also be possible to select fixed length sets of items, specifically tailored to the level of functional status or clinical characteristics of a certain group of patients.

RH, NW and RL were supported by a grant from the Anton Meelmeijer fonds, a charity supporting innovative research in the Academic Medical Center, Amsterdam, The Netherlands.