Validity and reliability of a performance evaluation tool based on the modified Barthel Index for stroke patients

A stroke is a life-threatening medical emergency, but if patients survive a stroke, they may still face prolonged difficulties in activities of daily living (ADL) due to severe brain damage. Since many different symptoms and conditions can develop after a stroke, multidisciplinary rehabilitation is critical for people recovering from this disease [1]. In applying rehabilitation therapies to patients with stroke (as with any other disorder), goal setting is essential, and to measure progress in achieving the goals, outcome measurement tools are necessary [2]. Accurate assessment of stroke patients’ ADL greatly helps evaluate the efficacy of stroke medications and rehabilitation. In fact, in measuring the progress of stroke rehabilitation, the Japanese Guidelines for the Management of Stroke [3] recommend implementing ADL assessment scales (in addition to overall, motor function, and muscle tone scales) that have been demonstrated to be reliable and valid. A variety of ADL assessment scales has been developed for stroke patients, as well as for patients with other diseases and conditions. These include the Barthel Index (BI) [4], the modified BI (MBI) [5‐8], the Functional Independence Measure (FIM) [9, 10], the Stroke Impairment Assessment Set (SIAS) [11], and the modified Rankin Scale (mRS) [12]. Each of these tools has different advantages and disadvantages. For example, some of them are easy to use, but not as detailed as the others. Some are sensitive to changes, but require advanced knowledge for their administration.

The BI [4] was originally established for assessing the ADL of stroke patients and has been widely used for this purpose. Several groups have developed revised versions [5‐8]. In particular, the MBI, created by Shah et al. [5], was developed to achieve greater sensitivity, and its internal consistency has been confirmed for use among stroke patients. The Japanese version of the performance evaluation tool based on the MBI (PET-MBI) was created with permission from the authors of the original MBI (including a back-translation process against Shah et al.’s MBI), and its reliability and validity were then verified in a study targeting 110 elderly individuals requiring care residing in care facilities [13]. In addition, the factorial validity of the PET-MBI for 126 elderly individuals requiring care living at home has also been verified [14].

However, it is still unclear if the PET-MBI can be used as a functional assessment instrument for patients with stroke. So far, the reliability of this tool has only been verified in studies limited to elderly individuals requiring care in which inter- and intra-rater reliabilities were assessed by only two raters. Thus, the reliability of the PET-MBI has not yet been rigorously tested. Furthermore, the concurrent validity of this tool against the established ADL assessment scales is unknown. To address these issues, the reliability and concurrent validity of the PET-MBI for stroke patients were examined in this study.

The aim of the research was twofold: 1) validity study, in which the concurrent validity of the PET-MBI (ADL performance) against the BI (ADL capacity) was examined by obtaining the ADL scores of these two measures in stroke patients using direct observation techniques (although this was within the same one-week period, raters evaluated BI in the rehabilitation room, and MBI in daily situations outside the rehabilitation rooms. Therefore, the observations did not pertain to the same scenes); and 2) reliability study, in which stroke patients’ ADL was videotaped, and following editing of these videos, they were evaluated by raters using the BI (ADL capacity) and the PET-MBI (ADL performance). The video raters belonged to different medical institutions from those that conducted the direct examination described above. Ten raters were used for assessment of inter-rater reliability. To examine intra-rater reliability, evaluation of videotaped ADL was carried out twice, one month apart.

Of the 30 stroke patients evaluated using both the direct and video evaluation methods, 23 (76.7%) were men. The mean age of all patients was 71.9 years, and 80.0% of them were ≥65 years old and over. The mean duration between the onset of stroke and video recording was 88.0 days. Table 1 shows the basic information for these patients. Both direct evaluation and video evaluation were conducted by physical and occupational therapists (five each) (Table 2).

For the total scores obtained by the direct evaluation method, Pearson’s correlation coefficient of the BI versus the PET-MBI was 0.95, and the lower limit of the 95% confidence interval (CI) was 0.90. Identical values were obtained by Spearman’s rank correlation coefficient (Table 3).

For inter-rater reliability, the ICC using the total score of the first PET-MBI, which was the primary outcome, was 0.99, and that using the total score of the second PET-MBI, which was the secondary outcome, was also 0.99. When the scores of 10 PET-MBI items were independently analyzed, κ and κ_w coefficients were 0.61–0.89 and 0.77–0.94, respectively, for the first session, and 0.63–0.89 and 0.74–0.95, respectively, for the second session (Table 4).

As the primary outcome of intra-rater reliability, an ICC of 0.99–1.00 was obtained for the PET-MBI total scores. Finally, the κ and κ_w coefficients of 10 PET-MBI item scores were independently calculated as the secondary outcome, and they were 0.78–0.92 and 0.85–0.96, respectively (Table 5).

In this study, the correlation between the PET-MBI and the BI was analyzed first. When hospitalized stroke patients were directly assessed by these two measures, high correlation coefficients were obtained, strongly suggesting their concurrent validity. The reliability of the PET-MBI was then evaluated by applying it to stroke patients using a video evaluation technique. The results demonstrated high intra- and inter-rater reliabilities. This is consistent with a previous study conducted with elderly individuals [13]. Video evaluation techniques have been used in previous studies assessing ADL, which change over time, because they enable investigation of reliability through evaluation of ADL at a single point by multiple raters [20, 21]. In the present study as well, the use of this technique as a main evaluation method enabled assessment of PET-MBI in which 10 raters observed the same stroke patients.

The correlation coefficients of the total scores of the BI and the PET-MBI were high when the rater directly evaluated using these two measures. A correlation between the BI and the MBI has been reported elsewhere [22]. However, in the present study, there was also a correlation between the BI (ADL capacity) and the PET-MBI (ADL performance), which is an important finding.

In conducting this research, factors that influence the outcomes of raters’ evaluation were eliminated as much as possible. For example, the viewing orders of the ADL videos of the 30 patients were randomized for all raters, making it harder for them to share information. Similarly, the viewing orders were different for each rater between the first and second evaluation sessions, reducing a potential bias in the second session caused by the memory of the first. Thus, the fact that the PET-MBI still showed high inter- and intra-rater reliabilities strongly suggests the reliability of the present results. This high reliability of the PET-MBI may be attributable to the fact that its evaluation criteria are easy to understand, and the fact that the raters received training in advance using the manuals for this assessment tool. However, the manuals provided only the minimum necessary information, and their use was considered to be within acceptable standards of the reliability assessment studies of various evaluation tools. It has been recommended that the outcome measures of stroke patients have established psychometrics (reliability, validity, and sensitivity to change) [23]. Since the usability, reliability, and validity of the PET-MBI have now been established, we expect that this tool will be invaluable in clinical settings in evaluating stroke patients.

The inter-rater reliability for the item of grooming was “substantial”, but its κ coefficients were lower than those of the other items that mostly showed “almost perfect” reliability. The reason for this relatively low reliability may be related to the fact that therapists have fewer opportunities to observe patients’ grooming activities in everyday settings in rehabilitation at Japanese hospitals compared to other activities, as well as the fact that the evaluation criteria for this item were complex. Grooming includes several different activities. In this study, however, only tooth brushing was evaluated, and this activity was divided into the following three processes: preparation (put toothpaste on a toothbrush); execution (brushing); and completion (rinsing and tidying up). Despite these preparatory efforts, the reliability data for this particular item were less than ideal. As such, one must be careful when applying the PET-MBI to stroke patients in actual clinical settings.

There were some limitations in the present study. First, the only stroke patients targeted in this study were inpatients. In addition, we excluded any patient whose condition was unstable due to stroke, as well as those who were not suitable for video recording. These exclusions may have limited the generalizability of our findings. The use of PET-MBI should be considered for outpatients as well, and be scored by healthcare professionals other than therapists. Second, the raters had many years of clinical experience, and thus they may have been skilled at ADL scoring. However, they were unaccustomed to the evaluation styles used in this study. Thus, it is unlikely that their experience significantly affected the data. In addition, it was unclear whether the results were influenced by differences in sex, age, and duration of time practicing between those raters in the direct observation versus the video observation group. Although there were methodological differences between the validity study (direct observation) and the reliability study (video observation), we do not feel that these differences were likely to have influenced the results, since each of the total ICC scores was nearly perfect. Third, scores of the BI and the PET-MBI might have been mutually influential, due to the direct evaluation. However, BI measures the capacity of ADL (“can do”), while the PET-MBI measures performance of ADL (“do”). Moreover, we consider the mutual influence to be small, given the differences in observation scenes. Fourth, the raters were trained in advance using sample videos and manuals. This may have helped to produce consistently uniform evaluation results. However, the manuals provided only the minimum necessary information, and their use was considered to be within acceptable standards of the reliability assessment studies of various evaluation tools. Lastly, some might suggest that the reliability of the video evaluation technique increased because the videos were specifically edited to include sufficient information necessary for functional assessment. However, these videos showed only part of the patients’ ADL. Therefore, when compared with direct observation, the raters acquired much less information regarding patients’ functional dependency; this is precisely the reason that this scoring method was introduced into the protocol. Thus, it is unlikely that there would be significant bias in evaluation caused by video editing.

The PET-MBI showed high concurrent validity when applied to stroke patients using the direct observation method and high intra- and inter-rater reliabilities in performing functional assessment by the video evaluation method. The PET-MBI is likely to become a convenient ADL evaluation tool that can be used by anyone.