Background
Neuropsychological deficits occur in about 40-65 % of patients diagnosed with multiple sclerosis (MS). Deficits in attention and information-processing speed as well as long-term and working memory are most common, [
1] whereas language and general intellectual ability seem to be largely unaffected [
2].
Although these deficits exert a profound impact on patients’ quality of life, they frequently remain undiscovered during routine clinical examinations [
3,
4]. This may be attributed to the fact that in clinical practice, time for exhaustive neuropsychological assessments is sparse. In the past, this problem has been acknowledged and tackled by employing short test batteries with the explicit purpose of diagnosing cognitive deficits in MS [
5,
6]. Among them, the Brief International Cognitive Assessment for Multiple Sclerosis (BICAMS) [
6] and the Brief Repeatable Battery of Neuropsychological Tests (BRB) [
7,
8] have been widely accepted as valid screening tools for testing MS-patients [
6,
9‐
11]. The BICAMS is extremely short, taking up only approximately 15 minutes of testing time, while the BRB entails a relatively lengthy testing procedure of approximately 90 minutes duration. However in the BICAMS, there is a decisive lack of assessment of executive functions, since it focuses on processing speed as well as verbal and nonverbal memory [
6].
In a pioneering study by Portaccio et al., the performance of a shortened version of the BRB as a quick and economic screening tool was assessed [
12]. This short form of the BRB comprises the Paced Auditory Serial Addition Test (PASAT), addressing working memory and attention, the Symbol Digit Modalities Test (SDMT), referring to attention and information processing speed, as well as the Selective Reminding Test (SRT) assessing verbal memory. The authors report that failure on one of these three subtests predicted neuropsychological deficits with high sensitivity (94 %) and specificity (84 %).
While the latter findings are promising concerning the application as a brief assessment tool, they need to be interpreted in the context of a noteworthy limitation. In particular, the authors examined sensitivity and specificity of the aforementioned subtests with regards to cognitive deficits, as determined by the whole BRB and an additional Stroop Test for additional information regarding potential executive dysfunction. Consequently, there was a considerable overlap of tests included in the short screening version of the BRB on the one hand (PASAT, SDMT, SRT), and the procedure which was implemented to derive reliable information about the actual presence of cognitive deficits on the other hand (BRB and Stroop Test). One may argue that similar classification patterns between the shortened version of the BRB and the extended procedure (BRB and Stroop Test) may have been confounded by the fact that all tests of the short version of the BRB were actually included in the more extensive procedure. By this reasoning, estimates of sensitivity and specificity might have been distorted.
The purpose of the current study was to reassess the findings of the pioneering study by Portaccio et al. [
12] while avoiding its methodological bias. Thus, a more economic neuropsychological testing of MS patients during clinical routine could be achieved. To this end, sensitivity and specificity of the short version of the BRB were examined with regards to the presence or absence of cognitive deficits as determined by an extensive neuropsychological diagnostic procedure not including the BRB subtests in question. The latter procedure of two hours duration was implemented to thoroughly examine all cognitive domains which may be found deficient in MS-patients. The use of further tests in the screening-procedure (including the Stroop Test) for validation – although desirable – would have significantly hampered the aim of keeping the screening short and was therefore relinquished.
Results
Considering the extensive testing procedure, 72 patients (57.1 %) showed cognitive deficits, whereas in the short version of the BRB, 75 patients (59.5 %) were identified as cognitively impaired. When matching the whole of the screening (SRT, SDMT, PASAT) with the gold standard of the extensive test battery, sensitivity was 77.8 % and specificity was 64.8 % (Table
3).
Table 3
Performance of screening-subtests (single and combined)
SRT | 38.36 % (27.44 % - 50.51 %) | 81.48 % (68.13 % - 90.30 %) |
SDMT | 43.84 % (32.41 % - 55.91 %) | 94.44 % (83.66 % - 98.55 %) |
PASAT | 41.67 % (30.35 % - 53.88 %) | 87.04 % (74.48 % - 94.19 %) |
SRT + SDMT + PASAT | 77.78 % (66.15 % - 86.39 %) | 64.81 % (50.55 % - 76.97 %) |
Screening subtests were also individually matched with the whole of the extensive test battery (Table
3), resulting in lower sensitivity values (38 % for the SRT, 41.7 % for the PASAT and 43.8 % for the SDMT), but increasing specificity values (81.5 % for the SRT, 87.0 % for the PASAT and 94.4 % for the SDMT).
Finally, obtained estimates of sensitivity and specificity of the three subtests of the screening matched with their respective domains from the extensive test battery are displayed in Table
4. Sensitivity ranged from 49.1 % in the working memory domain (PASAT) to 60.0 % in the memory domain (SRT), and a corresponding estimate in the speed domain of 52.9 % (SDMT). Specificity was considerably higher, ranging from 81.5 % in the memory domain to 89.5 % in the speed domain, with the memory domain falling in between these two with a value of 84.9 %.
Table 4
Performance of screening-subtests in respective extensive test battery domains
SRT | 60.00 % (42.21 % - 75.65 %) | 81.52 % (71.78 % - 88.57 %) |
SDMT | 52.94 % (38.60 % - 66.84 %) | 89.47 % (79.79 % - 95.02 %) |
PASAT | 49.06 % (35.25 % - 63.00 %) | 84.93 % (74.21 % - 91.88 %) |
In sum, consistent estimates above chance level for both, sensitivity and specificity, were only reached in case of the first approach, i.e. matching the whole of the screening with the extensive test battery. While utilizing individual screening subtests increased specificity, this approach involved considerably attenuated sensitivity. Further information regarding the number of cognitively impaired patients in each subtest of the screening as well as each cognitive domain in the extensive testing procedure can be reviewed in Additional file
1: Supplement 3.
Discussion
A thorough diagnosis of cognitive deficits in MS-patients is a time- and resource-consuming process consisting of a large number of neuropsychological test procedures. In clinical practice, where time for excessive assessments is sparse, such a thorough diagnosis is often not feasible. Therefore, short assessment methods could essentially improve the diagnostic process.
In the current study, MS-patients completed both, an economic screening session by means of a brief version of the BRB, as well as an extensive diagnostic procedure closely adhering to established standards of an extensive testing in MS
5,6. The proportion of patients who were identified as displaying cognitive deficits was relatively similar between the screening (59.5 %) and the extensive procedure (57.1 %). As such, it resembles common estimates, according to which neuropsychological deficits occur in about 40-65 % of MS-patients [
1].
Matching the whole of the short version of the BRB with the whole extensive testing procedure resulted in a sensitivity of 77.8 % and a specificity of 64.8 %. In the current work, this matching constellation was the only one which produced consistent estimates of sensitivity and specificity above chance level. In contrast, a clearly different pattern of results emerged when BRB subtests were individually matched with results of the entire extensive testing procedure. Here, sensitivity was lower (38.7 % to 43.8 %), whereas specificity was somewhat increased (81.5 % to 94.4 %). Sensitivity estimates for the individual subtests of the short version of the BRB referring to the respective domains of the extensive testing procedure resulted in only marginally increased sensitivity (49.1 %-60.0 %) and approximately equal specificity (81.5 %-89.5 %). While this common pattern of elevated specificity relative to sensitivity on the subtest-level is compatible with previous reports by Portaccio et al. [
12], overall estimates of sensitivity and specificity obtained in the current study are considerably lower than those reported by the latter authors.
While Portaccio et al. [
12] reported specificity estimates of 84 % for the short version of the BRB, in the current study, 64.8 % were obtained. The same pattern holds for sensitivity, where Portaccio et al. [
12] reported 94 %, whereas in the current study, 77.8 % were obtained. As previously indicated, in the study by Portaccio et al., [
12] there was a considerable overlap of tests included in the short screening version of the BRB on the one hand, and the procedure which was implemented to derive reliable information about the actual presence of cognitive deficits on the other hand. Similar classification patterns between the shortened version of the BRB and the more extensive procedure in the latter work might hence have been the result of overestimation. In support of this assumption, estimates of the current work, which were based on an extensive diagnostic procedure completely independent from the screening, were considerably lower.
While estimates of specificity and sensitivity in the current work were lower than those of Portaccio et al., [
12] it is remarkable that despite the fact that a relatively independent diagnostic procedure was implemented, estimates were still reasonable, when the screening was considered as a global predictor. In particular the important parameter
sensitivity, representing the proportion of patients adequately identified as cognitively impaired by the screening, showed a reasonable estimate. While adequacy in identifying patients without cognitive deficits, reflected by specificity, was somewhat lower, it may be argued that the latter group of patients would probably be examined by an extensive procedure subsequently by default in context of routine clinical practice. In that case, a false positive result would yield a subsequent extensive examination to verify the screening result. The fact that the screening actually identified
more patients as cognitively impaired than the extensive testing procedure further underlines its usefulness for a short assessment and its appropriateness as a screening tool. In sum, the current findings may be regarded as complementing previous suggestions by Portaccio et al., [
12] as they provide further support for the utility of the short version of the BRB as a valid screening tool. Nevertheless, it needs to be emphasized that according to the current findings, the feasibility is only given when the screening is regarded as a global indicator.
Another noteworthy aspect of the current study concerns the usefulness and necessity of the application of the PASAT in future diagnostic procedures. In the past, it has often been pointed out that the PASAT is somewhat flawed since it requires a certain amount of mathematical ability [
23]. It also acts as a potential stressor, since patients are required to keep up with the pace of the number reading [
24]. Therefore, the question has been raised whether to completely abstain from using it in test batteries and instead favouring the SDMT, which is generally better accepted by patients [
25]. The current study shows that, even though the SDMT is slightly superior to the PASAT in terms of specificity, each test by itself has a fairly low sensitivity. Only combined with each other and the SRT can they reach sufficiently high levels of sensitivity to be deemed as having an adequate predictive value.
In the current study, we decided against implementing a Stroop-Test in addition to the short version of the BRB. On the one hand, executive functions are already being addressed by the PASAT. On the other, a Stroop-Test would significantly lengthen the screening-procedure, [
26] which would countermand the aim of an economic screening tool. However, it should be noted that the underlying cognitive constructs of the PASAT are still a matter of debate [
23,
24] and that a short test for executive function in MS is still lacking. It is for this last reason that we decided against the BICAMS and for the short BRB in our study. Both approaches require approximately 15 minutes (testing time only) and both consist of three subtests. But even though validation of the BICAMS is currently underway in several countries as a short screening tool in MS [
27], we consider the three subtests of the BRB to better cover the width – if not the depth – of neuropsychological constructs possibly affected in MS, since they have the arguable benefit of also assessing executive functions through the PASAT.
While our results are generally compatible with the extant literature and provide an extension with regards to the previous work by Portaccio et al., [
12] it should be noted that in the current study, the extensive diagnostic procedure was not implemented on the same assessment occasion as the screening. Consequently, a second appointment had to be scheduled. Relatively decreased specificity and sensitivity estimates may have been affected by these circumstances. On the other hand, it is noteworthy that despite the delay between testing procedures, specificity and sensitivity estimates of the short version of the BRB were still reasonable. However, since stability of cognitive deficits in MS over time is still debated and longitudinal studies in this field of research are scarce, [
28] time intervals between screening and extensive testing should be avoided in further research in this area.
Competing interests
The current article was supported by Bayer Vital GmbH. Dr. Keune, Dr. Münßinger and Prof. Dr. Oschmann received research support and compensation for activities with Bayer Vital GmbH.
Authors’ contributions
SH coordinated the implementation of the study, performed the data analysis and drafted the manuscript. JM helped implement the study and supported the data analytic process. SK helped implement the study. SL and PO participated in the development of the study design and edited the manuscript. PK supported the data analytic process and edited the manuscript. All authors read and approved the final manuscript.