A Systematic Review and Meta-Analysis of the Brief Cognitive Assessment for Multiple Sclerosis (BICAMS)

The Brief International Cognitive Assessment for MS (BICAMS) was designed by an expert consensus group to facilitate the assessment of cognition in MS [9]. In June 2010, an expert committee of seven neurologists and five neuropsychologists convened to develop recommendations for a clinical tool for neurologists and healthcare professionals working with people with MS. The purpose of the monitoring instrument would be for baseline ratings and regular follow-up assessments, which could be incorporated into routine clinical practice. It was decided that the recommended battery should be completed in 15 min and should not require any special equipment (beyond paper, pen and stopwatch). It should also be appropriate for international use (i.e. translation).

The committee critically evaluated the available cognition scales, their psychometric properties and their feasibility in the clinical setting. The group agreed that the monitoring tool should assess the domains of information processing speed and immediate verbal and visual recall, and would be sufficiently specific and sensitive to measure significant cognitive impairment in large clinical samples. The following three tests were selected.

The Symbol Digit Modalities Test (SDMT) was selected to measure information processing speed. The test consists of a number of single digits, each paired with a particular abstract symbol. The patient is presented with rows of the nine symbols that are arranged pseudo-randomly, and must say the numbers that go with each symbol in turn. The SDMT can be completed within 5 min, including the delivery of instructions and time allocated for practice and testing. The SDMT has shown superior psychometric properties. In particular, it has been reported to have high sensitivity of 82% to cognitive impairment in MS [10‐12] and cognitive change [13‐15], and moderate specificity of 60% [16], and has been validated in several countries [10‐12]. It is most closely linked to magnetic resonance imaging (MRI) parameters [17]. The SDMT also has good external clinical validity and is associated with current [18] and future employment status [15].

The California Verbal Learning Test-II (CVLT-II), which comprises five learning trials, is an examination of immediate verbal recall. In the CVLT-II, patients are read a list of words at a slightly slower rate than one item per second, and asked to recall as many items as possible in any order, across five trials. The test is a 16-item word list, with four items belonging to each of four categories, arranged randomly. The first five recall trials of the CVLT-II have a high degree of interdependence compared to other sections [19, 20], and the committee decided that they had sufficient psychometric rigour to be sensitive to MS impairment when used alone [19, 20]. Total time to administer the CVLT-II five recall trials is 5–10 min including instructions, testing and responses. The full CVLT-II also has external clinical validity and has been shown to be able to differentiate employed MS patients from patients unemployed due to MS [20].

The Brief Visuospatial Memory Test–Revised (BVMT-R) is an assessment of immediate visual recall. The first three recall trials of the BVMT-R were selected by the committee. The psychometric properties of the BVMT-R recall trials are good [21]. The BVMT-R requires the patient to observe a 2 × 3 stimulus array of abstract geometric figures. There are three learning trials, each 10 s in length. The array is hidden from view, and the patient is required to draw the geometric figures in the correct position from memory after each 10 second exposure.

The BICAMS committee subsequently published an international validation protocol [22]. A number of countries have published national validations. The aim of the present systematic review and meta-analysis is to synthesise the relevant national validation literature regarding BICAMS.

Search terms were developed and used to identify studies which were conducted as part of the BICAMS international validation protocol (Table 1). These keywords were searched within the title or abstract of the PubMed, PsycINFO, Medline and PsycARTICLES databases in January 2018.

To refine the studies which were included as part of the international validation of the BICAMS protocol, only the following eligibility criteria were applied. The inclusion criteria for the studies in the present review were as follows: (a) peer-reviewed studies with no date restriction, written in the English language; (b) samples including adults with any clinical subtype of MS (or subtype combination); (c) studies which were undertaken as part of the international validation of the BICAMS protocol. The exclusion criteria were studies which included BICAMS but were not part of the international validation of the BICAMS protocol.

The additional criteria for inclusion in the meta-analysis were as follows: (d) studies included a healthy control (HC) comparison group; (e) they reported standard quantitative information based on the SDMT, CVLT-II and BVMT-R subscales (mean, standard deviation and sample size) of the MS cases and HC comparison group; and (f) there were a minimum of four studies, as specified by Rosenthal [24], which met the above criteria to be included in a meta-analysis.

To examine the eligibility criteria, all titles and abstracts which were returned were screened. The full texts of all studies considered to meet the eligibility criteria were accessed (see Fig. 1).

A headed table was developed to guide the extraction of relevant information from full texts, and to assess their eligibility for the final review. Extraction was initially carried out by the authors (FC and DL), and studies were organised according to whether they met criteria for the systematic review, meta-analysis or both. Following data extraction, 21 studies were excluded from the final review according to the exclusion criteria. Extrinsic moderators, including participant characteristics, were extracted from the 16 short-listed studies, comprising recruitment selection, diagnostic criteria, age, and subtype of MS, disease duration and time since diagnosis, education, and score on the Expanded Disability Status Scale (EDDS) [25]. Where appropriate, the characteristics of the control group were detailed, including sample size and age. The profession of the examiner administering the BICAMS was also recorded. Fourteen studies met the criteria for the meta-analysis from those included in the systematic review. For the meta-analysis, the standard quantitative information based on the subscales of SDMT, CVLT-II and BVMT-R (mean, standard deviation and sample size) of the MS cases and HC comparison group were extracted for baseline assessments of the BICAMS. All of the relevant data for the present review and meta-analysis was obtained from numerical information in texts, tables and graphs, and statistical analysis.

The quality ratings for the studies included in the systematic review were derived from the Effective Public Health Practice Project (EPHPP) [26] quality assessment tool. This instrument assigns ratings in quantitative studies according to selection bias, study design, confounders, blinding, data collection methods, and withdrawals or drop-outs. A total quality rating can be derived from the individual ratings of the measures. The EPHPP is particularly useful for examining the quality of studies in health care settings and has previously demonstrated strong inter-rater reliability [27]. The two authors (FC and DL) independently examined the articles, and any disagreements were discussed and resolved.

The meta-analysis was conducted using the Comprehensive Meta-Analysis version 3 software program [28]. Three individual analyses were performed based on the averages of the SDMT, CVLT-II and BVMT-R subscales, which measure information processing speed and immediate verbal and visual recall, respectively. Effect sizes were calculated as standardised mean differences with Hedges’ g using the following interpretation: 0.2 = small; 0.5 = medium; 0.8 = large. Hedges’ g was selected because it offers the same interpretation as Cohen's d, while correcting for any potential biases that occur from small sample sizes, whereas Cohen's d is disposed to overestimating the absolute value of the standardised mean difference in small samples.

The effect size was modelled using a random-effects model, which estimates a mean of a distribution of effects, to examine the degree of difference between the MS group and the HC based on their performance across the former BICAMS subscales. The random-effects model assumes that the allocation of study weights is based on the inverse of the total variance, which includes both within- and between-study variance. Compared to the fixed-effect model, this model yields a wider confidence interval (CI) when there is significant heterogeneity among effect sizes. The predicted direction of results was that HC would have superior BICAMS subscale totals compared with adults with MS.

Heterogeneity was assessed using Cochran’s Q test, and the magnitude of heterogeneity using the I² statistic. A significant Q statistic indicates dissimilar effect sizes across the included studies, and methodological or sample differences might be introducing variance in the results across studies. The I² statistic assesses the proportion of heterogeneity across studies not due to random error and is interpreted as a small (25%), moderate (50%) or high (75%) level of heterogeneity [29].

Forest plots were created to visually assess for the presence of outliers. Sensitivity analyses were conducted to assess for publication bias by visually inspecting funnel plots of standardised mean differences against standard error, and then assessed using Egger’s test of funnel plot asymmetry [30] and Rosenthal’s fail-safe N [24]. If publication bias was indicated (Egger regression test: p < 0.1), then the trim-and-fill method [31] for random-effects models would be applied in order to impute “missing studies” to redress funnel plot asymmetry with adjusted pooled effect sizes and 95% CIs reported after the addition of potential missing studies.

This article is based on previously conducted studies and does not contain any studies with human participants or animals performed by any of the authors.

The results from all three subscales of BICAMS—SDMT, CVLT-II and BVMT-R—showed that adults with MS performed significantly worse than controls. To examine the overall effect size for this, a meta-analysis was conducted.

The aim of the current paper was to synthesise the national BICAMS validations to date. A total of 16 studies were included in the systematic review, of which 14 articles were then assimilated into the meta-analysis. The systematic review showed that BICAMS had been widely applied in many different languages, cultures and locations, with a diverse range of clinical samples including different disease subtypes, durations and severity. Most studies had included an HC sample with a similar educational background. BICAMS was administered in person by a neuropsychologist in most cases, although this was underreported. BICAMS was able to identify cognitive difficulties in adults with MS compared to HC, with difficulties identified in all three areas of information processing speed and immediate verbal and visual recall. Cognitive impairment was most marked in the domain of information processing speed. These findings fit with established knowledge and opinion and show that the selected scales can detect cognitive dysfunction in a 15-min battery. BICAMS also relies on previous findings regarding the component scales for its psychometric provenance.

Several strengths of this review must be considered. Countries reported that BICAMS could be feasibly administered in around 15 min, with minimal materials, and was recommended for routine clinical cognitive assessment in MS. Within the validation studies there was a wide representation of cultures, languages and locations involved in the initiative. With the success of the international validation protocol and the number of those estimated within the population [33], it is possible to generalise the findings beyond the review. The proportion of samples represented varied between 0.2 and 1.91%, with the largest representation of adults with MS recruited from Lithuania. This is a significant first step in translating an understanding of cognitive impairment in MS globally. The direction of the results was as expected on all three scales; however, of note was that the CVLT-II was the most heterogeneous. This is likely a result of the extra linguistic and cultural demands of the stimuli in the subscale.

There are a number of limitations which need to be recognised. In terms of sampling, there was a great deal of heterogeneity in sample size and MS type, ranging from disproportionally mixed subtypes to the inclusion of only one subtype. It is possible this may have biased the size of effect found, as cognitive impairment is more common in progressive forms of the illness. The HC were recruited from the community or were related or known to the adult with MS. In cases where HC were related to the adult with MS, this would pose a threat to the statistical assumption of independence between the samples examined. There was also a disparity in inclusion criteria, with variations of the McDonald criteria used [35], the control of comorbidity and medication, the details of which would have likely contributed to the observed heterogeneity. The aforementioned is likely to be in addition to the unequal gender ratio, in favour of women. It is important to consider the average age of those tested in the current review, and that the degree of cognitive impairment identified by BICAMS is likely to be negatively skewed, as the range of illness duration exceeded 6 years. The studies were cross-sectional, and baseline rather than re-test scores were included in the review; therefore, further longitudinal data is needed to understand the course of cognitive decline. Although BICAMS is reportedly able to be administered by qualified clinical health professionals, due to underreporting in the current studies, it appeared that most BICAMS were delivered by a neuropsychologist. Clinical thresholds of BICAMS scores have been proposed, but these were not recorded in the current review [38]. Few studies included accompanying neurocognitive assessments. Finally, this review provides only an interim analysis of the wider international validation of the BICAMS endeavour, since many countries have not yet published their data.