In the present study, we explored a unique dataset registered during the training for the participation to the Cybathlon BCI race competitions in 2019 and 2020. This dataset contains EEG data from more than 40 training sessions with an end-user engaged with our MI BCI system. Moreover, it gave us the opportunity to test the robustness of the user-decoder interaction reinforced by the training in the long term (1 year) without using a BCI, which represents an unprecedented opportunity in the field.
Mutual learning in BCI
As shown in Fig.
4a, the first months of training were characterized by a rapid increase of classification accuracy and confidence in discriminating between the two mental tasks, particularly in the first three months of training (May-July 2019). These results are in line with well-established evidences that the concomitant training of the two main actors (i.e., user and decoder) improves BCI performances already after a few sessions of practice, and that they positively correlate with improved application performances (Additional file
2: Fig. S2) [
34]. Even if the concept of mutual learning is becoming more and more popular in the BCI community, also thanks to events like the Cybathlon competitions, the role of user learning in this process is often overlooked. Previous studies focused mostly on the machine learning aspects, with the most common approach to the mutual learning consisting in regularly adapting the decoder parameters to deal with the intra- and inter-session variability of the user’s brain patterns [
35‐
38]. However, a more or less frequent re-calibration of the BCI system could potentially mask the effect of pure user learning on the performance [
20].
In light of these considerations, we adopted a different strategy. On the one hand, we opted for a game control paradigm which allowed the use of a simpler 2-class BCI decoder to implement the four game control commands. Indeed, it is well known that the BCI classification performance drops sharply as the number of classes to be decoded increases. Since the BCI represents a difficult task in itself, a multi-class classification approach could have been a task too challenging for the pilot, with the risk of hampering the user learning [
52]. On the other hand, we decided to update the BCI system only a few times to leave more degrees of freedom to the user. This strategy was corroborated by previous experience and by the high accuracy of our BCI system [
34], entailing several advantages. Firstly, it allowed to significantly reduce the time spent in tedious calibration runs and to train our pilot in self-modulating his brain rhythms as soon as possible in the BrainDriver game. Indeed, a closed-loop BCI usage in the final application have shown to significantly boost the training effect [
34,
38]. Secondly, a BCI system that is too adaptive to the user could have compromised the learning process of our pilot by overly relying on the system capabilities rather than on his progress [
20]. Conversely, it is worth highlighting that we last updated the decoder at the beginning of July 2019 and we made no re-calibration until October 2020. On the first day of the 2020 training period (i.e., 2020/09/15), the pilot directly used the decoder that was previously calibrated on 2019/07/09, more than 1 year earlier. This fact ensures that the visible and significant improvements of both BCI and race performances in between this period can only be traced back to modifications of user’s ability in correctly modulating his SMRs while receiving training.
The stability of BCI and application performances of our pilot (Fig.
3,
4) demonstrates that, once learned, he was capable of recalling the BCI skills accurately and consistently even after a 1-year break. However, how the brain achieves this behavioral stability is an open question. Improvements in online classification accuracy and/or application performance do not necessarily imply changes at the brain level underpinning BCI learnability [
53]. Hence, the main contribution of this work is the provision of neural correlates associated with user learning of BCI skills and their evolution over time.
User learning in channels’ domain
The first attempt to investigate the evolution of the user’s brain activity concerned the analysis of the EEG features distribution used to create the decoder (i.e., the channels’ domain) since it directly reflects the ability of the user to control the BCI. In particular, the emergence of SMRs modulations associated with the mental tasks is the most commonly used index in the BCI literature to monitor longitudinal improvements [
14,
15,
34‐
36,
54]. Except for [
34], none of these studies explicitly related the evolution of SMRs modulations to consistent and continuous changes of BCI performance. Herein, we reported a strong correlation between classification accuracy and
bcDist in
\(\beta\) band, but not in
\(\mu\) band (Additional file
3: Fig. S3a), establishing the impact of operant conditioning training in user learning. Indeed, since the system output—thus the feedback to the user—depends solely on the features selected during decoder calibration, we expected our pilot to improve the between-class distance only in the frequency band for which he received a coherent feedback during training (Fig.
4d) [
55].
Despite its straightforwardness, the sole neuroimaging-based difference of class prototypes is likely to provide only a partial picture of the neural processes occurring in the user’s brain as a consequence of training. The time-dependent variability of the brain signals may lead to within-class modifications of neural patterns, which are not directly related to maximizing class separability. We found that the brain activity of our pilot during the execution of the mental tasks significantly shifted with regular training (Fig.
6a). Remarkably, this shift interested the two classes symmetrically (Additional file
4: Fig. S4), and both the
\(\mu\) and
\(\beta\) bands (Fig.
6b), indicating a more holistic phenomenon which is not limited to class-specific discriminant features. While an increase of the within-class distance is usually seen negatively in the literature—and a lot of effort is spent in the development of machine learning algorithms to minimize it [
14,
36]—recent literature promotes the idea that it could be linked with a positive user adaptation to the BCI system [
20,
35]. The user modifies his brain activity to produce brain signal modulations that match more closely those expected by the decoder [
35]. However, the analysis of the
wcDist in the channels’ domain reported in Fig.
6a, b shows that these modifications are limited to transitory effects which occur only when the user is regularly receiving training. These results are not consistent with the continuous improvements shown by our pilot, substantiated by the lack of correlation between the
wcDist in the channels’ domain and the classification accuracy (Additional file
3: Fig. S3c). This could mean that this representation shows only short-term adjustments of the user’s BCI aptitude, rather than a stable learning effect in the long term.
User learning in neural manifold
As previously discussed and widely established in the literature, the analysis of the EEG signals in the channels’ domain is effective and convenient for the real-time classification of mental tasks, but it fails in capturing the long-term stability of the BCI skills demonstrated by our pilot. In this paper, we overcome this limitation by proposing to investigate user learning in an hyperspace—the Riemann manifold—different with respect to the input space generated by the features that are exploited in the BCI system.
From the perspective of class discrimination (Fig.
5), our results show that the between-class distance in the Riemannian geometry matched the performance obtained in the channels’ input space. Like before, a strong correlation between classification accuracy and the Riemann
bcDist in
\(\beta\)-band, but not in
\(\mu\)-band, was found (Additional file
3: Fig. S3b). This finding was not completely unexpected since the use of the signal covariance matrices as features of interest for classification of mental tasks is becoming increasingly popular in BCI applications [
56‐
59] and they have been also exploited by a team during the Cybathlon competitions [
35]. Comparably to the channels’ domain, regular training induced a sharp shift of the within-class EEG covariance matrices as our pilot was adapting to the use of the BCI system in the 2019 (Fig.
6c). Nevertheless, of greater interest is the different evolution in the Riemann domain of the
wcDist between the two frequency bands after the one year break. While the
wcDist in the
\(\mu\) band significantly decreased from 2019 to 2020, the
\(\beta\) band did not show this regression and remained stable between the two years (Fig.
6d and Additional file
5: Fig. S5). Thanks to these findings, we pinpoint the existence and definition of two type of neural modifications as a consequence of user learning: the first are short-term modifications, that spontaneously arise during training since they involve also features which are not directly targeted by the training (i.e.,
\(\mu\) band). Given these characteristics, these modifications are likely to be not robust enough to endure for a long between-session period; but they may be at the basis of the mutual-learning process and of the emergence of new SMRs features which were not considered in the previous decoders [
20,
34]. On the other hand, the unique stability of the
wcDist in the Riemann domain of the
\(\beta\) band—whose features were consistently selected and reinforced over the all training period (Table
1)—may underpin stable modifications of our pilot’s brain activity related to a long-term learning of the BCI skills which allowed him to maintain his excellent performance even without a continuous training and/or re-calibration of the decoder. In support to this statement, a strong positive correlation between classification accuracy and
\(\beta\)-band
wcDist in the Riemann domain is reported (Additional file
3: Fig. S3d), which would not have been found by limiting the analyses to the channels’ domain only.
We believe that the results of this paper open a new way of studying and analysing user learning in BCI, breaking the typical approach of searching learning correlates only in the space spanned by the neural features used as input to the decoder. A similar approach is actually commonly considered to enhance the understanding of BCI training in stroke rehabilitation: neural mechanisms underlying the clinical effects of BCI therapy are often evaluated through various markers which are not limited to a stronger desynchronized activity during MI tasks; they include also interhemispheric imbalance, functional connectivity changes [
60‐
62], and even functional and structural assessments through different neuroimaging techniques (e.g., functional near-infrared spectroscopy (fNIRS), functional magnetic resonance imaging (fMRI)) [
63,
64]. In this line, the evolution of the
wcDist in the Riemann domain may be helpful to better follow the progress of user learning of BCI skills together with conventional metrics (e.g., classification accuracy). It is interesting to note an evident discontinuity of the Riemann’s
wcDist in the runs right before our decision of re-calibrating the decoder in October 2020 (Fig.
6c). Even if further analysis would be required to support this hypothesis, an abrupt changeover of the user’s neural state in performing the mental tasks could hide a break of the user-decoder interaction, wandering in an area of the neural manifold that prevented our pilot from efficiently controlling the BCI. We thus suggest, and better investigate in future studies, that a monitoring of the proposed Riemann’s within-class distance may hint when updates of the decoder parameters are required.
Overall this study confirms the effective learnability of BCI thanks to longitudinal usage, i.e. increase of classification accuracy across the training sessions [
34‐
38]. In addition, this study shows that by focusing the training strategy on improving user learning, it is feasible to achieve a robust stabilization of BCI skills and features over a long period of time. An hypothesis that might explain these results is that the regular user-centered BCI training applied by us induced a functional reorganization of our pilot’s neural networks that are responsible for the imagination of the two motor tasks. This hypothesis is supported by previous studies [
62,
63,
65] revealing that BCI-guided rehabilitation training induces long-term neuroplasticity modulations which are kept up to 6 months after the intervention [
63]. A similar phenomenon was previously seen in invasive BCI experiments, which showed that the brain is capable of encoding a stable representation of motor-related tasks for very long periods of time (i.e., from months up to years) [
66‐
68]. However, in this work we show for the first time in a non-invasive BCI application the possibility to achieve a stability of BCI skills through learning for a time period longer than 1 year of non-use of the BCI. This feature allowed our pilot to retain his performance at the beginning of the 2020 training period without the need of any decoder re-calibration. In future work, we will perform a more in depth analysis in relation to connectivity to identify if the seen stable performance of our pilot were due to the effective existence of a functional reorganization or to the adoption of a more efficient motor imagery strategy activating existing neural networks. Nevertheless, we believe that our findings—combined with the successful results obtained in the challenging scenarios of the Cybathlon competitions—may provide a strong contribution in shifting the focus of the BCI community: not only to the machine learning of the decoder, but also in investigating novel training procedures to boost the user learning and the mutual adaptation of the user to the BCI system.
Limitations
There are certain limitations of this study that need to be mentioned. The major limitation concerns the inclusion of only one subject in the dataset which holds us back from drawing definitive conclusions. Thus, in future work we aim at recruiting a larger cohort of users in a longitudinal BCI training in order to strengthen the preliminary results on user learning reported in this paper. Nevertheless, as previously mentioned, the type of training and the limited number of re-calibrations assure us that there has certainly been a learning process in our pilot, and that the significant improvements in performance over time are mainly due to this.
The second limitation to be mentioned is that the study was conceived to be observational and uncontrolled. Still, the results obtained by our competitors may be helpful as a fair control group to highlight the effective importance of user learning in BCI. Indeed, the other teams adopted a training strategy focused on frequent re-calibrations of their decoders [
35‐
38]. Except for [
36], all the other teams reported a significant worsening of their pilots’ performances during the competitions, explained by the presence of audience or, in general, conditions of stress. It is well-known that the psychophysiological state of the user has a negative effect on BCI operation [
69‐
71]. Interestingly, our pilot was able to achieve race completion times during the competitions that were comparable or even better than his average results in the previous days. The fact that our pilot was already used to competitions surely represents an important factor accounting for these results. Nevertheless, we speculate that the stable user learning attained by our pilot, identifiable both at the behavioral and neural level, strongly contributed to the achievement of not only a high BCI accuracy, but more importantly a high reliability which represents a critical challenge in the field.