The purpose of this study was to examine whether cognitive load increases in HMD-VR during visuomotor adaptation compared to a conventional computer screen (CS) environment, and whether increased cognitive load relates to long-term retention and context transfer. This was the first study to our knowledge that compared cognitive load in HMD-VR with known motor learning mechanisms and examined the relationship between cognitive load in HMD-VR with long-term motor memory formation. We found four main results. First, we showed that cognitive load is greater in HMD-VR compared to CS across adaptation. Second, we showed that higher cognitive load is related to decreased explicit, cognitive mechanisms, specifically early in adaptation. Third, we showed that visuomotor adaptation in HMD-VR leads to decreased long-term retention and context transfer, which appears to be due to greater forgetting of explicit processes. Fourth, we showed that increased cognitive load is related to decreased long-term motor memory formation. These findings have important implications for the development of clinical and motor learning applications in HMD-VR.
Cognitive load during visuomotor adaptation is greater in HMD-VR than CS and related to decreased explicit processes early in adaptation
HMD-VR has been shown to increase cognitive load while performing complex motor skill tasks [
16]. Here, we show that HMD-VR also increases cognitive load during a specific type of motor learning (i.e., visuomotor adaptation). Visuomotor adaptation is thought to be driven by explicit and implicit mechanisms. Explicit mechanisms are important early in adaptation and are thought to rely more on cognitive brain areas such as the dorsolateral prefrontal and premotor cortices [
34‐
36]. Implicit mechanisms on the other hand develop over the course of adaptation and are the reflection of new visuomotor mappings driven by the anterior-medial cerebellum [
34,
37]. These mechanisms are thought to work together in order to drive overall adaptation.
In this study, we found that early in adaptation, greater cognitive load was related to decreased explicit processes and that explicit processes—and subsequently, overall adaptation performance—were lower in HMD-VR than in CS. One interpretation of these findings is that greater cognitive load limits the use of explicit processes at the time when they are the primary drivers of overall adaptation. Put another way, increased cognitive load in HMD-VR limits the engagement of explicit processes specifically when they were most important for adaptation (i.e., early in adaptation). If this interpretation is true, then cognitive load may have the strongest affects early in the motor learning process. Motor learning in the real world has been shown to facilitate subsequent motor learning processes in HMD-VR, suggesting that HMD-VR may be more effectively used in later stages of motor learning [
38]. Thus, initial training done without the use of HMD-VR may then increase the effectiveness of HMD-VR applications.
Another interpretation of these findings is that the engagement of explicit processes is limited at times when cognitive load is beyond working memory limits. We found that, while cognitive load was greater in HMD-VR than in CS across adaptation, overall cognitive load decreased over the course of training. Therefore, cognitive load may have limited the cognitive resources dedicated to the visuomotor adaptation task when it was most needed, early in adaptation. If this interpretation is true, then cognitive load may affect the motor learning process whenever the load on working memory is beyond working memory limits. HMD-VR applications may need to be designed to decrease cognitive load throughout training, or training in HMD-VR may need to be extended to reduce the early effects of cognitive load. Future work should systematically test these two hypotheses as both interpretations would affect how HMD-VR is effectively designed and used for motor learning applications.
Furthermore, recent evidence found suggests that a history of task errors, as opposed to a history of sensory prediction errors, is necessary for encoding memories [
39]. In sensorimotor adaptation, explicit processes are thought to be driven by task errors and implicit processes are thought to be driven by sensory prediction errors [
40]. Therefore, a decreased use of explicit processes early in adaptation in HMD-VR suggests that task errors were less relied on to update overall adaptation during this time. This could potentially explain why greater cognitive load related to decreased explicit processes could then result in decreased long-term retention.
Here we found that early in adaptation, the CS group had greater explicit learning and better overall adaptation compared to the HMD-VR group. These results are different from our findings in Anglin et al., 2017, where we found that compared to HMD-VR, CS had less explicit learning and similar overall adaptation over the course of adaptation, including early in adaptation. Explanations for these differences could be because of inter-subject variability, or because of differences in experimental design with the addition of the dual-task probe. One point in favor of the differences being due to different experimental designs is the differences in the response time in each experiment. That is, because of the dual-task probe, participants needed to wait after the initial presentation of the target before making their reach, increasing their response time to reach for the target. In a recent study by Langsdorf et al., 2021, it was found that forcing a wait period increased explicit learning compared to free reaching [
41]. This finding is consistent with what can be found when comparing the CS groups between the present study and the study in Anglin et al., 2017. It is unclear why the HMD-VR group would not also experience an increase in explicit learning proportionately; however, this may be due to a ceiling effect given that the explicit processes are already increased due the HMD-VR environment.
Similar to our findings in Anglin et al., 2017, here we again found that at the end of adaptation, performance was the same whether training in HMD-VR or in CS, but the mechanisms driving performance were different between environments. Specifically, at the end of adaptation, the HMD-VR group showed a greater reliance on explicit mechanisms while the CS group showed a greater reliance on implicit mechanisms, although the net performance was the same across groups. Our interpretation of these results is that adaptation in HMD-VR relies more on explicit, cognitive strategies. If HMD-VR does rely more on explicit processes than implicit processes, then this can potentially explain why the performance was lower in HMD-VR than in CS early in adaptation, when explicit processes might have been affected by increased cognitive load.
One potential explanation for why cognitive load may be higher in HMD-VR could be how the brain processes vision for action. Vision for action is typically processed through the dorsal mode of control; however, artificial presentations of depth information in HMD-VR may cause a shift from a dorsal to ventral mode of control [
42,
43]. A ventral mode of control is thought to be dependent on visual perception and increased cognitive processes and therefore could potentially explain increased cognitive load in HMD-VR during motor learning [
44]. Separately, depth information has been found to uniquely affect explicit processes and therefore could also explain why HMD-VR may rely more on explicit, cognitive strategies [
45]. Further research is needed to examine whether HMD-VR relies more on a ventral mode of control and whether this shift in control could explain a greater reliance of explicit processes in HMD-VR.
Visuomotor adaptation in HMD-VR leads to decreased long-term retention and context transfer
In this study, we found that training in HMD-VR resulted in decreased long-term retention. Importantly, a decrease in retention occurred whether participants remained in an HMD-VR environment or transferred to a new context. Although the context transfer results reported here are relatively weak, this was likely due to large variability observed at delayed 24-h forgetting (Fig.
5B). Both findings from long-term retention and context transfer suggests that training in HMD-VR may lead to less efficient motor memory formation. That is, retention following training in HMD-VR cannot be explained by a context interference effect (i.e., better retention in the same environment as training), but is rather best explained by the training process itself.
Converging evidence suggests that explicit and implicit processes are homologous with the fast and slow processes of a dual-state model of sensorimotor learning [
46]. The fast process generally dominates early in adaptation, responding strongly to error but exhibiting fast forgetting, while the slow process increases gradually, becoming stable over time and contributing to motor memory formation [
47,
48]. Importantly, the slow process is thought to predict long-term retention, suggesting that implicit adaptation may also predict long-term retention. Implicit adaptation at the end of training was lower in HMD-VR than in CS and could potentially explain the decreased long-term retention and context transfer. We also found that long-term retention was related to greater forgetting of explicit processes. Taken together, these findings suggest that an increased reliance of explicit processes in visuomotor adaptation may lead to less efficient motor memory formation, explained by fast forgetting of explicit processes.
Given that in the present study a dual-task probe was combined with a typical visuomotor adaptation task in order to examine cognitive load, an important question is whether this modification might have altered the motor adaptation process. While this is a possibility, we would expect to see similar results in long-term retention and context transfer independent of the inclusion of the dual-task probe. This expectation is based on previous findings that the magnitude of implicit learning at the end of adaptation can be used as a predictor of retention (e.g., Joiner & Smith, 2008; Schweighofer et al. 2011). Both in the present study and in Anglin et al., 2017 which did not use a dual-task probe, the implicit learning at the end of adaptation was lower in HMD-VR than in CS. Therefore, because of this similarity we believe that the long-term retention and context transfer results in the present study were not meaningfully altered because of the dual-task probe modification.
Limitations
A limitation of this study was the use of a computer screen to measure context transfer from HMD-VR. Using a computer screen allowed for a well-controlled study design as the only difference between HMD-VR and CS environments was the head-mounted display, which allowed us to control for any transfer effects that may have occurred due to a task change. However, future studies should examine whether increased cognitive load in HMD-VR during motor learning affects transfer to more dynamic, interactive real-world tasks such as manipulating physical objects, like a cup or a ball. Additionally, while visuomotor adaption is a specific type of motor learning, over-generalization to the domain of motor skill learning may not always be applicable [
21]. Future studies should look to see if an increased cognitive load in HMD-VR during motor skill learning also affects long-term retention. Similarly, while the use of verbal reporting is a common way to measure explicit learning, this method has been shown to result in more explicit learning than other methods (e.g., exclusion) [
49]. Future studies should examine whether the use of the exclusion method would reduce explicit learning in visuomotor adaptation in HMD-VR.