Coordination of reach-to-grasp in physical and haptic-free virtual environments

Widespread accessibility to highly specialized and advanced digital technology has made virtual reality (VR) more and more common in daily life. While VR was originally developed as an enhancement for video games and entertainment, more recent applications of virtual environments (VE) are used in a number of rapidly growing fields including virtual assembly [1], medical training [2] and rehabilitation [3‐6]. Each of these applications requires the user to manually interact with virtual objects within the immersive computer-generated environment. Depending on the particular goal, the user may need to reach and grasp to a diverse set of objects in order to manipulate their position or state.

Such reach-to-grasp movements are a standard of VE based upper-extremity (UE) neurorehabilitation [7]. Though devices exist to provide haptic feedback of object properties, these often-expensive devices [8] have numerous limitations. Namely, wearable haptic devices are heavy, leading to a slowing down of movement, and non-wearable haptic devices restrict natural motion in the environment, for review see [1, 9]. These factors, along with cost and availability, greatly limit the widespread application of VE for rehabilitation. In contrast, motion tracking has a broad spectrum of commercially available products that support mobility at low cost [10], therefore making haptic-free VE (hf-VE) a more accessible option. Key to its use in rehabilitation, VE provides the opportunity for adaptive and progressive motor learning through perceptual modifications such as activity and workplace scaling, and error augmentation, resulting in greater dosage, intensity and engagement in training [11, 12]. While this capability makes VE a powerful technology for rehabilitation, it is still unclear to what extent movements produced in hf-VE resemble those produced in the physical environment (PE).

A comprehensive characterization of the kinematic differences between reach-to-grasp movements produced in the two environments will provide important information regarding the feasibility of hf-VE for UE rehabilitation. Therefore, in this investigation we sought to systematically describe the differences between environments by characterizing phase specific differences the reach and grasp components of reach-to-grasp.

To date, there have been few investigations of reach-to-grasp movements in VE. Magdalon and collaborators [13] reported that subjects instructed to reach and grasp three types of virtual and physical objects (a can, a screwdriver and a pen) reached significantly slower in VE, with a relative time to peak velocity (rTPV) around 10% earlier and a longer relative deceleration time. Though aperture scaling was preserved in VE, peak aperture (PA) was wider in VE when grasping the smaller objects (screwdriver and pen). In a follow-up experiment by the same group [14], several prehension measures were compared across VE and PE in individuals who have had stroke. They concluded that participants used similar movement strategies, and subjects were able to scale aperture to object size but with longer delays between time to peak transport velocity (TPV) and PA. It is important to note that all of the above studies comparing movement kinematics in PE versus VE provided some sort of haptic feedback. Given the dearth of evidence about how reach-to-grasp is coordinated in VE without haptics, we set out to compare performance in hf-VE and PE.

It is generally agreed that the transport and the grasp, though relatively independently controlled, also show evidence of temporal coordination [14‐20]. Some of this evidence includes a TPA at 60–70% of transport time as well as a PV during the first half of the movement [16, 21]. Levin et al. [14] tested the effects of VE with haptic feedback on the kinematics of reach-to-grasp movements. In order to quantify the coordination of the reach and grasp, the authors expressed temporal coordination as the delay between the times of arm peak velocity and hand maximal aperture. Their results showed that there were longer delays between the landmarks in VE with respect to movements in PE, suggesting a possible loss of coordination. Given the tight relationship between the transport and grasp in natural movements, our second goal was to determine if the kinematics of the individual components, as well as the coordination between the components, is preserved when performing under visual feedback alone (haptic-free) reach-to-grasp movements in VE.

We hypothesized that wider grip apertures and longer movement times would be observed in hf-VE (hypothesis 1). Moreover, we also hypothesized that coordination between the transport and grasp components would be preserved across environments (hypothesis 2). To test these hypotheses, we asked healthy subjects to produce reach-to grasp movements to size- position-matched physical and virtual objects of different sizes (manipulating the grasp component) and distances (manipulating the transport component) in both PE and VE.

Subjects sat comfortably in front of a table. They placed their dominant (right) hand on the table in a comfortable pinch position lightly holding a wooden peg which maintained a starting distance between thumb and index of 1.5 cm to prevent interference between the infrared markers. The position of the peg was 24 cm to the right, and 12 cm in front, of the body midline. The thumb was placed on the start switch next to the peg. Each trial began with an auditory signal, cuing the subject to reach, grasp, and lift the virtual or physical object (see Fig. 1). Subjects were instructed to reach naturally toward the object with the dominant upper limb, without leaning their trunk forward, at their preferred speed keeping their hand relatively parallel to the table to reduce vertical movements. The subjects’ view of their fingertip position in the VE condition was projected as two green spheres (0.8 cm, diameter) rendered at the 3D position of the fingertip IRED markers. To grasp the object, subjects placed the fingertip spheres of the thumb and index finger (pincer grip) approximately on the middle of each object’s lateral surface (see Fig. 1, panel b). Contact with the virtual object was controlled by a collision detection algorithm which colored the virtual object red when the fingertip spheres reached the location of the object (to make virtual objects easy to grasp the total collision error margin was set to 1.2 cm). As the object was lifted from the table, the subject kept it in the vertical position before putting the object back down (in VR, the object was held above the table until it disappeared) and returning their hand to the starting position. In PE, subjects completed the reach-to-grasp task in a dark room, seeing only the glow-in-the-dark object and the illuminated IRED markers on their fingertips (see Additional file 1: Video 1). Overhead lights were turned on between trials to prevent dark adaptation. Therefore, visual information remained similar across the environments.

Before data collection, subjects were familiarized with the setup and procedure, particularly to the VE environment. Familiarization consisted of 135 trials in VE (15 trials × 3 object sizes and × 3 distances). Five second ‘rest’ periods were provided after every 15 trials.

The experiment consisted of 216 trials (108 trials in the PE block, 108 trials in the VE block). Three seconds were allowed for each trial. After every 12 trials, a 5-s ‘rest’ period was provided. A 5-min ‘rest’ was provided between blocks.

The main goal of this study was to determine the potential of current VR systems as a platform for research and rehabilitation [4, 11, 24]. In order to assess the characteristics of movements produced in VR, we tested healthy participants in a comparable reach-to-grasp task in both the PE and the VE. Importantly, we decided not to include haptic feedback, as previous studies have, due to the relatively lower state of development of that technology which results in reduced accessibility and practicality [1, 8, 9].

While most of the previous work involving VE has provided haptic feedback to the participants [13, 14, 25], we were interested in determining whether visual feedback alone would support the production of properly coordinated reach-to-grasp movements. The characteristics of prehension movements observed in our study showed that even when haptic information is absent during reach-to-grasp actions in VE, the behavior across environments showed very similar kinematic patterns. High correlations between aperture (r = 0.95 ± 0.05) and transport velocity profiles (r = 0.97 ± 0.03, see Fig. 2), as well as similar trajectories of individual markers (index finger, thumb and wrist, see Fig. 3) support this notion. Similarly, as expected, grasp aperture was scaled to object size independently of the environment where the movement was carried out. These results coincide with previous studies comparing movement kinematics in PE and VE [13, 14, 26].

Many studies have postulated that in order to grasp any object successfully, the transport and grasp components must be coordinated [17, 18, 27, 28]. Moreover, the temporal relationship between the components appears to depend on task goal, object properties, and experience [18]. In our setup we observed that even if there were some differences in movement kinematics, coordination between the reach and grasp components was preserved across environments (see Fig. 6). Similarly, though the typical landmarks of the movement were slightly different across environments, in VE they still occurred within normal ranges: TPA occurred between 60 and 70%, and TPV between 30 and 40% of movement time [16, 17, 19]. We suggest that coordination across environments is probably a more relevant measure of whether VE is useful for research and rehabilitation than any specific landmark. The fact that people produce natural movements (they do not move ‘robotically’) suggests that they are selecting and deploying the appropriate movements at relatively consistent latencies.

Having said that, the differences we observed between environments as the participants deployed their grasping movements are important to understand for the possible improvement of this technology. For example, MT exhibited differences between PE and VE, as did hdPA and snPA. Some of these results were not unexpected, as previous studies have found that in VE individuals move slower, particularly showing longer deceleration times both during reaching and grasping as well as in pointing tasks [13, 14, 25, 26, 29]. Similarly, the increase in peak aperture we observed in VE for small-sized objects, was previously reported by [13]. Larger grip aperture in VE (especially during grasps to smaller objects) have been reported in previous studies comparing physical and virtual environments [13, 14]. The effects described suggest to us there was increased perceptual uncertainty during target acquisition in VE. Therefore, a strategy to increase the safety margin in terms of visibility of both the target and the hand during the movement is needed [30]. Given the geometry of our setup, where the marker (avatar) representing the tip of the index finger is eclipsed by the distant edge of the virtual object near the time of peak aperture, it is possible that participants required more time and larger distance between the digits compared to PE, when acquiring the target object. This effect was evident when the large virtual object was reached and grasped. The analysis of the timing of the reach-to-grasp phases confirmed that subjects needed noticeably more time during the closure phase in VE (see Fig. 7).

The demarcations of the reach-to-grasp phases (shown in Fig. 7) could be useful for researchers and clinicians as a means of evaluating the proportion of total movement time spent in each phase of the reach-to-grasp movement. This analysis differentiates between a mostly internally guided - initiation phase, an execution phase when sensory feedback of object state is obtained, and a closure phase when handshape is adjusted. Based on the proportion of reach-to-grasp phases, the programming and evaluation of functional recovery process could be better understood and tailored to individual patient needs.

A related question involves the participants’ behavioral response to the collision detection algorithm employed in our study. The collision detection algorithm determines whether the limits of a virtual object collide with those of the thumb and index finger avatars [31‐33]. Depending on the dimensions of the overlap allowed by the algorithm, an object may be too hard or too easy to ‘contact’. Therefore, this algorithm must be carefully calibrated in order to adjust collision detection to the participant’s ability. In this experiment, the virtual objects changed color once collision with the avatars was detected, allowing the subject to grasp and lift the object [34]. Therefore, as participants gauged the appropriate digit location to trigger the grasp they seemed to have taken longer to close their digits. This effect was observed as a relatively early beginning of aperture closing in VE (hand distance to object at peak aperture). We believe that it is possible to control a large portion of such uncertainty by modifying the collision detection algorithm and we are currently testing this possibility.

Finally, it is important to discuss the possibility that adding haptic feedback to virtual reality environments may result in a reduction of the perceptual uncertainty our participants encountered [35]. It is possible that, in the absence of haptic feedback, grasping an object became an aperture matching task under visual guidance, which might be a limitation of the study. While that may be the case, our participants were able to produce natural goal-directed movements within hf-VE.

Furthermore, lack of differences in the vertical distance between the digits at object grasp indicates that participants treated the virtual object as if it had physical properties. Placement of the digits at object contact in order to prevent roll of the object during grasp and lift has been previous shown to indicate anticipatory force control [36]. In the VE utilized here, nothing prevented the participants from grasping the virtual object with misalignment of the thumb and index, which would cause a rotation of a real object. However, this was not observed and, on average, the vertical distance between the thumb and index was in fact smaller in VE than PE. Cumulatively these results suggesting that visual feedback in current high-end virtual reality systems may be sufficient for research and rehabilitation purposes if tasks are carefully designed and calibrated.

We found that prehension movements are largely similarly coordinated in both physical and our 3D-immersed virtual environments. That is, the intrinsic plans for reach-to-grasp in hf-VE did not change movement structure nor the motor programs deployed. Nonetheless, we observed that intrinsic motor plans in VE may be tuned (scaled), particularly in the closure phase, perhaps due to perceptual uncertainty and/or due to lack of haptic feedback. Therefore, we suggest that if the collision detection algorithm is properly controlled, VR may be suitable for accurate and precise reach-to-grasp movements since the virtual environment presents appropriate visuo-motor scaling for reach-to-grasp coordinated action towards different object sizes and distances. This property, as well as the relatively easy manipulation of visual feedback, leverages VR as a flexible tool for research and rehabilitation.