Development of a 3D, networked multi-user virtual reality environment for home therapy after stroke

At its core, VERGE consists of a 3D VRE in which avatars interact with virtual objects. To date, we have created two such VREs, one depicting a dining room and the other a kitchen. The scenes were created in Maya (Autodesk Inc., San Rafael, CA) and imported into Unity 3D (Unity 4.5, Unity Technologies, San Francisco, CA), the software platform controlling VERGE. The VREs are rich in detail in order to provide depth cues [40]. Thus, depth can be conveyed without the need for stereovision, such as that provided by head mounted displays (HMDs). We have found that HMDs can be difficult for stroke survivors to use due to the limited field-of-view and, especially, involuntary coupling between neck and arm motion [41, 42]. The latter may lead to complications with moving the arm while keeping the head steady.

The avatars were created from a custom skeleton in Maya (Autodesk Inc., San Rafael, CA), which was rigged to an existing mesh of the “casual young man” 3D model, purchased and modified for our project (Fig. 1). We created the custom skeleton to match the topology of the existing character while corresponding to the skeletal joint naming convention in Unity 3D. The skeleton (and thus avatar) is animated according to joint angle data captured with a Kinect™ I optical tracker (Microsoft Corp., Redmont, WA). The 3D motion data from the Kinect™ are transmitted to the Unity code through UDP to drive the movement of the avatar in the virtual environment.

The VERGE system employs a central server interacting with peripheral client computers, one for each user. The server receives information from the client computers and controls updating of the scene so that the appropriate view of the scene is shown on each client computer through TCP/IP network architecture. We implemented communication between the client computers through custom libraries in C# (Microsoft Visual Studio). We used two multi-user network models: an authoritative server (server performs all physics calculations) for when more than one user could interact with virtual objects simultaneously and a non-authoritative server (local computation of physics) for when only one user at a time could interact with the virtual environment.

We have created three exercises (Ball Bump, Food Fight, and Trajectory Trace) employing Unity 3D and C#. These exercise were designed to encourage upper extremity movement, particularly to areas of the workspace that are often difficult to reach, such as those requiring raising the arm and reaching away from the body [43]. For each exercise, VERGE provides a first-person view of the virtual scene, as through the eyes of the avatar, to each user in accordance with previous studies [44]. This helps to establish a sense of presence for the user in the scene. The server displays a third-person view of the VRE.

Ball Bump is played on the table of the dining room VRE created in Maya (Fig. 2a). The goal is to hit a ball back and forth across the table, while avoiding the objects on the table. Contact between the ball and the avatar hand produces a collision that redirects the ball according to the Unity physics engine. Similarly, collisions between the ball and other objects redirect the ball. The ball will fall off the table if a fellow participant misses making contact with it and /or if the user hits the ball in the wrong direction. Pressing a red napkin, located to the side, produces a new ball. Thus, participants are encouraged to reach away from their bodies, especially to contact the napkin or to free the ball when it becomes stuck behind an object. This can be a collaborative exercise, in which the participants try to make as many successful passes as possible before the ball falls off the table, or a competitive game in which each player tries to hit the ball past the other player. Prior studies have shown that different users will have different preferences for competitive or collaborative exercises [30].

The Food Fight game takes place in the custom kitchen VRE created in Maya. Participants grab different food items and throw them at other avatars (Fig. 2b). The user “grasps” an object by placing the avatar’s hand in close proximity and clicking a button on a wireless optical pen mouse (2.4GHz Wireless Optical Pen Mouse Adjustable 500/1000DPI, Docooler) with either hand (stroke survivors typically operate the pen mouse with the ipsilesional hand). The user releases the object by operating another button on the pen mouse; the object’s trajectory is determined by the object’s velocity vector at the time of release. Once all of the food items have been thrown (e.g., cake, eggs, and fruit), the user can reset the food items by clicking on the reset button to continue play. We integrated mesh deformations and special effects for food items colliding with other objects. Eggs splatter and appear as yolks, the cake morphs into a pile of crumbs, and the pears and apples deform into broken fruit. Scenes are updated by the central server in authoritative mode.

In the Trajectory Trace exercise (Fig. 2c) one participant draws a 3D trajectory in the air. This trajectory is then passed to another participant who attempts to erase it by retracing. The state of the game (Draw, Claim, Trace, or Reset), as well as the initiation and termination of drawing the curve, is controlled by touching a button (located on the avatar’s chest) with the less affected hand. The trajectory is anchored to the avatar, such that the user must reach with the arm rather than using the trunk to move the hand to the trajectory, similar to a previous study [45]. The partner (such as a therapist) can specify to which part of the workspace the other player (e.g., a stroke survivor) should practice reaching, by drawing the trajectory in that region. To help with depth perception, a translucent 3D cube outlines the boundaries of the drawn 3D shape. Since the trajectory must be retraced in the same direction in which it was created, we display a yellow sphere to indicate the current starting point. As the user successfully traces the trajectory and it disappears, the 3D depth cube adjusts to match the volume of the remaining trajectory in real time.

Fifteen stroke survivors (10 male/ 5 female) in the chronic stage of recovery participated in this study. Subjects were at least 2 years post-stroke (mean of 17.4 years) and ranged in age from 33 to 81 years. Subjects had upper extremity hemiparesis, rated as Stage 3 - Stage 5 on the Stage of Arm subsection of the Chedoke-McMaster Stroke Assessment Scale [46] by a trained therapist. Subjects had no known orthopedic disease, significant visual deficits, contracture or pain (self-reported pain less than 6 on a 10-point scale) in the arm that would have hampered performing the experiments. Northwestern University’s Institutional Review Board (Chicago, IL) approved the study design and each participant signed an informed consent before study enrollment.

Each participant took part in a three-week intervention study consisting of 9 one-hour training sessions in a laboratory setting over 3 weeks. During each week, the participant was involved in one of three therapy modalities 1) VERGE, 2) HEP or 3) AWVR. The order of the therapy was randomized for each participant, and all participants took part in all three therapies.

The VERGE therapy utilized the three different exercises previously described. Ball Bump was always the first exercise introduced to the participant as it was deemed the simplest to understand, and thus provided a straightforward means for learning to control the avatar in the VRE. Trajectory Trace and Food Fight were then played, in that order. All exercises for the VERGE therapy were performed with the arm unsupported. Participants performed each exercise for 15 min, with a 5 min resting period between games. For the purposes of this study, the other user was always a member of the study team with experience performing the virtual exercises, in a situation akin to the expert user employed in an aforementioned study [29]. This individual was located in a different room within the same building to simulate home use.

The HEP therapy consisted of pre-defined sets of seated, self-paced arm-hand exercises derived from standard care. These 16 exercises were presented to the participant in the form of a printed handout (see Additional file 1). The HEP consisted of a generalized list of tasks for the specific purposes of this research study, and not one individualized to the needs of a particular patient. In accordance with the other modalities, we instructed participants to work through the handout in three blocks of 15 min of activity followed by 5 min of rest for each training session. Participants performed all tasks while seated at a table. Roughly 30% of the exercises involved arm support (provided by the tabletop) while 70% were unsupported. To limit the potential for participants to perform extra HEP at home, no copies of the HEP protocol were permitted to leave the research facility.

The AWVR is a rich VRE on which we have trained stroke survivors in the past [39]. The environment draws the user into the March Hare’s tea party where the participant is guided to perform a number of tasks to spur repetitive practice of movements. We chose three of the available exercises, based on their appropriateness for encouraging arm movement. One exercise, Tea Stir, entailed picking up a spoon from a jar and then reaching forward to stir tea in a teacup; the arm remained suspended, unsupported until the spoon “melted” and disappeared (Fig. 3a). This exercise involved crossing the midline of the body and extending the arm laterally. For the Crabby Cookies exercise, the participant reached out to a plate of virtual cookies. Once a cookie was touched, it transformed into a crab that scurried across the table and had to be “caught” by touching it (Fig. 3b). Thus, the Crabby Cookies exercise required multiple reaches away from the body. The third exercise, Bottomless Sherry, required the participant to reach for a glass of sherry, raise it to take a sip, and then set it back on the table to be refilled (Fig. 3c). We instructed participants to place the glass in different locations on the surface of the table to encourage reaching to her range of motion limits. All exercises were performed with the arm unsupported. Participants performed each exercise for 15 min, followed by a 5 min break, over the one-hour session. As the complexity was similar across exercises, order of play was random and often chosen by the participant.

During the first therapy session for each therapy modality, study staff explained the exercises and demonstrated performance. After that, study staff were available throughout each session to answer questions and monitor for safety, but each participant was encouraged to work independently, as if at home. As noted for VERGE, the participant worked together with another member of the study staff, but that individual was located in a separate room.

As we were especially interested in user response to VERGE, we administered a questionnaire (VERGE Survey) employing a 5-point Likert scale [47, 48] at the end of the VERGE training week (see Additional file 2). This questionnaire addressed the different exercises and issues specific to VERGE. Participants completed a different questionnaire (Weekly Survey) at the end of each therapy week (including VERGE) to measure the level of engagement in and the perceived potential for the therapy (See Additional file 3). We administered a final questionnaire (Final Survey), directly comparing the three therapies, at the end of the study (See Additional file 4). Additionally, we captured participant kinematics throughout the third session for each therapy modality with the Xsens 3D motion tracker system (MVN, Xsens, Culver City, CA), (Fig. 4). Participants donned the Xsens vest, containing eight inertial measurement units (IMUs) and a headband containing one IMU (per upper extremity configuration). The Xsens system continuously recorded upper extremity movement from the IMU data during all of the exercises.

Descriptive statistics were utilized to compare responses from the VERGE Survey. For the Weekly Surveys, we used the nonparametric Friedman test to compare participant responses, as measured by the Likert scores, across the treatment modalities. Hand and shoulder displacements were computed from the Xsens data by using the biomechanical toolkit Mokka [49]. Total shoulder movement was subtracted from total hand movement to account for arm displacement resulting purely from trunk translation. We performed non-inferiority tests to examine whether total arm displacement during VERGE was inferior to arm displacement during the other two modalities. The δ was set equal to 10% of the larger of the hand displacement means for AWVR and HEP. Furthermore, we examined which parts of the workspace were accessed. For example, Reach Distance was calculated as the relative distance, normalized by arm length, of the hand away from the shoulder. Thus, a Reach Distance = 0 indicated that the hand was coincident with the shoulder, while a Reach Distance = 1 indicated full arm extension. We defined Hand Elevation as the vertical location of the hand with respect to the shoulder. A value of 0 represented the hand at shoulder height, while a value of 1 indicated that the hand was located at its lowest possible position with respect to the shoulder (i.e., hand at the side with elbow fully extended). We performed non-inferiority tests to examine whether time spent with a Reach Distance ≥ 0.7 and whether the time spent with Hand Elevation ≤ 0.4 were inferior for VERGE than for the other two modalities. The δ was set to 4.5 min, equal to 10% of the total training time. We also created histograms to show the amount of time the hand was positioned at different bins of Reach Distance and Hand Elevation.