Development of the Home based Virtual Rehabilitation System (HoVRS) to remotely deliver an intense and customized upper extremity training

The LMC consists of three infrared LEDs and two cameras. The functional range of the LMC extends from approximately 25 to 600 mm above the device (1 in. to 2 feet). A data validation study showed the LMC to be accurate and reliable when the target is within its visual area (± 250 mm of the LMC center) [20]. The device’s USB controller reads the sensor data into its own local memory and performs any necessary resolution adjustments. This data is then streamed via USB to the Leap Motion Image Application Programming Interface (API). Using Unity, we programmed the system to feed tracking data into virtual reality activities by calling the Leap Motion API.

Some individuals with severe proximal arm impairment have difficulty maintaining their hand above the optimal LMC capture space for the duration of gameplay. An arm support (Fig. 1b) was supplied to these individuals at the initial visit to provide anti-gravity arm support to assist with positioning their hand above the LMC. This also allows patients that are unable to move their UE through space the ability to transport their hand in the LMC’s workspace. The arm support was positioned at the proximal aspect of the forearm. Therefore, wrist flexion/extension and forearm pronation/supination movements were not obstructed. Several patients demonstrated difficulties producing wrist and forearm movements when moving their upper extremities against gravity without assistance. The addition of the arm support facilitated distal movements in these subjects. Arm supports were mounted on the table where the LMC and laptop were located. If a subject’s living environment did not have the appropriate surface to attach the arm support, we provided an LMC stand to mount the arm support and LMC. The LMC stand was made in-house and was constructed of metal with a heavy and stable base. Another accommodation we provided to enable optimal hand positioning above the LMC was a hip wedge (Fig. 1c), which was secured between a subject’s legs. This was especially helpful for those with some active elbow stability as well as active shoulder rotation movement. The LMC was attached with Velcro on the top of the hip wedge and the subject’s hand was easily placed above the LMC and within its effective range. A third option was a forearm trough. This was used for subjects who only had distal finger movement, allowing them to interact with the system for hand and wrist training. With their arm supported by the trough, subjects were able to focus on finger movement without using the arm and shoulder to stabilize their hand. These accommodations demonstrate the flexibility of the system with regards to physical space and impairment level of the individual.

Rehabilitation games can either be installed by engineers during the initial system setup or downloaded from the HoVRS website at the instruction of an individual’s therapist. After a preliminary configuration session that involves system calibration, subjects start the system for subsequent sessions by choosing a game from a Graphic User Interface (GUI) with a single mouse-click. Games’ initial levels are carried forward from their previous training sessions, eliminating the need for calibration at the start of each session.

Currently, 12 games have been developed which can be grouped into one of four categories: Elbow-Shoulder, Wrist, Hand and Whole Arm (see game descriptions in Additional file 1). In summary, each game trains a specific movement pattern, such as wrist pronation or finger fractionation. The finger games include Car, Bowling, Finger Flying, and Piano. These games utilize the range of whole hand finger flexion/extension calibrations. With the exception of the Piano and Fruit Picking games, these games encourage hand opening to control the speed of movement of a virtual object in order to reach targets and avoid obstacles. The Piano game encourages finger individuation integrated with reaching. Wrist games include Whack a Mole and Wrist Flying and utilize the range of pronation/supination, range of radial/ulnar deviation or range of wrist extension/flexion calibrations. These games encourage the player to practice pronation/supination and extension/flexion in order to successfully catch or hit targets. The third category is shoulder-elbow games which include the Maze, Brick Break, Arm Flying, and Soccer Goalie games. These games require calibrations of vertical and horizontal arm range and use these arm positions to move the object or character in a game. The fourth category is whole arm games which includes Fruit Picking and Fruit Catching. Games in all categories adjust movement of the virtual objects to the range of player movement, calculated based on their calibrations. They all include either multiple levels that increase in difficulty or a single level that adjusts dynamically by algorithms. There is a configuration window that clinicians can use to set up game conditions for variables such as workspace size, activity speed, accuracy demands, etc. Clinicians can also track patient’s performance such as duration of gameplay and game score daily, weekly, and monthly.

HoVRS streams kinematic data of the hand and wrist for 22 degrees of freedom in addition to hand orientation and position. These include hand palm position, palm orientation, wrist position, three joint positions from each finger, (metacarpophalangeal joint, proximal interphalangeal joint, distal interphalangeal joint) and fingertip coordinates. These data are used to control the game progression with the help of various online algorithms. There is a target movement that is shaped by each game, for example, the Flying game shapes either finger extension, wrist extension, or shoulder elevation, depending on the specific version of the game. One of the main objectives of these algorithms is to maintain the difficulty levels for each of the games within a prescribed range [21]. Figure 2 shows an example of such an algorithm for the Piano game. The subject was required to flex his active finger, which was randomly assigned by the game, to press the piano key while keeping the other non-active fingers straight. Actual fractionation was calculated as the metacarpal joint (MCP) flexion angle difference between the active finger (solid black line in Fig. 2) and the most flexed non-active finger (dotted line) in real time. A piano key was successfully pressed (green line) when the target fractionation reached the pre-set target fractionation. The algorithm running in the background tracks subject’s successful key press rate. The algorithm increases target individuation if success rate is higher than 80% and decreases it if the rate is lower than 80%. By adjusting the target individuation, the subject is prompted to flex his active finger more if he/she is capable while not frustrating him when he/she is tired.

The following computer-based tests of hand kinematics were performed along with clinical tests to monitor adaptations to the games played during the pilot study.

In addition to the measures described above, the system is capable of collecting other motions, such as radial and ulnar deviation, that we have not yet piloted with stroke subjects. Additional metrics such as speed of hand opening, finger individuation, and smoothness of spatial trajectory of the hand, are available as well making the system capable of monitoring adaptations to the entire library of games offered and customizable to a wide variety of clinical conditions.

Subjects had no prior exposure to the system before their initial home visit. A Physical Therapist and an Engineer conducted the initial visits. The team evaluated the subject’s home for adequacy of space and furniture for interacting with the system. For subjects requiring antigravity support, the appropriate hardware as described above was provided (see “Hardware" under "Design" section). Finally, we established that the subject’s internet access was sufficient to administer telerehabilitation, transmit and receive daily game data via our Amazon Web Service (AWS) server, and provide later updates to the system.

Once the system was set up in their home, we showed them how to start the games and informed subjects to keep their affected hand at least 4 in. above the LMC. Visual prompts in the games remind them to keep their hand in this range if their hand strays from an optimal position for the infrared cameras to track.

After administering preliminary clinical tests (see below), we performed a series of calibrations that were used to customize the games to accommodate the movement abilities of each subject. These calibrations evaluated active range of motion for hand opening and closing, wrist extension and flexion, radial and ulnar deviation, pronation and supination, shoulder horizontal abduction/adduction, and shoulder flexion/extension. Testing games, outlined above, were then completed for a baseline pre-test.

After the initial calibrations and testing, we showed the subject their first three games starting with one game from each category—Arm, Wrist and Hand. We set the personalized adjustments for each game based on their impairment level, taught them to save and transmit their data to our server, and how to properly store their system.

HoVRS was placed in subjects’ homes for 3 months. A subset of 5 games (Maze, Wrist Flying, Finger Flying, Car, Fruit Catch) out of the 12-game library, at least one from each category (Elbow-Shoulder, Wrist, Hand, Whole Arm) was chosen for this exploratory pilot study. This ensured the sufficient amount of time on each game and standardized the exposure to gaming activities across subjects. We were able to calibrate four of the five games, so all subjects were able to play. The fifth, Fruit Catch, a complex multi joint activity could not be performed by the most impaired subject in the study. Each weekday, subjects were encouraged to play at least 3 rehabilitation activities for a total minimum of 15 min. We did not schedule subjects to use the system at specific times or interact with them during each session in order to examine frequency and duration of subjects’ independent usage.

During the first month, we made visits to their home once a week to check on calibrations, check their game performance, answer any questions, and provide any technical support. In the second month of the study we supplemented 50% of the home visits with video calls to check on their progress, remotely monitor their gameplay, and answer any questions. If they needed technical support, we did it remotely from lab computers. For the final month, we switched entirely to online check-ins and only went for a home visit if they required additional assistance that could not be completed online. During the final visit to remove the system, we repeated the clinical assessments and computer based kinematic tests.

HoVRS logs time and score data as well as finger and arm kinematic data as the subject performs training activities. This provides the clinician supervising the training with the information required to make a wide variety of clinical decisions. For example, Fig. 3a demonstrates weekly training times for all games performed by a representative subject who had the system longer than 3 months. The therapist supervising this subject used these data to determine that he was unable to play the Finger Flying game comfortably on his own. The therapist modified the game by introducing a new positioning scheme, which allowed the subject to use the system more frequently and productively. The effectiveness of clinical decisions can be evaluated using performance data, as demonstrated in Fig. 3b, which shows daily hand opening range of motion over the course of 1 month. This confirms that the calibration and configuration of the Car Game was sufficient to elicit sustained improvement in HOR. Figure 3c demonstrates changes in wrist extension active range of motion during a single session of playing Wrist Flying. This confirms that the subject responded to the game’s calibration and configuration with a steady increase in wrist extension, the game’s target motor behavior. Based on this, the therapist encouraged the subject to train with the system using the current set-up.

The Upper Extremity Fugl-Meyer Assessment (UEFMA) was conducted before and after the 3-month intervention. In addition, six hand and arm kinematics were measured using testing games (see “Design” section above).

A paired t-test was used to test the difference in UEFMA score between pre and post intervention. Pre and post training test values for kinematic measures and the UEFMA were utilized to examine the relationship between the kinematic measurements provided by LMC and the level of impairment. As there is no gold standard kinematic measure or model of kinematic measures used to describe upper extremity motor function in persons with stroke, we chose an exploratory analysis using Best Subsets Regression analysis to evaluate all 6 kinematic measures and all possible combinations of measures. We identified the models that met the following criteria. (1) the highest R²(adjusted). R²(adjusted) incorporates the number of predictors in the model to help us choose the correct model. (2) a large R²(predicted) that is not substantially smaller the R²(adjusted). R²(predicted) evaluates a regression model by pulling out each data point and uses the model to predict the missing output, controlling for models with too many predictors. (3) a Mallow’s CP that is not substantially larger than the number of predictors plus 1. Mallow’s CP is a measure of the bias in the sample not due to random sampling error.