Evaluation of an intelligent wheelchair system for older adults with cognitive impairments

As older adults age, there is a greater likelihood that they will develop chronic conditions that negatively impact their mobility, including arthritis, osteoporosis, or heart disease. Over time these chronic conditions can worsen and lead to difficulty in walking or to adverse events that may prevent walking altogether (for example, a fall or stroke). Older adults who are unable to walk are at risk of losing their independent mobility, an important aspect of their quality of life [1]. Fortunately, wheelchairs (both manual and powered) and scooters exist to compensate for this loss of walking ability and to offer a needed means of independence. It is known that older adults are the most prevalent users of wheelchairs in Canada, and it is estimated that 49% of older adults in Canadian institutional settings use a wheelchair [2]. With the demographic shift to a larger proportion of older adults, the demand for wheelchairs is expected to grow as well.

Powered wheelchairs in particular, are used by older adults who do not have the physical ability to propel a manual wheelchair or use a scooter. The benefits of powered wheelchair use have been documented: allowing older adults the freedom to engage with their environment and also relieving burden on caregivers [3]. However, a certain skill set is required to be able to operate a powered wheelchair, including: the ability to adequately control the driving interface; good reasoning to interpret and act upon the driving situation; sensory awareness of the environment; and adequate memory, attention, and focus [4]. Yet physical, sensory, and cognitive impairments could hamper an older adult’s ability to meet such requirements.

The impact of cognitive impairments on powered wheelchair use may be most prevalent within institutional settings. In Canada, it is estimated that approximately 65% of older adults in these settings have cognitive impairments [5]. Cognitive impairments could lead to agitation, poor impulse control, poor executive reasoning and planning, impaired attention, and memory difficulties [6]. The result of these deficits is an increased risk for accidents and difficulty in performing driving tasks. There is concern that older adults with cognitive impairments could cause collisions or accidents that harm themselves, others, or property. In many institutions, this concern has barred the use of powered wheelchairs entirely, whereas other institutions have applied screening procedures to judge safe and proper driving [7]. However, if an older adult cannot demonstrate competency with a powered wheelchair, then they are prevented from having a means of independent mobility.

Intelligent or smart wheelchairs have been in development to increase the accessibility and safety of powered wheelchairs for individuals with physical, sensory, or cognitive impairments. Over the past three decades researchers have investigated different input devices, environmental sensors, and control schemes to augment powered wheelchairs [8]. Although novel technology has been developed, much of this technology has not been transferred to the consumer market. One reason for this is the lack of user involvement in the design and testing of many intelligent wheelchairs [9]. Evidence to show the usefulness of the technology with its desired user population is a necessary component of technology transfer.

Since Simpson’s review of intelligent wheelchairs [8], more projects have included user testing in their evaluation or iterative design. However, only a subset of intelligent wheelchairs have been tested with individuals who have cognitive impairments (see Table 1, [9‐17]), and fewer still, with cognitively impaired older adults.

The overarching goal of this project was to develop and have users evaluate an intelligent system that is meant to help older adults with cognitive impairments drive a powered wheelchair safely and effectively. It is proposed that this project will give greater insight into the design of intelligent wheelchairs for this population.

The purpose of the study described in this paper is to evaluate an intelligent wheelchair in terms of its efficacy to perform within a potential environment of use, and efficacy to provide safety and usability for its users.

The Intelligent Wheelchair System (IWS) is design by the Intelligent Assistive Technology & Systems Lab (IATSL, University of Toronto) as an add-on to existing powered wheelchairs [18, 19]. Its purpose is to promote the safety and usability of powered wheelchairs for older adults with cognitive impairments. To achieve this purpose, the IWS adds two features when attached to a powered wheelchair: 1) anti-collision – to gently stop the wheelchair before it hits obstacles; and 2) navigation assistance – to help the user manoeuvre around obstacles in their environment by playing audio prompts that suggest navigation directions. For driving, a semi-autonomous control scheme is used, where the user retains control of the wheelchair in most situations, but is prevented from entering into collision situations and is aided (prompted) if they cannot navigate to avoid obstacles after a certain amount of time. This scheme was chosen because the authors believe that it would promote user enablement, exploration, and learning. The technology is meant to support the driver’s capabilities, not to replace the driver, a viewpoint that has been argued for by Nisbet [9].

The IWS is a continuation of intelligent wheelchairs designed by IATSL. Our previous systems have used a variety of sensors for obstacle detection, including: a mechanical contact skirt [16], an infrared sensor [20], and a stereovision camera [21]. Results from our previous work and a review of the field [8] suggest that vision is a promising sensor modality for detecting obstacles due to its large field of view (especially in terms of height, which is a limitation of planar lasers and sonar), low price point, and robustness over infrared to lighting noise. The IWS builds upon our previous work by improving the anti-collision algorithm and performance of the system, the details of which are presented over the next three sections. A distinguishing mark of our system, compared to other intelligent wheelchairs that have been tested with cognitively impaired individuals, is the depth (3D) vision sensor and the audio direction feedback (see Table 1 for technology comparison).

Figure 1 shows a block diagram of the IWS. A sensor is used to detect the environment in front of the wheelchair. This environmental information is then sent to an Onboard Computing Unit (OCU) for processing and analysis. The OCU determines the distance of obstacles from the front of the powered wheelchair and the free space surrounding those obstacles. If an obstacle is too close to the powered wheelchair the OCU will issue a prevention command to the joystickDCLM (Direction Control Logic Module). From this, the joystickDCLM will prevent any further wheelchair movement towards the obstacle. The joystickDCLM is situated between the user’s input device (i.e. joystick) and the wheelchair’s controller; it has the ability to prevent unsafe user inputs from reaching the wheelchair’s motors. When the wheelchair remains stopped by an obstacle for a certain period of time (~2 seconds), the OCU will play an audio prompt to help the user navigate into free space surrounding the obstacle (e.g., “try turning left”, “try turning right”). This audio prompt is repeated every five seconds if the wheelchair continues to remain stopped.

A FocusRobotic’s nDepth™ stereovision camera is used as the sensor. This camera was chosen because of its adequate field of view (64° horizontal FOV, 41° vertical FOV) and ability to produce depth/distance data at real-time speeds (30 Hz, 720x480 resolution) in its corresponding nDepth™ FPGA (field-programmable gate array) board. The FPGA board is housed within the OCU, and outputs depth data to a 1.8 MHz Pentium M single board computer (SBC). This SBC performs further image analysis (see “IWS: software implementation” section below), and is connected to speakers for audio output, as well as the joystickDCLM via an RS-232 link. The joystickDCLM is comprised of a microcontroller (Atmel ATmega644p) and has two A/D-D/A (analogue/digital-digital/analogue) channels for interfacing with the input device and the wheelchair’s controller. Communication between the OCU and the joystickDCLM utilizes a custom serial protocol through the RS-232 link.

When the SBC receives a depth image from the FPGA (Figure 2A) it performs two operations. One operation is to convert the depth image into a top-down occupancy grid of the environment (Figure 2B). The algorithm for this takes the maximum disparity (i.e. closest depth) of each column in the depth image and performs ray tracing to map these points into their corresponding world location. For a detailed description of this algorithm, the reader is referred to an earlier paper [20]. The other operation is to analyze the depth image for conjoined pixels (blobs) of high disparity, and create bounding boxes around blobs greater than a minimum size threshold (Figure 2C); these blobs are assumed to be obstacles in the environment. Disparities that are bounded relate to the desired stopping distance from obstacles. For example, if the stopping distance is 700 mm, the disparities of interest include values related to 700 mm and closer. A minimum size threshold was used in order to eliminate the effects of noise (both local and random) within the depth image, while still maintaining the ability to detect small objects (for example, canes). This method of noise reduction has been described by Murray and Little [22].

When a blob is bounded, the SBC determines which zone it occupies in the image. With the current sensor’s FOV, the image is segmented into three zones that each corresponds to a different powered wheelchair motion (i.e., forward-left, forward, and forward-right). Once a zone is occupied, the SBC sends a serial command to the joystickDCLM to prevent the corresponding powered wheelchair motion (Figure 2D). When the zone is free, this command is reversed. If the wheelchair remains stopped by an obstacle for a set duration, the left and right sides of the occupancy grid (Figure 2B) are analyzed to determine which side has the greatest free space, and the outcome of this calculation determines the audio prompt’s direction played to the user.

Figure 3, shows the IWS as it is mounted onto a powered wheelchair. Compared to our last iteration [21], the latest IWS has a new vision algorithm for detecting obstacles in the environment and the capability of preventing specific directions of wheelchair movement. The latest IWS also has significantly improved sensor coverage (50° to 64° horizontal FOV), image resolution (320x240 to 720x480), system update rate (2.25 to 24.47 Hz), and occupancy grid detail resolution (10 cm to 1 cm). As well, the system has been miniaturized to mount on the back of a powered wheelchair.

The goal of stage one was to evaluate the IWS’s anti-collision and navigation assistance features within a potential environment of use: institutional homes for the elderly. Trials within an experimental environment (with fixed fluorescent lighting) were designed to test the IWS’s ability to perform with common real-world objects from this setting. If the IWS proved capable of performing well with these objects, then it has the potential to be deployed in the same real-world setting. Similar trials were conducted with a previous iteration of the system [21] and results from those trials were used as a basis of comparison to evaluate the impact of the improvements made to the IWS.

To evaluate the IWS’s anti-collision feature, the IWS was mounted on a Pride Mobility™ Quantum 6000z powered wheelchair and driven towards six different object scenarios that would typically be found in a long-term care facility: 1) white wall pillar, 2) aluminum four-wheeled walker, 3) aluminum walking cane, 4) stationary person, 5) moving person, and 6) no object (to test for false detections). The wheelchair was driven from a distance of 3 m toward the objects to allow for a constant velocity of 0.16 m/s to be reached (Figure 4A). This velocity was chosen to allow for comparison with previous testing results [21]. Driving motion continued until the anti-collision feature stopped the wheelchair or the wheelchair hit the object. A threshold of 700 mm was set as the anti-collision stopping distance from the front of the sensor; this value was chosen to compare with previous testing results, and was inferred as a safe stopping distance with the camera mounted 300 mm behind the furthest forward point of the wheelchair (i.e. the footrests). For the moving person scenario, the person remained outside the field of view (FOV) of the camera until the camera came within 700 mm of the person (Figure 4B). At this time, the person would step into the FOV and stop in front of the wheelchair; this represented the worst case scenario of an object entering the FOV of the camera for the set threshold distance. The wheelchair was driven toward each object scenario 20 times. Both the response of the anti-collision feature and the stopping distance from the front of the sensor to the object were recorded.

To test the IWS’s navigation feature, the same wheelchair was driven toward three different navigation scenarios: 1) an object placed left of centre (Figure 4C); 2) an object placed right of centre (Figure 4D); and 3) no object (to detect false prompting). Each scenario was repeated 20 times and the object used in testing was the four-wheeled walker. Driving motion was allowed to continue until the wheelchair stopped and prompted the user, or the wheelchair hit the object. In each scenario, the audio prompt given by the navigation feature was recorded.

Signal detection theory was used to group the responses from the anti-collision and navigation assistance features. The categories of responses were: 1) Hit (object present, object detected/correct prompt issued); 2) Miss (object present, no object detected/absent or incorrect prompt); 3) False Alarm (no object present, object detected/prompt issued); and 4) Correct Reject (no object present, no object detected/no prompt issued).

The IWS performed successfully in all anti-collision test conditions. Table 3 compares the anti-collision performance of our previous system [21] and the IWS. Two misses occurred in both the wall and cane scenarios with the previous system. Figure 6 summarizes the average stopping from the object to the sensor^a. In all but one condition, the IWS had a closer stopping average to the set threshold than the previous system. Only the moving person scenario was on average 60 mm farther from the threshold than the previous system. For all conditions the IWS had smaller standard deviations in stopping distances (ranging from 16 - 37 mm) than the previous system (ranging from 78 -108 mm).

Table 4 describes the navigation performance of the previous system and the IWS. Both had successful prompting results in all navigation scenarios.

For objective measures: Figure 7, shows the movement pass rates totalled across each of the participant’s phases. A change in pass rate occurred between the phases for participant 1’s left turn (80% to 70%), right turn (10% to 100%), straight path (60% to 80%), and stopping (80% to 100%); and participant 2’s left turn (90% to 100%), and straight path (70% to 100%). The number of FOV collisions and the time to complete each run are shown in Figures 8 and 9 respectively for both participants. Participant 1’s FOV collisions had a sharp discontinuity when the IWS was introduced (one of the criteria to support the impact of an intervention in single-subject research [23]), and both participants maintained a lower magnitude of collisions during the IWS phase. It is noted that a peak in collisions occurred in both participant 1 and 2’s run 4. For participant 1’s time to completion, there was no sharp discontinuity between the phases, and the average in each phase is similar; whereas participant 2’s time to completion shows a rise in time when the IWS was introduced, and also a trend of decreasing time in each phase. Both participants did not reach a high adherence to audio prompts: participant 1 had an adherence of 76.5% and participant 2 had an adherence of 56.4%.

For subjective measures: Table 5 and Figure 10 report the scores for QUEST 2.0 and the NASA-TLX (averaged across each phase) respectively. In most cases the QUEST 2.0 scores for satisfaction increased with the IWS intervention. The exception to this was participant 2’s simplicity of use and total device score, which remained the same between the two phases. In all cases, the average NASA-TLX scores were lower with the IWS.

The IWS has the potential to perform well in an institutional setting as it enabled completion of all anti-collision and navigation scenarios successfully. When compared to our previous iteration [21], the IWS has improved detection of obstacles (for example, wall, and cane) and improved accuracy in stopping distances (that is, lower standard deviation in stopping distances). The improvement in detection of obstacles can be attributed to the higher sensor resolution – which is able to detect smaller objects; and the new blob algorithm – which is able to bound small profile objects with high sensitivity due to its noise rejection methods. Improvement in stopping distance accuracy is attributed to the higher update rate of the new system. Stopping distance accuracy is important, as it means the system will be more consistent to respond to objects in its environment. Having a high variability in stopping distance corresponds to unpredictability in anti-collision performance during real-world use. Although both the previous system and the IWS performed the same in navigation scenarios, it is expected that the IWS will be able to handle complex scenes with greater accuracy due to its higher resolution occupancy grid. In terms of the deviation of stopping distances from the set threshold, only the moving person scenario performed worse with the IWS. An explanation for this is that when a large object (for example, a person) abruptly enters the FOV of the sensor, the algorithm to bound regions of high disparity is computationally slowed down because of the size of the object. When compared to the standing person scenario, the IWS outperformed the previous system because the closest point of the person (for example, the hand) can be identified and bounded quickly. Overall, the improvements made to the system show promising results. Future work can focus on increasing the system’s speed and accuracy for correctly responding to obstacles in the environment.

Although the IWS performed well, the evaluation was conducted in a controlled environment without natural light. It is expected that natural light will affect the accuracy of obstacle detection. The developed blob detection algorithm is able withstand small lighting noise; however it is still susceptible to whiteout, which is an inherent difficulty for vision sensors.

Another issue is that this evaluation covered a limited number of scenarios. Other common activities such as table docking, bedside transfer, and doorway entry should be designed for and evaluated in future iterations.

Lastly, trials were conducted at a slow velocity (0.16 m/s) to allow for comparison to the previous system [21]. It is expected that the improvement in the update rate of the new system allows for higher velocities to be supported for collision detection. In future work, the effect of increasing wheelchair velocity on anti-collision performance should be explored.

Participant 1’s results support the idea that safety of powered wheelchair use can be improved with the IWS. Results from the movement pass rates show a marked increase in the right turn task (10% to 100%), which demonstrates that the IWS has the potential to help the user complete essential movements safely. Although there were changes in other pass rates between phases, these differences are not conclusive and may also be attributed to day-to-day driving variability. There was also a noticeable reduction in FOV collisions when the IWS was introduced, again supporting the idea of increased safety. As well, from a subjective viewpoint participant 1 felt more satisfied with safety when the IWS was used. Although these are positive remarks, it is important to note that not all FOV collisions were prevented by the IWS and the participant’s pass rates with the IWS were not 100% in all cases (also due to the IWS’s inability to prevent side/rear collisions). Finally, the sharp increase in collisions for participant 1’s phase A, run 4, could be attributed to a bad driving day, which in turn could be caused by other factors such as tiredness.

In terms of the IWS’s usability for participant 1, from an objective standpoint there was little improvement when the IWS was introduced because the time to complete the course did not show a change between phases. For subjective usability, QUEST 2.0 and NASA-TLX results showed an improvement, which is an interesting contrast to the objective result. The QUEST 2.0 simplicity of use score increased from “not satisfied at all” (1.0) to “more or less satisfied” (3.0), which is a positive note, yet also suggests that usability can be further improved for this participant. NASA-TLX results suggest that the IWS helped lower the perceived workload for this participant; however it is noted that this perceived improvement did not translate into improved objective performance with the powered wheelchair. Objective performance was inferred to be supported by the navigation feature of the IWS (that is, participants would remain stopped at obstacles for less time due to navigation prompts).

Participant 2 also showed an improvement in safety when the IWS was used. For movement pass rates, a significant increase in the straight path was seen (70% to 100%). Although FOV collisions were not observed to have sharp change between phases, it is inferred that the IWS can limit the severity of bad driving days (for example, participant 2’s phase A, run 4) by limiting the number of collisions that occur. Satisfaction with safety (QUEST 2.0) also improved when the IWS was used, but not all collisions were prevented with the IWS.

For usability, participant 2’s objective usability was negatively affected with the intervention of the IWS; however the participant had a trend of improvement in later runs, which suggests a learning effect for using the system better. Subjectively, QUEST 2.0 scores remained high with the IWS, and NASA-TLX results suggest that the IWS can lower the perceived workload for this participant. It is noted that the subjective usability was improved even though objective usability decreased. In an ideal case, both objective and subjective usability would be improved by the IWS. The powered wheelchair should be perceived easy to use by the user and also efficient to manoeuvre around the environment. The decrease in objective usability is likely linked to prompting adherence, as participant 2 had a low adherence (56.4%) for following the prompts.

Due to the limitations of single-subject research designs, the small sample size, and the selection criteria used in this study – it is difficult to generalize these results to the entire population of cognitively impaired older adults. These results nevertheless give insight into how the IWS can affect members within the desired user population and how our system’s design might be improved.

The trials, with three participants not able to complete the full study, also highlight the difficulties with conducting multi-day evaluation studies with this population. Future researchers should consider the time necessary for evaluation studies, or identify alternative study designs to involve users, as there are unique challenges with the cognitively impaired older adult population and the institutional setting [34, 35].

Because of participant drop out, the effects of learning to drive a powered wheelchair could not be negated through the phases as originally designed. Participants with a B-A phase ordering did not complete this study. In future research, a study with an A-B-A design could help mitigate the learning effect issue. Despite this limitation, there were other visual indicators of the intervention effect which have been described in the discussion section.

It is also acknowledged that the validity of subjective ratings given by individuals with cognitive impairments may be unclear. In this study, it was deemed that the participants demonstrated sufficient understanding of the subjective safety questions as their answers were corroborated by the objective results (for example, participants scores on subjective safety improved as objective performance of safety improved). Subjective usability results can be interpreted with caution. For future studies, subjective response validity could be improved by using observational data and open-ended questions during trials as additional corroborative evidence. Even though there is difficulty with validating subjective ratings with this population, subjective responses should still be investigated as it is an important aspect of assistive technology use.

The IWS has the potential to improve safety of powered wheelchair driving by lowering the number of collisions that can occur and by helping individuals to complete essential movements safely. Because the participants drove through a structured obstacle course based on movements related to daily life, it is also easier to extrapolate how the IWS would affect daily powered wheelchair driving. At this time, the IWS will likely perform well to help users navigate through hallways and around large rooms, but will have difficulty weaving through crowds of people or objects successfully. Previous groups have evaluated their intelligent/smart wheelchair with obstacle courses, but some of these courses were small and only cover a few movements related to powered wheelchair driving in daily life [12, 14]. This study highlights the need for the movements included in obstacle courses to be justified, relevant, and as encompassing of movements encountered in daily life as possible. Due to limitations of the IWS (only frontal sensor detection), not all movements of the Wheelchair Skills Test and Powered-Indoor Driving Assessment were included in the course as the system would have no impact on some of these movements. However, the Wheelchair Skills Test has been proposed as a standard evaluation course for intelligent wheelchairs by a group from McGill University [36]. Proper movement evaluation of intelligent wheelchairs will give insight into areas of improvement that might otherwise be missed, and a standard measure would allow for objective comparison between wheelchair designs. In this study, the weaving motion from the Powered-Indoor Driving Assessment identified the need for improvement of the IWS’s anti-collision feature when tightly turning around obstacles.

Although the Wheelchair Skills Test presents a methodology to evaluate common movements with intelligent wheelchairs, there is still no standard method of comparing intelligent wheelchairs and their response to dynamic obstacles. The structure of our environmental trials could be a precursor to such an evaluation, as it tested both static and dynamic obstacles. Yet, it should be noted that additional object scenarios in the environmental trials will be beneficial for comprehensive testing.

Upon further analysis of the collisions that occurred with the IWS (that is, the failures in movement pass rate), it was found that there were two main issues. The first, was that the IWS lacked sensor coverage in the side and rear areas of the powered wheelchair, which allowed collisions to occur in these regions. The second was that the stereovision camera had a blind spot at close proximity (~500 mm of sensor). Combined with the lack of full sensor coverage around the wheelchair, obstacles could enter into this blind spot and cause a collision when the wheelchair turned into them at close proximity. Future work will involve eliminating these collisions by extending sensor coverage around the wheelchair.

Usability of the powered wheelchair was an interesting issue. Although both participants showed improvements subjectively with the IWS, the objective performance of powered wheelchair use did not increase substantially. One possible explanation for the subjective improvements is that the IWS’s ability to prevent collisions and offer navigation tips gave the users assurance that lowered their perceived driving demands. However, the lack of objective improvement shows that navigation assistance of the IWS did not produce its desired effect. When linked with the low joystick adherence observed, it is clear that the IWS can still be improved for usability. The difference in participant prompting adherences suggests that audio prompting may be more effective for certain individuals. Perhaps prompting methods that are tailored to individual preferences (e.g., audio, visual, haptic feedback [37]) would be useful in increasing adherence and user proficiency of navigating the environment. This is an important point because users who navigate poorly in their environment can be seen as a hindrance to others [38].

System customizability could also extend beyond inter-person variability (differences in preferences and ability), to intra-person day-to-day variability. In the user trials, both participants had a bad driving day compared to other trials of the same phase. The occurrence of such a day could be due to a number of factors, such as changing alertness throughout the day or emotional/cognitive distraction. Regardless of the reason, the existence of day-to-day variability should be accounted for. One way to account for both inter-person and intra-person variability is to have multiple levels of assistance given by the IWS. In this way, the wheelchair could be set to a level of assistance that matches the user’s retained abilities.