In this study, we investigated the utility of cutaneous and kinesthetic haptic feedback in a myoelectric prosthesis and found that both feedback modalities resulted in performance that was significantly better than the clinical standard, no haptic feedback. We arrived at this conclusion through an experimental investigation involving a custom myoelectric prosthesis that can be worn by non-amputee participants and features joint-torque (kinesthetic) and vibrotactile (cutaneous) feedback that can be conditionally removed. In addition, we employed a cross-modality matching technique that generated user-defined equivalence mappings between joint-torque and vibrotactile stimulus intensities. In this way, we are able to compare each feedback modality to the clinical standard, as well as directly compare the two different modalities against each other without the confound caused by perceptual differences due to stimulus intensity.
The current research on haptic feedback in upper extremity prostheses is mixed with regard to the feedback’s potential utility. Not all investigations of haptic feedback in prostheses have resulted in improved functionality. In previous work done by members of our research group, it was found that neither vibrotactile nor joint-torque feedback improved grasp and lift performance over vision with a myoelectric prosthesis [
16]. Likewise, Saunders et al. found that vibrotactile feedback was only useful in a grasp and lift task with a myoelectric prosthesis when feedforward uncertainty was present in the control loop [
26]. On the contrary, it has previously been shown that vibrotactile feedback prevents amputees from breaking objects when wearing a myoelectric prosthesis [
12]. Likewise, a previous study done by members of our lab showed that joint-torque feedback provided added utility over vision in a stiffness discrimination task with a body-powered prosthesis [
7]. Other studies have also shown that stiffness discrimination is improved with the addition of haptic feedback [
27,
28]. Therefore, our finding, that the addition of vibrotactile and joint-torque feedback to myoelectric prostheses allows for improved object discrimination over no feedback for the soft-hard block combination, supports the argument that both cutaneous and kinesthetic forms of feedback can provide added utility and function to a myoelectric prosthesis. We believe the decreased performance in the no haptic condition was likely do to the absence of grip force information and not due to learning effects as we found no significant effect of trial order in our original model selection. Although the efficacy of haptic feedback appears to be related to how close the blocks are in stiffness, incorporating haptic feedback into a prosthesis would give users a better chance at object discrimination, which is critical for many activities of daily living.
Our secondary result indicated that there was no difference in object discrimination between joint-torque feedback and vibrotactile feedback in any of the block combinations. The literature comparing joint-torque feedback and vibrotactile feedback is limited to research done previously by members of this research group. Brown et al. found that there was no difference between joint-torque and vibrotactile feedback in a grasp and lift task using a myoelectric prosthesis [
16]. Similarly, Brown et al. found that there was no difference in kinesthetic feedback delivered to a different part of the body than the part used for exploration (non-colocated) and vibrotactile feedback in a single-DoF spring stiffness discrimination task [
29].
More broadly, studies have explored differences between kinesthetic and cutaneous feedback. In an angle discrimination task, Frisoli et al. found no difference between kinesthetic and cutaneous feedback [
30]. Recently, Kamikawa and Okamura found that hand-grounded kinesthetic feedback performed better than skin deformation (cutaneous) feedback in a force-matching task relative to a world-grounded kinesthetic device [
31]. This result, while contradictory to our own, might well be explained by the experimental task used, the kinesthetic control condition employed, or the colocated feedback (action originates from the same body part as where the haptic feedback is felt), which was shown previously by members of our research group to be superior to cutaneous feedback [
29].
Any stimulus can be reduced to its intensity and somatosensory encoding. In doing cross-modal matching, we attempted to equalize the intensities for each modality to compare differences in somatosensory encoding only. The lack of significant differences between the joint-torque and vibrotactile conditions indicate that participants’ ability to encode force as vibration or torque was not different, as both modalities were sensory substitutions of force. If one of the modalities was not a sensory substitution, it is possible that differences between modalities would emerge, as in [
31].
It is also possible that friction in the exoskeleton masked the object’s true force, thereby making it difficult to discriminate objects with relatively close stiffness values. For both vibrotactile and joint-torque conditions, we calculated the difference in haptic feedback stimulus intensity for each block pair and determined how many were below the just noticeable difference (JND). We found that 30%, 3.3%, and 0% of the stimuli differences were below an elbow joint-torque JND of 0.13 as reported in [
23] for the medium-hard, soft-medium, and soft-hard block pairs, respectively. For vibrotacile difference thresholds, Rothenberg et al. determined that the JND for forearm stimulation at 250 Hz was 0.2, while Mahns et al. found that it was 0.3 [
24]. Therefore, we chose a difference threshold of 0.25 which is in between these reported values. The proportion of stimulus intensity differences within a block pair less than or equal to 0.25 was 5%, 8.3%, and 0% for the medium-hard, soft-medium, and soft-hard block pairs, respectively. Compared to the joint-torque condition, the vibrotactile condition had much fewer pair differences below the JND in the medium-hard block pair combination, which may have contributed to it being significantly better than no haptic feedback, whereas joint-torque was not. In addition to improving the proportion of differences above the JND, an increased number of participants may also have helped to strengthen the significance of the results in the other combinations and conditions.
Feature Analysis
In addition to analyzing performance differences between the task conditions, we also examined how participants chose which block was stiffer. In all conditions, participants were privy to a number of incidental cues that could be used to make an educated guess as to which block was stiffer. Those cues included visual feedback of the terminal device aperture, visual indications of the overshoot and undershoot of the target aperture, and for the haptic feedback conditions, the perceived haptic stimulus intensity. Our analysis found that, in the absence of haptic feedback, participants’ decisions as to which block was stiffer aligned well with the visual based indicators, even though they did not always correlate with the true object stiffness. When feedback, kinesthetic or cutaneous, was available, reliance on these incidental cues diminished. This reliance on incidental cues is similar to the behavior anecdotally observed in amputees who wear myoelectric prostheses and must rely on visual feedback to properly control and understand interactions between the prosthesis and the environment. In a study done by Subah et al., it was found that in a reach-and-grasp task with a prosthesis simulator, both able-bodied users and amputees extensively monitored their virtual hand during grasping [
32]. This is in direct contrast with able-bodied individuals using their intact hand to manipulate objects, as their gaze is rarely fixated on the hand, but rather on the object [
33]. By adding haptic feedback, it is possible to reduce dependence on visual cues, thereby allowing amputees to focus attention elsewhere. In fact, Raveh et al. showed that the addition of vibrotactile feedback in a myoelectric prosthesis decreased the time to complete object manipulation tasks alongside a secondary task, indicating that feedback reduced some of the cognitive load associated with the prosthesis task [
10].
Interestingly, in the joint-torque feedback condition, in addition to the torque intensity, the aperture cue was also significant for the participants’ choice, while the exoskeleton angle was not. Previously it was noted that 30% of stimuli pairs in the medium-hard block combination in the joint-torque condition were below the JND. This might have led participants to rely more on visual cues like aperture when the feedback was hard to discern.
Limitations
It should be noted, that while our current results have positive implications for prosthesis development, there were several study limitations that should be addressed in future empirical investigations.
First, all haptic feedback was provided to the contralateral arm, thereby making it less natural than providing ipsilateral feedback. Chatterjee et al. along with investigations from members of our research group (Brown et al.) showed that haptic feedback on the contralateral limb had reduced performance compared to haptic feedback on the ipsilateral limb [
29,
34]. In this present study, we were limited by the exoskeleton, as it was infeasible to put this device on the same arm as the prosthesis. Despite this limitation, we were still able to show significant improvement over no haptic feedback.
Second, the no haptic feedback condition substituted pure visual feedback of the block deformations with a display scope of aperture. Therefore, it was an interpreted method of visual feedback, which may or may not have been more difficult than pure visual feedback. As it was possible to overshoot on any of the blocks, especially for able-bodied individuals who are not experts at EMG control, aperture is not a wholly reliable indicator of stiffness for this particular setup. Therefore, actually seeing the blocks deform may have improved accuracy. On the other hand, participants would not receive quantifiable visual measurements without the scope. We initially decided to use the scope method to ensure participants were consistent in their grip aperture, since closing down more on a soft object could yield a similar grip force to closing down less on a hard object. Additionally, this method allowed us to investigate how much participants used visual or haptic feedback cues in making their decision. A future alternative approach to visual feedback would be to limit the closure of the prosthesis to a set amount and provide feedback on grip force that is discounted by the aperture. Therefore, differences in grip aperture between block pairs would not affect the stimulus feedback.
Third, the load cell was at times sensitive to the relative placement of the prosthesis. The orientation and position of the participant’s arm controlled the slack in the Bowden cable, which in turn controlled the friction in the linear actuator. The orientation could change as the participant made small shifts throughout the duration of the experiment. As the load cell measures the tension in the cable, these values could become inconsistent between two test blocks. We attempted to remedy this issue by marking the places on the table where the prosthesis should rest, and on each of the blocks where the terminal device should make contact. In the future, the interior of the Bowden cable could be lined with Teflon to reduce friction. Although it was possible to simplify the setup by detaching the prosthesis from the subject completely, we wanted to encourage embodiment, and therefore attempted to mimic the amputee’s physical experience as closely as possible.
Fourth, the haptic feedback was held constant after reaching the target aperture threshold (Et), which is not representative of true force perception when squeezing objects. We avoided the natural increase in force that occurs when squeezing an object because the load cell, in addition to measuring the object force, also measured the spring force of the terminal device and the friction in the Bowden cable, which could mask the force of the block and cause difficulty in perceiving the difference between two blocks. By holding the feedback constant after passing Et, we intended to ensure distinct feedback for each test object despite the hardware limitations. This limitation, in addition to the ones mentioned above, would not come into play in a traditional myoelectric prosthesis or a body-powered device as there would be no linear actuator or load cell. Here, we attempted to control for these factors. While cable friction is present in standard body-powered devices, amputees learn to account for this friction with long-term repeated use. We were unable to provide the amount of training for our able-bodied participants to reach this level of proficiency. Additionally, even though the zero-order hold on feedback reduces to force discrimination, the inclusion of visual information encourages the integration of both force and aperture. Consequently, it is feasible for participants to discriminate the various objects based on more than just the haptic feedback of force. Finally, it is worth mentioning that a commercial myoelectric prosthesis outfitted with force sensors on the end effector could be used in future experiments.
As a final limitation, we tested only a few able-bodied individuals. In future studies, amputees will be evaluated as their performance may differ in the no haptic feedback condition, given their prior experience with myoelectric prostheses. In addition, we will also use maximum voluntary contraction (MVC) to calibrate EMG control. As our feature analysis is a model, we cannot draw definitive conclusions on how participants actually used the cues, and whether the addition of haptic feedback did indeed reduce cognitive loading. In future experiments, direct measures of cognitive load should be included.
Previous literature has shown that myoelectric prostheses may benefit from the addition of haptic feedback [
7,
18,
21]. In this study, we show that task performance is equally improved over no haptic feedback, regardless of the specific haptic feedback modality used. Contrary to our hypothesis, kinesthetic feedback was not superior to cutaneous feedback; perhaps one reason is that the dynamics of EMG are not exactly coupled to the myoelectric terminal device like the dynamics of the body are coupled to body-powered terminal device. In some respects, the mechanical operation of the body-powered prosthesis provides a colocated form of kinesthetic feedback, which as shown in our previous research, is superior to non-colocated cutaneous and kinesthetic feedback [
29]. It may be worth investigating whether there are performance differences between colocated (close to the EMG sensors) and non-colocated (far from the EMG sensors) feedback in a myoelectric prosthesis, as well as the effect of long-term training on the efficacy of haptic feedback. As it stands currently, cutaneous sensory substitution, which is cheaper and simpler to implement, will likely improve object discrimination performance over no haptic feedback.