Background
Human hand is a dexterous end-effector and a sophisticated instrument for sensory exploration [
1]. After an amputation, these important motor and sensory functions are abruptly lost. Myoelectric hand prostheses can be used to restore grasping. The control signal (input voltage) driving the prosthesis motor is obtained by applying simple processing (smoothing) to the electromyography (EMG) signals recorded from the user muscles. The commercial state-of-the-art myoelectric interface uses two channels of EMG: the activity of hand and wrist flexor muscles is proportional to the prosthesis closing speed and grasping force, while the extensor activity controls proportionally the speed of opening [
2,
3]. Therefore, the commercial myoelectric prostheses provide the grasping function by restoring the feedforward pathway between the user’s brain and the artificial hand, but there is no sensory feedback from the prosthesis to the user. There is only one commercially available system [
4], presented recently, implementing a simple feedback about the hand grasping force. In principle, sensor data can be transmitted from the prosthesis to the user invasively, through a direct stimulation of the nerves [
5], and non-invasively, by electrically [
6] and/or mechanically [
7] stimulating the skin. Closing the loop in myoelectric prostheses was acknowledged as an important future goal by the prospective users as well as researchers in the field [
8]. Sensory feedback might improve the utility of the assistive devices as well as facilitate the embodiment [
9].
Two-channel myoelectric interface is a simple and intuitive control method since the user operates the prosthesis by activating the same muscles (finger flexors/extensors) that were responsible for those functions (hand open/close) before the amputation. However, the EMG signals acquired using surface electrodes are noisy and variable, due to inherent limitations of the recording setup (e.g., detection separated from the signal source), and the control is thereby rather imprecise [
10]. For this reason, as demonstrated in [
11], the prosthesis may respond inconsistently to the user intentions. Repeatedly closing the prosthesis to generate the same grasping force was characterized with a large variability, which also increased with higher target forces. The subjects could not repeat muscle contractions in a reliable manner using the natural proprioceptive feedback from own muscles to provide consistent control signals. Imprecise control can produce user frustration, often leading to the abandonment of the prosthesis [
12]. Furthermore, it can be a limiting factor for the effectiveness of the sensory feedback [
11]. Indeed, it can be rather useless for the user to sense the state of the system (e.g., aperture or grasping force), if he/she cannot produce a sequence of commands driving the prosthesis reliably towards the desired state (e.g., target aperture or grasping force). Improving the consistency of the command is thereby an extremely relevant goal. A reliable control loop would allow the benefits of the sensory feedback to be fully expressed. A well-controllable prosthesis following faithfully the user intentions would also better emulate the operation of its biological counterpart, potentially facilitating embodiment.
In the current study, we propose a novel concept for closing the loop in myoelectric prostheses, designed specifically to improve the consistency of the prosthesis response by allowing the user to reduce the variability of the control signals he/she generates by muscle activation. The new approach was tested experimentally and the tests demonstrated that it significantly improved the performance both in routine grasping and force steering tasks.
Discussion
A novel concept for closing the loop in myoelectric prostheses was demonstrated. In addition to feeding back the system output (generated grasping force), which is the classic method [
9], in the novel approach the system input (myoelectric control signal) was also transmitted back to the user. The tests demonstrated that the provision of the EMG biofeedback improved the performance in both routine grasping and force tracking tasks. In the routine grasping, the online information about the prosthesis input allowed the subjects to adjust the motor command during the closing of the prosthesis so that the desired level of grasping force was achieved when the object was contacted. The subjects employed this simple predictive control scheme to anticipate the resulting grasping force. The EMG biofeedback also assisted the modulation of force while the prosthesis was closed (force tracking). During this task, the biofeedback allowed the subjects to monitor the ongoing myoelectric activity and compare it to the current thresholds for the prosthesis activation (force increase/decrease). With this, they were able to finely regulate the myoelectric signals around the respective threshold levels and thereby control the timing as well as the magnitude of the force increase/decrease, improving the effective resolution of the generated force trajectory as well as the stability of tracking (fewer force drops). The statistically significant but overall modest decrease in the RMSE reflects the inherent limitations of the force modulation mechanism in the prosthesis (force jumps) as well as the nature of the task (continuous force tracking). The reference force trajectory was such that the subjects gradually modulated the strength of the muscle contraction. The advantage of the EMG biofeedback might be even better expressed during a step force regulation: grasp an object with a certain force, relax muscles (prosthesis maintains the force), and then increase/decrease the force to a higher/lower force level. Importantly, the tests in two amputee subjects demonstrated that the EMG biofeedback can improve the performance even in experienced users of myoelectric prostheses. This is a preliminary but optimistic result that will be further evaluated in a future study including a larger pool of amputee subjects.
EMG biofeedback has been extensively used in the past in many fields of application, including rehabilitation, but the context was different [
16]. For example, it is used during the user training to explain the principle of operation of the myoelectric prosthesis (e.g., as a didactic instrument) [
17]. To our knowledge, this study is the first demonstration that the subjects can employ this type of information to improve the online control of the prosthesis grasping forces. The envisioned goal is to integrate this feedback as a standard component to enhance a daily-life prosthesis application. For the latter, the EMG biofeedback would have to be delivered through a tactile interface, as discussed later. Another possibility would be to implement the same protocol as in the current study by using a wearable augmented reality module (e.g., Google Glass). The module could connect to the prosthesis directly via a Bluetooth link and the EMG biofeedback bars could be shown on the wearable displays in the peripheral vision field. This was however outside the scope of the current proof-of-concept study. Nevertheless, even the current setup, with a host PC and the EMG biofeedback delivered on the computer screen, could be used as an instrument for the functional prosthesis training. It could assist the subjects in learning consistent force control, since it explicitly depicts the predictive mapping between the myoelectric command and the resulting grasping force. In addition, the EMG biofeedback could be utilized in daily life (electrotactile, augmented reality) or in the lab (host PC setup) to train the subjects to better exploit the natural proprioceptive feedback coming from their own muscles for the closed-loop prosthesis control. By controlling the prosthesis while assisted through the EMG biofeedback, the subjects could learn the mapping between the sensation of muscle contraction, including the sense of effort, and the resulting grasping force. After some time, this mapping could stabilize and even render the EMG biofeedback redundant. To investigate this possibility, a future study will include a multi-session biofeedback protocol. In that sense, it would be especially relevant to test this training in the subjects that are experienced in myoelectric control. These subjects might have already learned to utilize the muscle proprioceptive feedback for control and the EMG biofeedback might not improve the performance substantially. However, the preliminary tests in the present study as well as the results in [
11] point out that this might not be the case.
The presented approach can be related to a model of the biological motor control [
18,
19]. It is hypothesized that humans acquire internal models of the body dynamics and use them to control the movements in a predictive manner. By applying the motor commands to the forward models, the system can be simulated to predict the expected sensory consequences of the movement (reafference). The estimated reafference can then be used for the closed-loop control, compensating for the delays that are inherent to the “conventional” sensory feedback transmitted through the peripheral neural pathways. In essence, the EMG biofeedback can be regarded as a simple feedforward simulation of a linearized prosthesis. It provides the subject with an estimate (prediction) of the grasping force, which will be developed when the hand contacts the object. This allows the subject to adjust the current online command (reafference-based control) even before the force begins developing (control based on the online sensory feedback).
In our previous work [
11], we demonstrated that the velocity of prosthesis closing can be used for a predictive control of grasping force. In the present study, the subjects had access to this information indirectly, since they had a clear view on the virtual gripper. Yet, the EMG biofeedback still improved the performance of force control. One more possibility would be to provide the closing velocity explicitly, using a visual bar (as for the EMG). However, implementing the predictive force control using EMG rather than velocity has several advantages. First, the feedback on velocity belongs to a classic scheme, in which the system state is transmitted to the user. Therefore, the system dynamics is still in the loop, i.e., the modulation of velocity is limited by the system responsiveness to user commands, including both mechanical (e.g., inertia) and computational (e.g., command processing and implementation) factors. On the other side, the modulation of EMG is virtually instantaneous. Second, feedback on velocity is meaningless after contact, since the velocity becomes zero. Therefore, it cannot be used to assist force steering. Thirdly, the EMG biofeedback can be implemented using standard prosthesis components, while to transmit the velocity one needs a velocity sensor (gyroscope) or a position sensor, where the latter has to provide a signal good enough to allow differentiation (which is not the case in Michelangelo Hand).
The aim of the current study was to describe the approach and test the concept feasibility. Therefore, the feedback was provided using an ideal interface (visual bar). The same approach could be implemented using electrotactile stimulation by transmitting the information about the magnitude of the control signal through a single-channel intensity and/or frequency and/or multi-channel spatial modulation. In the latter case, multiple stimulation electrodes can be used to implement an electrotactile equivalent of the visual bar, i.e., each electrode is associated to a signal range, and the current level of EMG is communicated by the currently active electrode within the array. Since the prosthesis is linearized, this also indicates the corresponding level of grasping force, once the prosthesis contacts the object. In order to produce a certain grasping force, the subject needs to activate the muscles so that a desired electrode starts stimulating. Providing the EMG biofeedback in this manner could result in a self-contained prosthetic system with an improved consistency of force control. The users would be able to produce a desired level of force repeatedly and reliably, eliminating the baseline variability as well as sudden large outliers that are characteristic for classic myocontrol [
11]. Implementing the electrotactile EMG biofeedback to test these hypotheses is the work in progress.
This is not however a simple task since there a number of questions still to be answered. Ideally, two variables (EMG and force) need to be communicated to the user. This can be accomplished by using separate interfaces (dedicated electrodes) or the same interface with separate coding (see the video EMGBiofeedback.wmv and accompanying explanation in the Additional file
1). In any case, this adds an additional complexity to the system and also for the user, regarding his/her ability to perceive and utilize this information. In principle, however, the system can be simplified by implementing only the EMG biofeedback. Leaving out the force feedback would not affect the performance during routine grasping and the upward force steering, since in these cases the force corresponds to the level of EMG (linearized prosthesis). For the downwards force steering, the feedback would not communicate the current force level (force feedback), but the user would still be able to control the force transitions (EMG biofeedback). In any case, substituting the visual with a tactile interface, certainly decreases the quality of the information transfer. Pure spatial coding, for example, is intuitive for the subject to understand, but also limited to transmitting a set of discrete levels (each electrode one level). Mixed coding can increase the resolution but also the user cognitive effort. There are also limitations due to the technologies, such as, narrow dynamic range in electrostimulation due to discomfort at the higher stimulation intensities. All in all, it is still to be investigated how these factors (e.g., decrease in resolution, cognitive efforts) would affect the hereby demonstrated advantages of the EMG biofeedback as well as the overall user experience and acceptance of this approach.
Importantly, there are also limitations that must be considered when applying this approach in amputees. In the present study, the quality of myoelectric interfacing was improved by applying abrasive gel. In the real-life application, this is not available as only normal gel is used to moisturize the skin. Also, the quality of the myoelectric signals will depend on the condition of the residual limb (e.g., weaker muscles, scar tissue). This can compromise the myoelectric control in both cases, with classical force and EMG biofeedback. The impact of these factors and possible mitigation strategies have to be tested in the future work.
The consistency and accuracy of grasping reflect how reliable the system is in reproducing the user intention to grasp an object with a specific force, repeatedly and routinely. This is relevant for utility but also embodiment. Human hand is a reliable end effector, which responds promptly and consistently to user intentions, and if the artificial substitute would have similar characteristics, this would promote the effective substitution, both functionally and psychologically. In addition to improving the repeated grasping with the same force, the EMG biofeedback could also facilitate switching between forces across trials, as explained in the previous paragraphs. From the functional viewpoint, the provision of feedback makes the task demands explicit, i.e., the user can establish a mapping between daily life tasks and the grasping forces that are necessary to perform those tasks. If the user is also confident that he/she can generate those forces accurately and consistently, this could facilitate the optimal utilization of the prosthesis (economical grasping paradigm [
20]). For example, if the EMG biofeedback is implemented using electrotactile stimulation with spatial coding at N levels, the user would know that he/she can generate N levels of force reliably. Through the use of the prosthesis, he/she would learn that specific tasks can be accomplished using certain forces, e.g., to pick grapes without squeezing them the force should be set at the level 2. Therefore, the user would determine the target force based on experience, and then generate that force fast and reliably using the EMG biofeedback interface.
The quality of force steering assessed through RMSE is relevant for object holding and manipulation. For example, when the force is gradually applied to a delicate object (e.g., wine glass) or when the force needs to be gradually decreased, e.g., for a smooth passing of an object from the prosthesis to a contralateral hand or to another person. In practice, unilateral amputees accomplish such sensitive tasks most often using a healthy hand, due to a poor controllability and other limitations [
21]. A system that would improve the force modulation could increase the applicability of the prosthesis, and therefore improve the tradeoff between the efforts (training, mounting, maintenance) and gained functionality.
In the present experiment, some of the feedback cues that would normally be available to the prostheses users have been blocked. For example, most of the present day prostheses, including Michelangelo Hand, produce noise during movement and force modulation. However, it is unlikely that these additional feedback sources would affect the results and conclusions of the present study. Those cues indicate the prosthesis state (aperture and force), which was anyway clearly disclosed to the subjects using visual feedback (virtual gripper and force bar). Due to this and the phenomenon of visual dominance [
22], it is unlikely that the additional cues, such as sound, would significantly improve the state assessment and therefore affect the overall performance. However, in a real-life application when the feedback is communicated through a practical electrotactile and/or vibrotactile interface and a visual assessment is non-ideal (e.g., viewing angle, occlusions), the incidental feedback could be more relevant. Importantly, this would mainly affect the force control using classic force feedback. From that point of view, the EMG biofeedback is rather robust, since the myoelectric command is adjusted based on the feedback about the state of the user (and not that of the prosthesis).
Myoelectric control can also be improved by applying specialized processing to the surface signals [
23] and/or acquiring better signals through implanted interfaces [
24]. Both approaches can substantially improve the stability and precision of the myoelectric waveforms. Importantly, these developments do not rule out the usefulness of the EMG biofeedback. More consistent signals lead to more consistent control, but the mapping between the subjective sense of muscle contraction and the resulting grasping force would still remain elusive. The latter connection can be made explicit by providing the EMG biofeedback to the user.
In this study, we have used a state of the art myoelectric hand, the latest model from Otto Bock. Importantly, the obtained insights and conclusions are general, since most myoelectric prostheses share the same principle of operation. Furthermore, the EMG biofeedback is not specific to force control. It could be utilized in a similar manner to facilitate the control of other prosthesis variables/degrees-of-freedom (e.g., velocity of opening/closing, velocity of wrist rotation).
Conclusions
The present study proposes a novel paradigm to close the loop in a myoelectric prosthesis. In the classic approach, the feedback transmits to the user the state of the prosthesis (aperture, velocity and/or force), whereas in the novel method the feedback also informs the user about his/her own latent variables, i.e., the myoelectric signals he/she generates (EMG biofeedback). The experiments demonstrated that the provision of the EMG biofeedback improved the quality of force control both in routine grasping and force steering tasks, and both in able-bodied subjects and two amputees who were experienced users of myoelectric prostheses. With the EMG biofeedback displayed as a visual bar on the computer screen, the subjects could see and modulate the current level of their muscle activity, and thereby explicitly control the command they send to the prosthesis. In the conventional approach, the myoelectric signals are latent variables, which can be controlled only by using indirect cues, such as subjective experience (sensation of muscle contraction) and/or observable consequences (e.g., prosthesis movement). These sources are however unreliable, especially due to the inherent variability of the myoelectric signals recorded using surface electrodes. EMG biofeedback allows the user to improve the precision and accuracy of myoelectric commands using active control, i.e., fast local loop in which the user modulates the strength of muscle contraction based on the online EMG biofeedback. The present study demonstrated the feasibility, and the next step is the implementation of this approach using practical interfaces, such as electrotactile stimulation and augmented reality glasses, and the validation in a larger pool of subjects. Therefore, there are many practical questions still to address (e.g., functional gain vs. user efforts vs. acceptability), but the present results are very optimistic. The prosthesis equipped with the EMG biofeedback might increase the user confidence in the system, by allowing consistent and reliable force control, and this can improve the utility, embodiment and ultimately the acceptance rate. Furthermore, the EMG biofeedback could be also considered as a temporary add-on to the prosthesis, an instrument for training the subject to exploit the natural feedback from his/her own muscles for the closed-loop prosthesis control.
Competing interests
MM and BG are employed by Otto Bock HealthCare Gmbh, which is a company that produced the prosthesis used in the study (Michelangelo Hand).
Authors’ contributions
SD, MM, BG, DF conceptualized the study. MM and KF conducted the experiments. SD, MM and KF analyzed the data. All authors participated in writing the manuscript. All authors read and approved the final manuscript.