Feasibility of force myography for the direct control of an assistive robotic hand orthosis in non-impaired individuals

Using an instrumented armband has shown to be a valid approach to detect relevant changes in FMG signals [19, 20] and allows for simplified sensor placement. Due to the lack of suitable commercially available solutions, a custom-built sensor band similar to the one used used by Xiao and Menon [19] was designed (Fig. 1A, B). To detect the force generated by the muscle’s volume changes in the forearm during muscle activations, force-sensitive resistors (FSRs) were used. Twelve FSRs (FSR402, Interlink Electronics, Inc., Los Angeles, CA) were embedded on the inner textile layer of the band with a 3cm distance between two consecutive sensors. The resulting band was suitable for arm circumferences between 20cm and 36cm which covers adult forearm circumferences reported by Delva et al. [21]. Circular acrylic plates were fixed on the back of the sensing area of the FSRs to enable a more even pressure distribution, and increase sensitivity for a consistent sensor read-out [22].

Signal collection and analog-digital-conversion was done on a microcontroller attached to the band (Arduino Nano BLE, Arduino s.r.l., Monza, Italy), where each FSR was connected to an individual digital output and a shared analog input. To quantify individual sensor readings at the analog input, a voltage divider circuit was implemented using a 56 kOhm pull-down resistor [17]. As commonly applied for FMG data acquisition [23], the targeted sampling frequency was approximately 100 Hz. The collected data was sent via USB to a computer running a Python script where it was saved for further analysis.

In this study, we investigated whether FMG signals could be used to control a fully wearable and portable RHO, the RELab tenoexo (Fig. 1C). This RHO has been designed to support grasp function in people with sensorimotor hand impairments through active assistance in finger flexion and extension [24]. Its fingers consist of a three-layered sliding spring mechanism, which allows for a compliant grasping of objects of different sizes and shapes without the need to control individual interphalangeal joints [25]. Two Bowden cable-based remote actuation systems transmit the force from the motors stored in a back module to the spring mechanism in the hand module [26]. One system actuates thumb flexion and extension, while the second actuates combined flexion and extension of the remaining four fingers. A manual slider allows for thumb opposition to enable the most relevant grasp types for daily tasks.

Participants were recruited at ETH Zurich for a single experimental session. All participants gave written informed consent, and all experimental procedures were approved by the ethics committee of ETH Zurich (2021-N-171). Neurologically-intact individuals with no impaired hand function who were above 18 years of age, able to give informed consent and understand the tasks involved in the study were eligible to participate. Major depression or deficits in cognition, communication, comprehension, or memory were defined as exclusion criteria.

A total of ten participants were enrolled in the study (all male, mean age 27.2 years, standard deviation SD = 2.2 years). Forearm circumferences ranged between 26 and 30 cm (mean: 27.0 cm, SD = 1.6 cm).

The full study setup and protocol is depicted in Fig. 2. Participants were seated at a table and had the RHO donned on their right hands. The FMG band was donned on the ipsilateral (i.e., right) forearm at the location of the largest circumference and rotated until the microcontroller was placed above the brachioradialis muscle. The band was adjusted until it was tight, yet comfortable as subjectively reported by the participants. Depending on the participant’s forearm circumference, only signals from sensors needed for one full loop around the forearm were acquired and sensors laying on top of those were considered inactive, resulting in a lower number of active sensors. The state of the RHO, i.e., “open” and “closed”, was manually changed by the examiner using a push button for the duration of the full experiment.

In an initial phase, the participants were instructed to try performing four gestures while wearing the RHO. While the state of the RHO was open, the two gestures were “relaxed open” (RO) and “try to close” (TC). For the gesture TC, the participants had to try to close their hand while the closing movement was counteracted by the force generated by the finger spring mechanism of the RHO. Vice versa, when the state of the RHO was closed, the two gestures were “relaxed closed” (RC) and “try to open” (TO). Participants were instructed to perform the two “not relaxed” gestures TO and TC with approximately half of their individually perceived maximal force. During this initial phase, a first visual inspection of signal quality was performed before the commencement of the experiments.

In the data collection phase, participants were asked to perform one of the four gestures while in a specific arm configuration. A sequence of performing each gesture continuously for 10s followed by a 5s break was conducted (Fig. 2B) and repeated for a total of nine arm configurations, i.e., seven static hand positions (positions 1–7) and two dynamic movements (movement 8–9) with a break of 60s after each arm configuration as depicted in Fig. 2A. The static positions were selected based on the positions proposed by Radmand et al. [27] and their relevance in terms of activities of daily living that could be supported by the RHO. Since the intended target population of the RHO, i.e., people after spinal cord injury with upper limb weakness, predominantly are wheelchair users, the proposed positions below table level and above head level were left out. The two closer positions at face level were fused into a single position in front of the participants’ mouth, since eating and drinking have been reported to be among the most desired tasks for target users of RHO [28]. The two dynamic movements consisted of a circular movement horizontally on table level and vertically in the right humeral plane, respectively. For all arm configurations and all gestures, the participants were instructed to sit comfortably and relaxed, and to have their hand oriented such that the thumb was facing upwards (except in static position 7) and the elbow facing downwards, and to have the shoulders in a non-elevated position. For position 7, the thumb faced the participant’s mouth in order to simulate the hand position during drinking. At all times, visual cues which showed the current arm configuration as well as the current gesture to be performed, were provided to the participants as text and image on a computer screen (Fig. 3). During the breaks, the participants had time to switch between arm configurations. The experimenter then inspected the arm configuration visually and, if required, provided instructions for adjustment to the participants. The whole process for each arm configuration including all four gestures was repeated five times.

There were two possible states of the RHO, i.e., “RHO open” or “RHO closed”. During “RHO open”, only the gestures RO or TC could be performed, whereas during “RHO closed”, only RC and TO could be performed. Since the current RHO state is always known and each state effectively only allows two possible gestures, we could use that information to simplify the four-class gesture recognition problem into two separate binary classifications, which in the following will be referred to as “RHO open” (RO/TC) and “RHO closed” (RC/TO).

In order to detect the user’s intent based from the collected raw FMG signals, different data processing pipelines, i.e., combinations of preprocessing steps and classification algorithms, were examined. Assessed preprocessing steps were: moving average filtering, Min-Max scaling, standard scaling, feature selection, and Principal Component Analysis (PCA), whereas assessed classification algorithms were: Support Vector Machine (SVM), K Nearest Neighbor (KNN), Random Forest (RF), and Linear Discriminant Analysis (LDA) as these have previously shown promising results for gesture classification using FMG [7, 13, 22].

For each participant, out of the five collected repetitions, four were used for classifier training and validation and the remaining repetition was used for testing. Different signal processing pipelines were evaluated in a four-fold leave-one-out-cross-validation manner and the overall highest performing pipeline was then selected to be applied for individual model generation. Reported classification accuracies are test accuracies obtained using the fifth repetition. Additionally, we measured training, as well as inference times. The training time was the time needed to train the model with our training data. The inference time was the time the model needed to make a prediction for a single sample. To measure the latter, we measured the time required for the total test set and divided it by the number of samples in that set.

In order to determine if the data collected in any arm configuration contributed more to the classification performance, a sequential forward selection technique was used [29, 30]. For this technique, we first determined the classification accuracies when only including data from a single configuration into the training set, and the configuration with the highest achieved accuracy was selected. In a second step, data from the selected configuration was combined with data from each of the remaining configurations individually to form the training set with two configurations. The respectively achieved classification accuracies were again compared and the combination achieving the highest accuracy was selected. This procedure was repeated until all configurations were included. The same signal processing pipeline as in the previous analysis was used for all participants. Despite being trained on data collected only in individual arm configurations or combinations thereof, the classifiers were always tested on the full test set which included data from all arm configurations. The resulting classification accuracies were averaged across all participants and both classifications in order to determine the overall contribution of individual arm configurations.

A Wilcoxon signed-rank test at a 0.05 significance level was performed to analyse statistical differences in performance between the classifiers “RHO open” and “RHO closed” as well as for the sequential forward selection of training arm configurations. A Mann–Whitney-U-Test was used to determine statistical differences between the performance of participants using 9 and those using 12 sensors. All data analyses were conducted offline using Python 3.7.12 in Google Colab (12 GB RAM, no use of a GPU).

The pipeline resulting in the highest classification accuracy in validation and which was consequently used for testing included no filtering, no feature selection, standard scaling and a SVM classifier using a Radial Basis Function (RBF) kernel. Parameters were tuned using a grid search. The resulting confusion matrices for each classification averaged over all participants are shown in Fig. 4. For the classification “RHO open”, the true class RO was about 1.5 times better predicted than TC (error rates \(3.6\%\) and \(5.5\%\), respectively). Similarly, for the classification “RHO closed”, there were about 1.6 times more false predictions (\(12.0\%\)) compared with the RC class (\(7.5\%\)).

Individual overall classification accuracies are shown in Fig. 5, ranging between \(75.4\%\) (ID8, classifier “RHO closed”) and \(100\%\) (ID4 and ID7, classifier “RHO open”). Averaged over both classifiers, participant ID1 achieved the highest accuracy (\(98.6\%\)) and ID2 performed the worst (\(81.4\%\)). No significant differences were found between participants using nine (n=8, i.e., ID1, ID3, ID4, ID5, ID6, ID7, ID8, ID9) and those using twelve (n=2, i.e., ID2 and ID10) active sensors (\(p > 0.05\)). For all participants except ID2 and ID6, the classification “RHO closed” performed worse than “RHO open”, whereas the difference in performance overall was significant (\(p = 0.049\)). The difference between the distinctness of the gestures in the two classifications is also visible when looking at the difference in raw voltage readings of the sensors (i.e., during “RHO open”, the readings between the gestures RO and TC were compared and during “RHO closed”, the readings between the gestures RC and TC were compared). As depicted in Fig. 6, the difference in sensor readings averaged across all nine arm configurations and all participants using nine or twelve active sensors, is smaller for the classification “RHO closed” than “RHO open” for each sensor. However, it has to be noted that the relation from sensor readings to pressure generated by the muscle contraction is not linear.

Classifier training took, on average, 16.35 s (SD = 15.06 s) for the classification “RHO open” and 46.44 s (SD = 25.20 s) for the classification “RHO closed”, whereas inference times for both classifications were approximately 0.05 ms (SD = 0.04ms) and 0.13 ms (SD = 0.08 ms) per sample, respectively. The largest measured inference time was 0.26 ms (classification “RHO closed”, ID 9). Significant differences were found in the training and inference times between the two classifications “RHO open” and “RHO closed” (training time: \(p = 0.013\), inference time: \(p = 0.010\)).

The achieved test accuracies during sequential forward selection is given in Fig. 7. The arm configuration which contributed the most to the ability of the model to predict gestures was the dynamic movement 9, i.e., a circular horizontal movement on table level. Listed by the order of contribution, the static positions 6, 7, 2, and 3 followed before the dynamic movement 8 and the three static positions on the right humeral plane, i.e., 5, 1, and 4. For the classification “RHO open”, the peak was achieved after adding position 3, and for the classification “RHO closed” after adding the dynamic movement 8, respectively. Compared to training on position 9 only, the first significant improvement in accuracy for both classifications was found after the three positions 6, 7, and 2 were added (\(p_{RHO open} = 0.002\), \(p_{RHO closed} = 0.037\)). For the classification “RHO open”, no further significant increase was found after adding the remaining positions 3, 8, 5, 1, and 4. However, for the classification “RHO closed”, a further significant improvement was found after additionally adding position 3 and movement 8 (\(p = 0.004\)).

The overall achieved average classification accuracy of 92.9% (individual classification RO/TC: 95.5%, RC/TO: 90.3%) exceeds the target accuracy of 90% proposed by Scheme and Englehart for reasonable, non-frustrating use in upper-limb prosthetics [31]. The total response time of the system, i.e., the time from movement intention to classification, consists of four aspects: the time from movement intention (measurable by electroencephalography) to electrical muscle activation (measurable by EMG), typically around 15–25 ms [32], the time between the onset of electrical muscle activity and the onset of muscle contraction (measurable by FMG), i.e., the electromechanical delay of the muscles, typically around 50 ms [33], the delay by the sampling of the FMG signal, i.e., 10 ms at 100 Hz, as well as the inference time. Thus, even considering the largest measured inference time of 0.26 ms, the total speed of response adds up to only approximately 85 ms, which is still more than fast enough to produce predictions in a real-time application, assuming that the bandwidth of human hand movement is typically below 4.5 Hz [34]. We can assume that this holds true even when switching to a microcontroller on-board the RHO with less processing power than the processor used for the analysis in this study, since the inference time makes only a very small fraction of the overall response time. Accordingly, classification accuracy and response time indicate general feasibility, but the fact that one of the classification accuracies only marginally falls above that limit calls for further investigation.

FMG has previously been examined for the control of an RHO by Yap et al. [15] who reported 95% online classification accuracy distinguishing four hand gestures in a fixed elbow configuration in three neurologically-intact participants in a setting where FMG was measured from the forearm contralateral to the RHO. Such a contralateral setting could foster bilateral movement training, which in turn can promote functional recovery in the contralateral paretic hand [35]. However, for practical control of an assistive RHO in activities of daily living, it is important that FMG data is acquired on the forearm ipsilateral to a worn RHO so that it does not limit bimanual activities. No previous studies have reported quantitative performance metrics such as classification accuracies using FMG in the ipsilateral setting. Xiao et al. [17] employed the ipsilateral setting in their study to help the participants familiarize themselves with the control of a RHO by using FMG signals collected from their ipsilateral forearm. Esposito et al. [18] implemented both, contra- and ipsilateral control and selected between those settings depending on the quality of the measurable FMG signal of the user. However, neither of these studies reported any quantitative results of the ipsilateral setting.

Despite only conducting a presumably simple binary classification compared to previous works distinguishing more gestures, the classification accuracies achieved in our work were not notably superior to other studies that employed FMG for gesture recognition. Using a similar sensor setup as the one used for our work, Xiao et al. [19] detected grasping actions in pick and place tasks with a comparable accuracy of 92% in neurologically-intact participants; Jiang et al. [29] were able to distinguish between 48 hand gestures in a cross-trial evaluation with an accuracy of 83.5%. The main reason that we could not achieve notably higher accuracies than these studies is likely the inherent physical restrictions posed by the RHO on the hand. Therefore, the gestures to classify were not as distinguishable as, e.g., a fully open hand and a closed fist, but rather resembled an isometric muscle contraction in the transition phases between the current states of the RHO (trying to close while RHO open/trying to open while RHO closed). In addition, Xiao et al. [19] also reported significantly higher prediction accuracies when using FMG signals collected from the wrist instead of the forearm. However, for many RHO, the wrist is covered by the device and does therefore not allow placement of FMG sensors in that area. All these observations support that wearing the RHO makes it more difficult to get distinguishable signals and thus correctly classify a desired action compared to gesture classification without wearing a RHO.

The significantly worse performance of the classification “RHO closed” compared to “RHO open” matched the expectations from observations during data collection and unstructured feedback by the participants. In neurologically-intact participants, finger flexors are usually stronger than finger extensors [36]. This leads to the expectation that during flexion (i.e., gesture TC) larger and hence more distinguishable FMG signals could be measurable than during extension (i.e., gesture TO). Further, when the RHO was open, the participants only had to counteract the stiffness of the springs in the fingers to perform the gesture TC. However, when the RHO was closed, it provided an additional force intended for grasp assistance, which the participants had to counteract in order to perform the gesture TO. Thus, it might be that the maximum force applied by the participants, as instructed by the experimenters, was too small to achieve a more distinct volumetric change in the muscles.

Looking at the different error modes in Fig. 4, we can distinguish between two types of failure for each classification: when a desired change of RHO-state is wrongly detected or when an actual desired change is not detected. The most frequently occurring error was found when the participants tried to change the state from RHO closed to open (true class TO), yet the RHO stayed closed (predicted class RC). While this type of failure might be annoying, it can usually be solved by just trying to conduct the desired gesture again. On the other hand, a misclassification when intending to keep the RHO closed yields an unintended opening (true class RC, predicted class TO). When the user is holding an object, this leads to dropping it. This failure mode is therefore considered to be the most critical. Although this failure mode occurred less than \(7\%\) of the time, it could be considered to adapt the decision threshold of the classifier for future iterations, in order to reduce the risk even further at the expense of a potential increase of undetected opening attempts [37].

Some variation in performance between participants was observed. Out of the ten participants, only six achieved an acceptable classification accuracy above \(90\%\) for both classifications “RHO open” and “RHO closed”. Further, although the classification “RHO open” performing overall significantly better than “RHO closed”, there were two participants for which the contrary was observed. Besides the participant’s individual ability to generate consistent muscle activations, further sources potentially introducing variability could be differences in band tightness based on the participant’s oral feedback [7], inconsistency in the sensor locations as previously investigated in EMG [38], e.g., after prolonged wearing time, or the amount of force applied against the finger mechanism of the RHO to perform the intended gesture.

An approach to try to compensate for the observed inter-participant variability is to use individually optimized data processing pipelines, i.e., combinations of preprocessing steps and classification algorithms, instead of a general pipeline for all participants. However, when investigating this option in a preliminary analysis, we found that such individual pipelines only performed marginally better in validation and, when translating to unseen data during testing, they performed even slightly worse than the general pipeline which was used in this work. Using the same pipeline across all participants leads to simpler processing and is consistent with other studies involving neurologically-intact participants [15, 19, 29]. However, it should be investigated whether an individual pipeline could provide a meaningful improvement in accuracy in case of lower muscle activations, e.g., for users with sensorimotor hand impairments.

Analysing the contribution of individual arm configurations to the overall classification performance, we found that the dynamic movement 9, i.e., the one horizontally on table level, contributed the highest. Including dynamic movements provides data with a higher variability in the training set, which in turn can make a model more suitable for testing in scenarios which also include such variabilities, such as eating or other activities of daily living. The three highest contributing static positions 6, 7, and 2 required the participants to extend the elbow straight to the left humeral plane (Positions 6 and 2) or to mouth level on the sagittal plane close to the body with a flexed elbow. Including these three static positions on different vertical levels yielded in a significant improvement in classification accuracy compared to only using one dynamic movement on table level, matching the suggestion by Radmand et al. [27] to include positions in both, straight and bent elbow configurations. Introducing the second dynamic movement, 8, resulted in a further significant increase for classification “RHO closed”. This observation matches our expectation, since adding this movement not only introduced dynamic, and thus highly variable data, but also, for the first time, introduces data from the right humeral plane. Accordingly, the data from static positions 5, 1 and 4, which all lie in the same plane as movement 8 might be partially redundant and therefore don’t contribute to further significant improvements. However, surprisingly, introducing the dynamic movement 8 led to a decrease in accuracy for classification “RHO open”. Yet, as the decrease was not significant and the achieved accuracy is still relatively high (\(>95\%\)) we assume that in this case the additionally introduced variability in data was not required since a plateau was already achieved earlier.

For this work, no data collection time constraints (i.e., large number of repetitions and arm configurations) were considered for the sake of achieving a large training set for the investigations. However, in real-life applications, such a long training time is critical as the users may lose motivation and experience fatigue during prolonged training. These findings lead to the conclusion that the static positions 4, 1, and 5 (in that order) could be removed in a future training data collection without notably decreasing the classification accuracy, reducing the overall training time and therefore the burden on the participants.

While this study provided valuable novel insights and a first indication towards the feasibility of using FMG to control a RHO, important aspects need to be taken into account when transferring these findings to practical applications. In a first step, the feasibility should also be investigated for online control. Previous works using EMG have found significantly worse performance in online classification compared to offline [39]. Further, all study participants were neurologically-intact, male, and represented a relatively low variability in age and forearm circumference. Unfortunately, there is only very limited data available on FMG signals collected from people with neurological hand impairments. For grasping detection of one grasp type, Sadarangani et al. [40] found that, compared to neurologically-intact individuals, individuals after stroke achieved inferior, but still acceptable (\(>90\%\)) classification accuracies. For people with hand impairments due to spinal cord injury, no such data is available. However, in this population, the residual muscle activity (and therefore the volumetric change) in the forearm depends on the type and level of lesion which could impact the distinctness of measurable FMG signals. For both these reasons, we expect that the classification accuracy in online control for people with neurological hand impairment would be lower than what was reported in this study. In order to improve online classification accuracy to be suitable for this population, some improvements on the hardware and the data collection are needed. On the hardware side, fusing data from multiple sensor types could be considered. Previous works have shown superior performance when using EMG and FMG data simultaneously compared to only EMG or only FMG [13, 41]. Further, the inclusion of inertial measurement units could identify dynamic motions in order to avoid misclassifications when transitioning between different hand positions during activities of daily living [42]. From an application point of view, different gestures could be used to trigger the desired actions. While the gestures used in this study were selected to resemble the targeted action (e.g., “trying to close” in order to close the hand), other less intuitive gestures such as, e.g., a simple co-contraction of the forearm, might be easier to perform for the participants and produce better distinguishable FMG signals [43]. Further, with data from a larger number of participants, more sophisticated signal analysis such as transfer learning could further allow for improved classification accuracies while keeping the required amount of training in an acceptable range [44].

This work, for the first time, assessed the feasibility of classifying opening and closing based on FMG data from the forearm while wearing a robotic hand orthosis on the ipsilateral hand in neurologically-intact participants. In an offline analysis, we found that using FMG could be a viable intention detection strategy for such assistive devices, yet for a more conclusive statement, further investigations involving people with hand impairments are required. Additionally, this work identified trade-offs between gesture recognition accuracy and the burden on the user during collection of training data and determined methods to optimize the training procedure and time without reducing gesture classification performance. Based on these findings, methods were identified which could potentially overcome challenges arising when transferring such technologies to their intended context of use, i.e., assisting people with sensorimotor hand impairments during activities of daily living.