Psychometric characterization of incidental feedback sources during grasping with a hand prosthesis

Twelve able-bodied volunteers (24 ± 3 years old, between 18 and 27 years, 6 women, 6 men) participated in the two experiments. The subjects were informed about the experiments and confirmed that they understood the experimental procedure. They gave informed consent to take part in the study and were reimbursed with 10 Euro per hour of participation. The study was approved by the ethics committee of the Universitätsmedizin Göttingen. Ten of the participants were right handed, two left handed. Four had corrected vision. All subjects had taken part in one previous study on prosthetics [46] but were otherwise naïve to prosthesis control.

The current study was divided into two experiments, which were performed on different days, with Experiment I (Exp I) being at least 2 weeks before Experiment II (Exp II). The setup was similar in both experiments, as shown in Fig. 1a. It comprised 1) a prosthetic hand (Michelangelo prosthetic hand, Otto Bock Healthcare GmbH, Vienna, AT), controlled via Bluetooth, 2) a standard desktop PC with a 22″ computer screen (hidden from the subject’s view) controlling the prosthesis, 3) headphones playing white noise, and 4) a wooden block. In Exp II, the setup was extended by 5) two C2 tactors with a control unit (Engineering Acoustics, US) connected to the PC via a USB port to deliver vibrations, and 6) a laptop with a mouse used by the subject to indicate answers with the right hand (as explained below). The subjects were seated in a chair with the prosthetic hand in a table-top position in front of them at a distance of approx. 0.5 m, allowing a lateral view on the prosthesis (thumb-side, see Fig. 1). When closed using pinch grip, the prosthesis grasped a wooden block, which was fixed to the table. In Exp II, one C2 tactor was placed on the ventral aspect of the forearm of the subject’s left arm. The subject placed the arm comfortably on a sideboard. The second vibrator was hanging freely in the air and was activated at the maximum intensity in each trial with vibration feedback, effectively preventing the subject to use the sound produced by the first vibrator as potential source of feedback. In addition, the subject wore headphones playing white noise.

The prosthesis and the tactors were controlled by a control-loop implemented in Matlab Simulink 2015b (Mathworks, US) using the prosthetic closed-loop test bench [49] that operated in real-time at 100 Hz. In this study, the prosthesis was controlled by a computer, i.e. the subjects did not control the prosthesis themselves, making them passive observers. During a single trial, the prosthesis closed with constant velocity, simulating a routine-grasping protocol [46]. Across trials, the closing velocity was changed according to the psychometric test procedure (explained below). In Exp I, the subjects verbally reported their discrete answers to the experimenter, whereas in Exp II they used the mouse to indicate the answer via a Matlab GUI.

The aim of the first experiment was to assess the just noticeable difference (JND) in the incidental feedback using a staircase procedure. The JND is the minimal change in the amplitude of the feedback variable that can be perceived by the subject [50], therefore determining the effective resolution of the feedback channel. In the context of the present study, the JND corresponds to the minimal change in the prosthesis closing velocity that can be perceived by looking at the prosthesis (visual JND) and/or by listening to the prosthesis sound (auditory JND).

The experiment started with a familiarization phase in which the subjects watched as the prosthesis closed 10 times. The closing velocities changed from 20 to 100% of the maximum velocity (in steps of 20%), which corresponds to absolute average closing speeds between 38 mm/s and 324 mm/s (see Fig. 8 in the Appendix). This sequence was repeated twice. The subjects could see and hear the prosthesis while it closed.

The JND was assessed using a transformed 1 up / 2 down staircase procedure in a two-alternative forced-choice (= 2AFC) task, which targets 70.71% correct performance [50]. Each step of the staircase procedure comprised two trials of prosthesis closing. The prosthesis closed once at a fixed predefined velocity (standard value) and once at a velocity (comparison value) that was changed adaptively across trials based on the participant’s previous answers. In each trial, the standard and comparison value were provided in random order. The prosthesis stayed for 1 s in the opened or closed position, before closing or opening again. The comparison value was always higher than the standard value. The subjects observed the prosthesis using the provided auditory information (SND), visual information (VIS), and visual and auditory information (VIS + SND), depending on the condition, and they were then asked to report in which trial the prosthesis closed faster. If subjects answered correctly two times in a row, the comparison value was reduced by 1%. If they made a mistake, it was increased by 1%. The first five steps of the staircase were used for reinforced learning, where the experimenter revealed the correct response, to make sure that the subjects understood the task. The initial comparison velocity was set to be 15% higher than the standard one, which was a large difference and hence simple to discriminate. After the reinforcement learning, the normal staircase procedure resumed, i.e., the subjects received no more feedback on the correct answer. The staircase procedure was terminated if 13 reversals occurred, with a reversal being the change of the comparison value from increasing to decreasing or vice versa (see Fig. 2, VIS for an example). The JND (staircase outcome) was then calculated as the difference between the mean of the last 10 reversals and the standard value. The procedure was also terminated if the comparison value was only 1% higher (i.e., as low as possible) than the standard value 7 times in a row (see Fig. 2, SND and VIS + SND for an example). In this case, the JND was set to 1%. To assess the JND at both low and high velocities, the staircase was performed two times, at the standard values of 35 and 80%, respectively (order pseudorandomized across subjects), that correspond to absolute average closing speeds of 72 mm/s and 231 mm/s (see Fig. 8 in the Appendix).

The JND was measured in several conditions with different implicit feedback. In SND the vision was blocked by a screen in front of the prosthesis, which made only the closing sound available. In VIS, the subjects could see but not hear the prosthesis since the sound was blocked by noise-cancelling earphones playing white noise. In the final condition (VIS + SND), the subjects could normally hear and see the prosthesis. The order of the feedback conditions was pseudorandomized across subjects. Generally, the subjects were encouraged to focus on every detail of the closing and grasping process that they deemed could be useful to evaluate the presented closing velocity. This included the interaction between the wooden block and the prosthesis, the duration of the closing process, the frequency and intensity of the sound, etc. All these possibilities were explained to the subject beforehand.

The aim of the second experiment was to assess how well the subjects could estimate the prosthesis closing velocity using different sources of incidental feedback. In this case, the prosthesis closed at a constant but arbitrary velocity and the subjects were asked to provide an estimate of the closing speed (absolute estimation versus relative discrimination in Exp I). In addition, the subjects’ ability of estimating the intensity/frequency of vibrotactile stimulation was assessed, a method which is commonly used in prosthetics to implement sensory feedback [10].

The experiment comprised four feedback conditions, three with implicit (SND, VIS, and VIS + SND), and one with vibrotactile feedback (VIB). Each condition comprised a familiarization phase, a reinforcement-learning phase, and a testing phase. The only exception was VIS + SND, where only the test phase was implemented to assess the potential fusion of information from the incidental feedback sources.

During the familiarization phase in SND and VIS, the prosthetic hand was closed at five velocities (20, 40, 60, 80, and 100%) first in increasing, then in decreasing, and finally in random order. In the reinforcement-learning phase, the prosthesis closed randomly five times at each of these velocities (25 trials). After each trial, the subjects were asked to estimate the closing speed and the experimenter then informed them about the correct response. In the testing phase (correct answer not provided), the prosthesis closed at 17 velocities (20–100% with 5% steps), five times per velocity in random order. Importantly, the subjects were not informed about the discrete step-sizes (5%) but were told that any velocity between 20 and 100% might occur, in order to eliminate a bias towards the 5% steps. After the prosthetic hand closed, the subjects reported their closing-speed estimate by a mouse click to an analog scale (20–100%) shown at the laptop.

For the vibrotactile condition, the protocol was similar except that the task was to estimate the intensity/frequency of a vibrotactile stimulus (the prosthesis was not used). In each trial, a 2.5 s vibration was delivered using a C2 tactor fixed on the subjects’ left arm. The same levels and number of trials were used as in VIS and SND. The levels were implemented by simultaneously changing the frequency and intensity of stimulation. The dual-parameter modulation was used to facilitate the discrimination. The frequency (1–100%) ranged from 30 Hz to 270 Hz, and the intensity (1–100%) was chosen between 8 and 71% of the maximum intensity. Therefore, this condition was equivalent to using vibration feedback to transmit the prosthesis closing velocity via linear mapping.

The order of the four feedback conditions was randomized between subjects. However, the VIS + SND was always placed after VIS and SND such that the subjects received the same amount of learning in each of the two modalities before the fusion of modalities (VIS + SND) was tested. To investigate to which extent specific features were important when interpreting the feedback in each condition, the subjects were asked to indicate how much they relied on each feature using a visual-analog scale (0–10) in a questionnaire. They indicated how much they had focused on the speed of closing, duration of closing, and mechanical interaction between the prosthesis and the object for VIS, duration, frequency, and loudness of sound for SND, and frequency and intensity of vibration for VIB. In addition, in each condition they were asked to estimate their average absolute deviation from the correct value, to assess how confident they were in their estimation.

For evaluation, the success rate for the reinforcement-learning phase and the estimation errors in the testing phase were calculated for each trial, separately for each subject, level, and feedback condition. Again, non-parametric statistics were used. For the analysis of the reinforcement learning, the subjects’ success rate, averaged across the five tested levels, was compared across the three trained feedback conditions using a Friedman test and, upon significance, three Bonferroni-corrected Wilcoxon signed-rank tests were applied. In the testing phase, the median of the estimated value was computed for each level and feedback condition, reflecting whether and how much the subject over- or underestimated the correct level. For each level (except for the lowest and highest where over- and undershooting were not possible, respectively) and feedback condition, Bonferroni corrected Wilcoxon signed-rank tests were used to assess whether the subjects consistently over- or underestimated the level, by subtracting the estimated from the correct value and testing whether that value differed significantly from zero. Then, by pooling the absolute errors of all respective trials, the median absolute error (MAE) was determined for each subject and feedback condition, independently of the level. To test for differences between feedback conditions, a Friedman test and, upon significance, six Bonferroni-corrected Wilcoxon signed-rank tests on the MAE were used.

For analysis of the subjective results from the questionnaire, for both SND and VIS as well as separately for the acoustic and the visual features in VIS + SND, a Friedman test was applied and, upon significance, three Bonferroni-corrected Wilcoxon signed-rank tests were used for pairwise comparison. For VIB, a single Wilcoxon signed-rank test was used (only two features available). Six Bonferroni-corrected Wilcoxon signed-rank tests compared whether the feature usage differed when using one-feedback modality (SND or VIS) instead of full feedback (VIS + SND), e.g., the subjective reliance on the auditory duration of the prosthesis closing was compared between SND and VIS + SND condition. In addition, a dominance towards visual versus auditory features was explored by comparing the amount of reliance on the visual versus auditory features in VIS + SND, indicating which modality captured more attention when the full implicit feedback was available. To that end, in the VIS + SND condition for each subject the most strongly used auditory and visual features were selected across the three possible features per modality and these two were compared via a Wilcoxon signed-rank test.

A representative example of a staircase trial is shown in Fig. 2. In general, at the high level of the standard stimulus, the subjects were able to discriminate the smallest possible change in prosthesis closing velocity (1%) in SND and VIS + SND conditions. The staircase sequence “saturated” at the difference of 1% in 9 and 10 out of the 12 subjects, respectively. However, this did not happen for any of the subjects in the VIS condition, and it was also uncommon at the low level of the standard stimulus regardless of the feedback condition (i.e., only a single subject per condition succeeded in saturating the staircase). Nevertheless, all feedback conditions resulted in a small JND with a median value of less than 4%. The summary results are shown in Fig. 3. The visual feedback was characterized with slightly higher JNDs. With median JND of 3.7 and 3.5% for the low and high level, respectively, VIS led to low but still significantly higher JNDs compared to SND (1.5 and 1%, respectively; p < 0.001) and VIS + SND (1.5 and 1%, respectively; p = 0.007). The median JND in the latter two conditions was very close or even equal to the smallest possible change in velocity (1%). The JND was slightly but significantly smaller at the high compared to the low standard-stimulus level for SND (p = 0.006) and VIS + SND (p = 0.004), while no significant difference was observed for VIS. Generally, these data show a surprisingly good discrimination no matter which type of implicit feedback was provided to the subjects.

During reinforcement learning, the success rates (median [interquartile range]) were 92% [8%] in SND, 88% [82%] in VIS, and 74% [18%] in VIB. Hence, the performance in VIB was worse compared to the two other conditions.

The data from the testing phase of one representative subject are shown in Fig. 4, where the estimated versus correct closing velocities (SND, VIS, VIS + SND) and vibration levels (VIB) are plotted against each other for all feedback conditions and tested levels. The points are scattered around the optimal-performance line (black line), with a slightly worse quality of estimation in VIB.

The overall results are shown in Figs. 5 and 6. The medians of the estimated values followed consistenly the optimal-performance line (in black) across levels (Fig. 5). Statistically significant deviations from the line were not observed at any level for SND and VIS, at only two levels for VIS + SND, and at 5 out of 15 levels for VIB, indicating a general trend to underestimate the levels between 40 and 70% in this condition. Figure 6 shows the boxplots of the median absolute errors (MAE) across subjects (obtained by pooling trials across tested levels) for all feedback conditions. The performance was similar in SND (5.2% MAE) and VIS + SND (5.1% MAE). The estimation error in VIS + SND was significantly smaller than in VIS (6.7%; p = 0.005) and VIB (8.4%; p < 0.001) and the error in SND was significantly smaller than in VIB (p = 0.002). No significant difference in the quality of estimation was found between VIS and VIB.

Figure 7 summarizes the results of the questionnaire in which the subjects reported on the relevance assigned to different features of the feedback sources. Regarding the auditory features, most subjects seemed to focus on closing duration (median 80% for SND and 70% for VIS + SND) and frequency (80% for SND and 80% for VIS + SND) instead of intensity (45% for SND and 35% for VIS + SND). However, this trend was not significant neither in SND nor in VIS + SND likely due to high variability across subjects. Regarding visual features in VIS (Friedman test p < 0.001) and VIS + SND (Friedman test p < 0.001), the subjects were significantly more attentive to the duration (90% for VIS, p < 0.001, and 70% for VIS + SND, p = 0.001) and closing speed (60% for VIS, p = 0.001, and 70% for VIS + SND, p < 0.001) than to the interaction between prosthesis and wooden block (10% for VIS and 0% for VIS + SND), which was hardly exploited at all. No significant differences in used features between full (VIS + SND) and partial (SND or VIS) feedback were observed, except for a higher focus on the closing duration in VIS compared to VIS + SND (shown by a black asterisk in Fig. 7; p = 0.002). In VIS + SND, the subjects concentrated similarly on visual and auditory features, with none of the two modalities being rated significantly higher than the other. For the vibratory task, the subjects clearly focused significantly more on the intensity (median 100%) than on the frequency information (10%; p < 0.001). Concerning the subjective estimation of performance, they reported a subjective MAE of 10% (IQR 6%) for SND, 17% (IQR 8%) for VIS, 9% (IQR 7%) for VIS + SND, and 15% (IQR 12%) for VIB.