Convolutional neural network in upper limb functional motion analysis after stroke

Related work

The class of ANN covers several network architectures including Convolutional Neural Networks (CNNs). The CNNs due to implementation of local connectivity patterns efficiently with shared weights, have quickly become a state-of-the-art method for image, video, and natural language processing etc. They are even frequently applied for the purpose of biomedical signals’ processing such as inter alia ECG, (Xu & Liu 2020).

The CNN enable local patterns recognition and is used in order to identify human actions from the temporal variation of these features, which are distorted due to the inconsistency in the execution of actions across observations and subjects, (Ijjina & Mohan, 2014).

Research presented by Guo, Gok & Sahin (2018) focused on implementation of the CNN model for prediction of the short-term dynamics of qualified forelimb movements based on neural signals in multiple animals. In Panwar et al. (2019), the effective classification of three upper arm movements is presented. The ANN methods were used to assess rehabilitation based on Cao et al. (2019), assess the progress of rehabilitation under the influence of a computer game (Bai, Song & Li, 2019) and analysis of the myoelectric signal during movement of upper limbs (Mukhopadhyay & Samui, 2020).

The CNNs achieved an almost perfect classification on physical activities, especially very similar ones which were previously perceived to be very difficult to classify. The CNNs outperform other state-of-the-art data mining techniques in HAR (Human Activity Recognition) (Ronao & Cho, 2016; Cho & Yoon, 2018).

The CNN is very efficient in image recognition, in which local spatial dependencies do exist, (Islam, Wijewickrema & OLeary, 2019). The same advantage can be usefully in motion analysis where the input data is organized into array represented with some snapshots of motion in form of activity image (Liu et al., 2016; Ijjina & Mohan, 2014).

Recently, CNNs have emerged as the 3D full-body human pose estimation from a marker-less monocular RGB (Red-Green-Blue) image sequence, (Zhou et al., 2018). In Ma, Sun & Sun (2018), the cascade CNN was used to upper limb estimation of joints position based on RGB-D quad channels image (color and depth information). These two approaches prove the usefulness of the CNNs in motion data analysis but are intended for a marker-less motion data acquisition systems. In the proposed application, the input is data from an optical marker-based system.

This work presents application of the general motion patterns, which are an upper limb lifting movements. The carried out analysis concerns some functional differences in motion. The thorough literature studies have proven that upper limb kinematics change as a result of stroke in comparison to the healthy ones from the control participants. The differences identified between individuals affected with stroke and those healthy can be used to advance interventions targeted at the underlying movement deficits. One way of improving our understanding of the time course of recovery may be to proceed by investigating the factors, which allow to predict the pattern of functional recovery as a function of time. The way the stroke affected patients try to control their functional movements is consistent in accordance with the Bernstein’s theory of human movement behavior. The most fundamental solution to the problem of controlling co-ordination functional movements consists of reducing the number of independent elements to be controlled. These changes enable patients to control functional tasks with more accuracy and less energy by reducing the number of degrees of freedom (Latash & Anson, 1996; Kwakkel, Kollen & Lindeman, 2004; Collins et al., 2018).

Conducted experiments

A total number of 54 participants were recruited to this study, including 35 patients of the stroke group (G1) and 19 healthy subjects (control group, G2).

Among the 35 affected with the ischemic stroke participants there were 16 women and 19 men (mean age 67 ± 8.9 years). These patients were 3–16 months after the first stroke, 20 of them were with right-hand paresis and 15 with left-hand paresis.

The control group (G2) consisted of 19 recruited healthy participants, whose age-matched the post-stroke patients and included 14 women and 5 men (mean age: 64 ± 9.0 years).

The below criteria were met for the stroke group of participants (G1): spasticity ≤2 in accordance with the modified Ashworth scale, who were able to stretch the affected arms out; no apraxia or shoulder pain that may interfere with task accomplishment; no any neuromuscular or orthopaedic disorders or major visual attention problems, major perceptual or cognitive deficits and ability to provide informed consent, disturbances of cognitive functions measured by Mini–Mental State Examination (MMSE ≥ 24); NIHSS (National Institutes of Health Stroke Scale) total score: median 4 (2 –5); Fugl-Meyer Motor Assessment (FMA –upper limb ): 44 ± 8.2.

All participants have received detailed information regarding the experimental procedure, which was going to be carried out and all of them gave their written consent for the study participation. The research project’s protocol was approved by the Bioethics Committee at the Opole Medical Chamber No. 215, on 25th March 2015 and the study was performed in accordance with the Helsinki Declaration’s recommendations for clinical trials on humans.

Motion data acquisition

For the purpose of 3D trajectory segments acquisition the optical motion analysis system based on passive, reflective markers with eight infrared cameras, was used (OptiTrack system, NaturalPoint, Inc. https://optitrack.com/). The motion acquisition frequency was 100 Hz.

The procedure was the same as in Blaszczyszyn et al. (2018) where only forward and back transporting phases of a cylinder movement were taken into account. The third performed task was drinking, where the first phase was to reach out for the glass from the starting position and then to grasp and move the glass forward to the mouth to drink and to place it back to the the table behind a marked line; and finally—to return to the initial position in accordance with the drinking phase definition in Murphy, Willén & Sunnerhagen (2011).

The markers were located on both upper limbs (right –R and left –L) on the following locations: clavicular sternal (CLAVR/L), acromion process (ACRR/L), middle part of the humeri (MPHR/L), lateral epicondyle (LEPR/L), radial styloid (RSR/L), ulnar styloid (USR/L), index finger nails (FNR/L) (Fig. 1). For presented application only trajectory of four markers ACRR/L (shoulder segment), MPHR/L (arm segment), LEPR/L (forearm segment) and FNR/L (hand segment) was applied.

The protocol describes following functional activities as follows: drinking from glass (DG), lifting a small cylinder (SC), lifting a large cylinder (LC).

The motion activities were measured with the OptiTrack system during the above defined tasks. The presented analysis was based on the 3D trajectory of the markers placed on the upper limb skeleton segments.

Convolutional neural network model

The CNNs are a specialized kind of neural network for processing data that has a known, grid-like topology. Classification using grid-like topology in case of motion data allows to process whole motion (lifting and lowering the hand) without split into windows. Also the motion of skeleton segments (markers) are treated in a correlated way. The proposed solution is a new application of the CNN based on the collected motion data of upper limb movement.

The convolutional layers used in the network are characterized by the learning of local patterns. This behavior makes these networks to have two key properties of the convolution networks: it allows to find patterns, regardless of their displacement and distortion; it has the ability to create a hierarchy of patterns. They take advantage of the hierarchical pattern in data and assemble more complex patterns using smaller and simpler patterns learned by previous layers.

This paper presents a solution for analysis of human functional motion during ADLs based on the implementation of the CNNs, which allows local patterns recognition by translation of invariance characteristics. Due to the fact that the position and scale of learned pattern is not important—it allows to find differences in movement pattern of the upper limb lifting movement.

The proposed CNN model consists of the following layers (Fig. 2) with the following hyperparameters:

• First convolution layer, where the input samples were processed using set numbers of filters (hyperparameter f_Conv1). The convolution layers used in the proposed network use a 3 × 3 kernel. The Rectified Linear Unit (ReLU) was used as an activation function.

• In the next stage, the processed data goes to the scaling layer, which performs the max-pooling operation with 2 × 2 window size.

• The third layer is convolution layer with set of filters (hyperparameter f_Conv2) and 3 × 3 kernel. The Rectified Linear Unit (ReLU) is used as an activation function.

• In the next stage, the processed data goes to the scaling layer, which performs the max-pooling operation with 2 × 2 window size.

• Next is a flattening layer.

• Dense layer with the Rectified Linear Unit (ReLU) used as activation function.

• Next is a dropout layer. This layer is applied between the last hidden layer and the output layer. The dropout rate is hyperparameter d_Drop1.

• Second dense layer with 3 or 2 outputs (depends on numbers of output classes). The activation function is softmax.

The Rectified Linear Unit (ReLU) is used as the activation function for hidden layers as it is default recommendation for CNN in (Goodfellow, Bengio & Courville, 2016). The ReLU function overcomes the vanishing gradient problem, allowing models to learn faster and perform more efficiently.

Classification accuracy is the total number of correct predictions divided by the total number of predictions made for a data-set. It is so, because for presented class definition unbalance occurs the AUC (area under the receiver operating characteristic curve) metric was used as a performance measure.

For the training purpose of the proposed CNN model the Adam optimizer were used. The loss value, which will be minimized by the model, was multi-class cross-entropy.

Data prepossessing

The Table 1 presents the summary of all input data with numbers of recordings (input data files). One recording consists of one functional motion (for example lifting and lowering the large cylinder). The G1 is group of participants after stroke and G2 is a control group of elderly participants. Dataset was splitted into training and validation set in ratio 80% to 20%. The given results are evaluation results of the validation data.

During the study, two classification tasks were proposed and tested. The first one was based on the upper limb type with the three following classes:

• HUL (Healthy Upper Limb) for upper limb of G2 participants (healthy control group), and contains all activities performed by participants form G2 (114 input data sets);

• NPUL (Non-Paretic Upper Limb) for non-paresis upper limb of G1 participants, contains activities performed by participants from G1 with non-paresis limb (105 input data sets);

• PUL (Paretic Upper Limb) for paresis upper limb of G1 group, contains of activities performed by paresis limb are labeled as class PUL (105 input data sets).

Second classification task is based on type of participant group, where:

• class G1 is for all data sets of participants after stroke in G1, regardless of the type (paresis/non paresis) of arm (210 input data sets);

• class G2 contains all data sets (left and right upper limb) of participants form control group G2 (114 input data sets).

Each plane (XY, YZ, XZ) of motion was analyzed separately, the input signal is the coordinate value in 3D space. The data prepossessing steps are following:

• Cut out the signal represented coordinates regarding movement in every plane. Rejection of insignificant data from the beginning and the end of signals (Fig. 3). In Fig. 3 positions of the FNL marker (marker on left index finger) when lifting a small cylinder were presented. It illustrates the coordinate signal clipped from the beginning of the movement (marked with green border) to the end (marked with red border) for each plane independently.

• Unification of the motion side, all left limb movement signals has been transformed (reflected) to the form of right limb movement. After this unification, the marker symbols are given without the last letter denoting the body side.

• Resampling of the input signals in meaning of change the sample rate. Regardless of the signal length (duration), trajectory signals have been resampled to fixed numbers of samples by linear interpolation. The predefined samples number was set to 32.

• Based on clipped coordinate signal consisting of N + 1 samples, p_k = (p_x,k, p_y,k, p_z,k), k ∈ [0, N], features based on 3D trajectory in each plane are determined: displacement (d_x, d_y, d_z), velocity (v_x, v_y, v_z), acceleration (a_x, a_y, a_z) and jerk (j_x, j_y, j_z). Displacement is the Euclidean distance between the subsequent markers positions for each dimension d_x|y|z,i = |p_x|y|z,i − p_x|y|z,i−1|. Velocity feature is vector of temporal velocities $v_{x\left|y\right|z,i}=\frac{d_{x\left|y\right|z,i}-d_{x\left|y\right|z,i-1}}{T}$, where T is sampling interval. Similarly the acceleration feature is obtained $a_{x\left|y\right|z,i}=\frac{v_{x\left|y\right|z,i}-v_{x\left|y\right|z,i-1}}{T}$. Jerk is the rate of acceleration changes with respect to time. Jerk can be expressed as the first time derivative of acceleration, second time derivative of velocity, and third time derivative of position $j_{x\left|y\right|z,i}=\frac{a_{x\left|y\right|z,i}-a_{x\left|y\right|z,i-1}}{T}$, where i ∈ [1, N].

• Input data normalization by min-max normalization to $\left[0,1\right]$ range using following equitation $f^{\prime }=\frac{f-min}{max-min}$, where f′ is normalized feature value f.

After cutting out the motion data from the raw input signal, being in the form of trajectory coordinates in three-dimensional space, the values of features (d_x, d_y, d_z, v_x, v_y, v_z, a_x, a_y, a_z, j_x, j_y, j_z) were calculated. Then they were converted to a multidimensional representation, depending on the number of markers used, but with fixed width (predefined 32 samples), height (4 features for each plane gives 12). The depth is based on the number of markers used.

The obtained features were arranged in an array, where the rows represented one feature and the columns contain next time feature samples. The obtained data is a result of calculations for separate markers, which were placed in subsequent layers of this array. The following set of 4 markers placed on upper limb was used in experiments: FN, LEP, MPH and ACR. The names after unification of side are without (L/R). The input data format was illustrated with the Fig. 4.

This representation of the data could be seen as an image, respectively: grey scale image (one channel represents 1 marker), two channel image (two channels for 2 markers), three channel image (RGB image, three channels for 3 markers), four channel image (RGBA image, four channels for 4 markers, Figs. 5 and 6).

The obtained data is based on four markers representing four segments of upper limb: hand (FN), forearm (LEP), arm (ACR) and shoulder (ACR). Such data allows to carry out analysis regarding which segment or segment configuration gives better classification results. Such sensitive location analysis is based on results’ accuracy concerning markers configuration. Firstly only one marker data was used, where depth of input array equals one (grey scale image). Next based on the best results the model obtained from two markers (depth equals 2), three (RGB image) and four markers (RGBA image) were tested.

Hyperparameter optimization finds a tuple of hyperparameters yielding an optimal model which minimizes a predefined loss function on the given data. In the proposed application of the CNN model the following set of hyperparametres was defined: learning rate (lr), number of epochs to train the model (ep), number of filters used in first convolution layer (f_Conv1), number of filters used in second convolution layer (f_Conv2), the dropout rate of dropout layer (d_Drop1), the batch size of the output space for first dense layer (out_Dense1).

Bayesian optimization using Gaussian processes was applied in order to find values of defined hyperparameters to obtain model with the highest metric AUC value on the validation set. Number of calls to find the minimum of (1 − AUC) was set to 40. Optimization was run for each experiment independently.

Implementation

For the development purposes the authors decided to use the Microsoft Visual Studio 2019. For the implementation Python 3.7 (Millman & Aivazis, 2011) was applied with the following most important libraries: Kreas as a high-level neural networks programming interface with TensorFlow backend (Erickson et al., 2017; Gulli & Pal, 2017 and Chollet, 2015); Scikit-Optimize for hyperparameters optimization; NumPy for the input data structure preparation as a multidimensional array (Van Der Walt, Colbert & Varoquaux, 2011 and Oliphant, 2006).