Development of an electrooculogram-based eye-computer interface for communication of individuals with amyotrophic lateral sclerosis

A total of 23 participants (20 healthy participants and three individuals with ALS) were recruited for this study. Prior to the experiments, all participants received a detailed explanation about the research purpose and design and provided written consent. The study protocol was approved by the Institutional Review Board (IRB) of Hanyang University Hospital. Among the 20 healthy participants (15 males and 5 females, mean age 24.2 ± 4.17 years), data from two were discarded due to severe artifacts caused by sweat. Six of the 18 healthy participants wore glasses/lenses. The first individual with ALS (female) had been diagnosed for 4 years, aged 59 years, and the ALSFRS (ALS functional rating scale) was 17 at the time of the participation. The second individual with ALS (male) was aged 63 years, had been diagnosed for 3.5 years, and the ALSFRS was 18. The third individual with ALS (male) had been diagnosed for 8 years, aged 41 years, and the ALSFRS was 25. The ALSFRS is a well-known measure for evaluating the functional status of individuals with ALS and is based on a questionnaire. The questionnaire evaluates daily activities in 12 categories: speech, salivation, swallowing, handwriting, cutting food, dressing and hygiene, turning in bed, walking, climbing stairs, dyspnea, orthopnea, and respiratory insufficiency [20]. The average ALSFRS score for a healthy participant is 48, while the score of a patient with ALS in a completely locked-in state is 0. Throughout the experiments, caregivers helped the experimenters to communicate with the participants with ALS.

EOG signals were acquired using an ActiveTwo biosignal recording system (Biosemi, Amsterdam, Netherlands) at a sampling rate of 2048 Hz. Four electrodes were placed around the eyes: two on the left and right sides of the eyes, and two above and below the right eye. A common mode sense electrode and a driven right leg electrode, which functioned as a reference and a ground electrode, respectively, were placed at the left and right mastoid. Prior to electrode attachment, the skin was cleaned with antiseptic wipes to eliminate sweat or other materials that could interfere with signal acquisition.

We designed 10 different eye-writing patterns, each corresponding to an Arabic number. These patterns were designed to minimize eye movement but to maintain the shapes of the original numbers (see Fig. 1). A red dot in each pattern indicates the starting point of eye-writing, and an arrow indicates the end point. The same patterns were used for healthy participants and participants with ALS; however, the pattern of the number ‘1’ was shifted to the center of the canvas to facilitate eye-writing by participants with ALS.

Data acquisition was controlled using E-Prime software (Psychology Software Tools, Inc., Sharpsburg, PA, USA). This software was also used to display numbers or other graphical instructions on a monitor with auditory instruction. All the raw EOG data recorded from healthy participants during the experiments are available at the EyeWriting repository (https://​github.​com/​EyeWriting/​EyewritingNumber​), but data from participants with ALS are not shared because IRB did not allow sharing of this dataset.

All experiments with healthy participants were conducted in a quiet room. Each participant was asked to sit on a comfortable armchair in front of a 24-in. monitor. The width and height of the monitor were 61 cm and 28.5 cm, respectively. Each participant was asked to place his/her head on a chin rest in order to minimize head movement, and the height of the chin rest was adjusted according to the participant’s preference. The distance between the monitor and the participant was set to approximately 62.5 cm, making 52.02° maximum visual angle that participants could move their eyes horizontally.

The experiments with participants with ALS used a different display device. Specifically, the first two experiments with participants with ALS (No. 19 and No. 20) were conducted in a hospital room, where the visual stimuli were projected on a wall using a video projector. The size of the projected screen was approximately 135 cm (width) × 102 cm (height). For the third experiment (participant No. 21), a 55-in. [86 cm (width) × 48 cm (height)] television was installed in the participant’s home. During the experiment, the participants were seated either in a wheelchair approximately 150 cm away from the projector screen (participants 19 and 20) or 125 cm away from the TV display (participant 21). Of note, a chin rest was not used for the participants with ALS.

The experimental procedure for healthy participants consisted of two consecutive sessions: a practice session and an exercise session (Fig. 2). During the practice session, the participants were asked to eye-write the number patterns along the given guide lines; the guide lines were not provided in the exercise session. For both sessions, the participants were instructed to eye-write each number in three steps. In the first step, each participant fixed his/her gaze at a fixation mark for 3 s. After keeping the shape of the number in mind, the participant began to eye-write the number on the 3 × 2 grid. The practice session was repeated until each participant became accustomed to eye-writing, without recording any EOG signals. The exercise session was then repeated three times; EOG signals were recorded during these sessions. The time durations for eye-writing a number varied every time the participants eye-wrote.

For the experiments with participants with ALS, the experimental procedures were modified to facilitate learning and to minimize user feedback. Each experiment was composed of three sessions: instruction, practice, and exercise (see Fig. 3). During the instruction session, the fundamental concepts of eye-writing were explained. First, a pattern corresponding to the number ‘0’ was shown to the participants for 7 s, allowing them to identify the number pattern. The patients were then asked to fix their gaze at the center point in the screen, after which a small red dot was displayed at the starting point. The red dot then disappeared and soon reappeared at the next corner point. In this manner, the red dot sequentially moved and drew the number pattern until reaching the end point. Each participant eye-wrote the number pattern by shifting his/her gaze and following the sequential displacement of the red dot. The disappearance and appearance of the red dot were manually controlled by an experimenter; the experimenter changed the location of the red dot when saccadic movement of the participant was observed. When the red dot was displaced, a beep was presented to the participants to notify them of this displacement. In this instruction session, the participants only eye-wrote the number ‘0.’ Please note that the purpose of this session was to let the participants know how to eye-write a number with their eye-movements. The experimenter could sometimes misrecognize the participants’ eye-movements, but such a little misrecognition did not significantly affect the training of eye-writing.

In the practice session, the participants were asked to eye-write all numbers from ‘0’ to ‘9.’ The procedure of the practice session was the same as in the instruction session, except that the position of the red dot shifted automatically. The participants were given a short rest of 11 s before eye-writing next the number in this session. The exercise session was the same as the practice session, except that the time frame for eye-writing a number was fixed to 5 s and no guide line or red dot was displayed (only dots on a 3 × 3 grid were displayed). A 9.5-s short break was given between consecutive eye-writings to prevent eye fatigue of the participants for the exercise session. This session was repeated three times. However, participant No. 19 only performed the session twice because of fatigue. The overall experimental procedures of the three sessions—instruction, practice, and exercise—were verbally explained to the participants (see Appendix for further details).

EOG data were obtained during the exercise sessions, and the eye movement traces were reconstructed using signal processing techniques introduced in previous studies [17, 21‐23] (see Fig. 4 for the overall procedure). First, source signals were down-sampled at a sampling rate of 64 Hz and median-filtered. High-frequency noises were removed in this step. Second, eye blink artifacts were automatically detected and removed as described in [23]. The time interval, including eye blink artifacts, was determined using the following equation:where f(t) is the output of a digital filter that emphasizes eye blink signals [24], and T(Max _i ) and T(Min _i ) are time points of the ith local maximum and minimum, respectively. Third, baselines were removed for each signal channel. A median value calculated from 100 ms of preceding data was used as the baseline of each eye-written number. Fourth, the horizontal and vertical EOG components were obtained. The horizontal EOG component was obtained by subtracting the EOG signal of the left channel from that of the right channel, whereas the vertical component was obtained by subtracting the EOG signal acquired at the channel below the eye from the corresponding channel above the eye. The fifth and sixth steps consisted of signal interpolation due to eye blinks and the removal of low-frequency monotonic drifts, respectively. Specifically, missing data due to the eye blink removal process were linearly interpolated using adjacent signals, and the low-frequency drifts were removed using linear regression. Saccadic eye movements were then detected using the continuous wavelet transform-saccade detection algorithm introduced by Bulling et al. [22], which extracts signals whose absolute wavelet coefficient values exceed a preset threshold (θ). The wavelet coefficient \( {C}_b^a \) of data s at scale a and position b is defined aswhere ψ represents a Haar mother wavelet. The wavelet scale was set to 20, as proposed by Bulling et al. The threshold value was derived by collecting data from an additional participant (male, 24 years old) who did not participate in the main experiments. To derive the threshold value, saccadic regions in the data were manually labeled, and the minimum value of the wavelet coefficient within the saccadic region was used as the threshold. After saccade detection, the signals were resampled to have the same Euclidean distance between adjacent points, and the signal sizes were normalized to make both the width and height to be one. In addition, the interdependency between horizontal and vertical components was removed without any calibration data, as follows:where EOG ^c _v is the compensated vertical component, α is a parameter that describes the amount of interdependency between the horizontal and vertical components, and EOG _v and EOG _h are the vertical and horizontal EOG components, respectively.

Unlike the conventional method that requires some preliminary calibration data [23], we developed a new method for estimating α without any calibration. Data in a sequence were differentiated by a finite difference function:

The value of α can be estimated even in the absence of calibration data, simply by linearly regressing the values in the differentiated data. The processes for estimating α and removing the interdependency are described in Fig. 5. Figure 5b shows an example of an eye-written character ‘3,’ where the eye-written shape is severely skewed compared with the template pattern (Fig. 5a). We found that the degree of skewness can be well described as the linear regression slope of data distribution in the finite difference space of adjacent values (Fig. 5d). The parameter α was determined as the value that makes the slope to be zero, as shown in Fig. 5e. Using the determined α value, the eye-written pattern became more similar to the template pattern (Fig. 5c).

We tested four different types of classifiers for the proposed system: dynamic time warping (DTW), dynamic positional warping (DPW), a combination of DTW with support vector machine (SVM), and a combination of DPW with SVM. DTW measures dissimilarity between two time-series with different lengths and is often utilized for handwritten character recognition or signature verification [25, 26]. DPW is an extended version of DTW that has been specialized for two-dimensional shapes by allowing warping on value axes [27]. Conventional DTW/DPW have utilized a very simple classifier which just choose one with the minimum distance from the reference. In this study, we tried to verify whether the combination of the DTW/DPW with a more advanced classifier could increase recognition accuracy. In this study, SVM was chosen as it is a well-known classifier that is used in many research fields [22, 28‐30].

Since it is not yet possible to use the dissimilarities obtained from DTW or DPW as features for SVM, we introduced a new way of combining the DTW/DPW and SVM.

In DTW, the dissimilarities between two signals (A and B) are defined as follows: where L _A and L _B are the lengths of two signals, i _prev  = i − 1, j _prev  = j − 1, τ(1, 1) = 0, and C _A and C _B are the pairs of constraints for warping time. The constraints (C _A , B ) are defined aswhere M is the maximum distance for time-warping.

In DPW, the definition of dissimilarity is similar to that in DTW. Specifically, dissimilarity can be written by substituting i _prev and j _prev , as follows:

Detailed descriptions of DTW and DPW can be found in [23, 27].

To combine DTW/DPW with SVM, we used a set of normalized dissimilarities to the templates as a feature vector for SVM. Dissimilarities to all templates were calculated for a test signal using DTW/DPW. These dissimilarities were then divided by the normalization factors of each template. These normalization factors were calculated using training data. Specifically, dissimilarities to the corresponding template were calculated for each of the training datasets, and the mean dissimilarity in a class was utilized as the normalization factor of the class [31].

We evaluated the performance of each method by leave-one-subject-out testing to verify whether our eye-writing approach is user-independent. Each participant’s data were tested individually, where data from all healthy participants except the participant being tested were used for training. Data from participants with ALS were not used to train the models due to the limited sample size. All the data during the exercise sessions were used for the validation. Please note that three sets of eye-written numbers were collected per participant during the exercise session, except the participant #19.

Recognition performance was evaluated in three different ways to express different aspects of the results. First, the overall recognition performance was measured as follows:where TP _overall denotes the number of patterns (eye-written numbers) that were correctly classified, and P _overall is the total number of patterns. The overall performance was measured for each group (healthy or ALS) to enable comparison of the performance of participants with ALS versus that of healthy participants. Second, the recognition performance were evaluated for each participant as follows:where TP _participant denotes the number of the patterns that were correctly classified for a given participant, and P _participant is the total number of patterns attempted by the participant. The performance was also evaluated for each of the Arabic numbers, as follows:

where TP, FP, and FN represent the true positive, false positive, and false negative rates, respectively.