Background
Methods
Dataset and feature extraction
Experimental data
Online subjective annotation | |
Number of videos | 120 |
Video duration | 1 min affective highlight |
Selection method | 60 via last.fm affective tags, 60 manually selected |
Number of ratings per video | 14–16 |
Rating scales | Arousal, Valence and Dominance |
Rating values | 1–9 |
Physiological experiment | |
Number of subjects | 32 |
Number of videos | 40 |
Selection method | Subset of online annotated videos with clearest responses |
Rating scales | Arousal, Valence, Dominance, Liking and Familiarity |
Rating values | Familiarity: discrete scale of 1-5, others: continuous scale of 1-9 |
Recorded signals | 32-channel 512Hz EEG, peripheral physiological signals, face video from 22 subjects |
Feature extraction
Feature index | Notation of the extracted features |
---|---|
No. 1-448 EEG time and frequency-domain features (14 feature types × 32 channels) | Mean, Var, peak-to-peak amplitude, Skewness, Kurtosis |
Average PSD in theta (4-7 Hz), alpha (8-15 Hz),beta (16-31 Hz), gamma (32-45 Hz), beta/theta, beta/alpha | |
Three Hjorth parameters: mobility, activity and complexity | |
No. 449-504 EEG hemispheric asymmetry (4 feature types × 14 channel pairs) | Difference of average PSD in theta, alpha, beta and gamma bands for 14 channel pairs between right and left scalp |
No. 505-600 EEG nonlinear features (3 feature types × 32 channels) | Spectral Entropy, Shannon Entropy and C0 complexity |
No. 601-608 EOG features (4 feature types × 2 channels) | Mean, Var, peak-to-peak amplitude, Energy |
No. 609-642 EMG features (17 feature types × 2 channels) | Mean, Var, Total spectral power |
1Diff-Mean, 1Diff-Median, 1Diff-Min, 1Diff-Var, 1Diff-Max, 1Diff-MinRatio, 1Diff-MaxRatio | |
2Diff-Mean, 2Diff-Median, 2Diff-Min, 2Diff-Var, 2Diff-Max, 2Diff-MinRatio, 2Diff-MaxRatio | |
No. 643-646 TMP features (4 feature types × 1 channel) | Mean, 1Diff-Mean, Spectral power in the bands (0-0.1 Hz) and (0.1-0.2 Hz) |
No. 647-666 BVP features (20 feature types × 1 channel) | Hr-Mean, Hr-Var, Hr-Range |
Hrv-Mean, Hrv-Var, Hrv-Min, Hrv-Max, Hrv-Range, Hrv-pNN50 | |
HrvDistr-Mean, HrvDistr-Median, HrvDistr-Var, HrvDistr-Min, HrvDistr-Max, HrvDistr-Range, HrvDistr-Triind | |
PSD in bands (0-0.2 Hz), (0.2-0.4 Hz), (0.4-0.6 Hz), and (0.6-0.8 Hz) of Hrv | |
No. 667-721 RSP features (55 feature types × 1 channel) | Mean, Var, Range, MaxRatio |
1Diff-Mean, 1Diff-Median, 1Diff-Var, 1Diff-Range, 1Diff-MaxRatio | |
2Diff-Mean, 2Diff-Median, 2Diff-Var, 2Diff-Range, 2Diff-MaxRatio | |
RSPPulse-Mean, RSPPulse-Var, RSPPulse-Range, RSPPulse-MaxRatio | |
RSPPulse-1Diff-Mean, RSPPulse-1Diff-Median, RSPPulse-1Diff-Var, RSPPulse-1Diff-Min, RSPPulse-1Diff-Max, RSPPulse-1Diff-Range, RSPPulse-1Diff-MaxRatio | |
RSPPulse-2Diff-Mean, RSPPulse-2Diff-Median, RSPPulse-2Diff-Var, RSPPulse-2Diff-Min, RSPPulse-2Diff-Max, RSPPulse-2Diff-Range, RSPPulse-2Diff-MaxRatio | |
RSPAmpl-Mean, RSPAmpl-Var, RSPAmpl-Range, RSPAmpl-MaxRatio | |
RSPAmpl-1Diff-Mean, RSPAmpl-1Diff-Median, RSPAmpl-1Diff-Var, RSPAmpl-1Diff-Min, RSPAmpl-1Diff-Max, RSPAmpl-1Diff-Range, RSPAmpl-1Diff-MaxRatio | |
RSPAmpl-2Diff-Mean, RSPAmpl-2Diff-Median, RSPAmpl-2Diff-Var, RSPAmpl-2Diff-Min, RSPAmpl-2Diff-Max, RSPAmpl-2Diff-Range, RSPAmpl-2Diff-MaxRatio | |
PSD in the bands (0-0.1 Hz), (0.1-0.2 Hz), (0.2-0.3 Hz), and (0.3-0.4 Hz), Ratio of PSD in the band (0-0.25 Hz) to PSD in the band (0.25-0.45 Hz) | |
No. 722-742 GSR features (21 feature types × 1 channel) | Rising time, Decay time |
Sc-Mean, Sc-Median, Sc-Var, Sc-MinRatio, Sc-MaxRatio | |
Sc-1Diff-Mean, Sc-1Diff-Median, Sc-1Diff-Var, Sc-1Diff-Min, Sc-1Diff-Max, Sc-1Diff-MinRatio, Sc-1Diff-MaxRatio | |
Sc-2Diff-Mean, Sc-2Diff-Median, Sc-2Diff-Var, Sc-2Diff-Min, Sc-2Diff-Max, Sc-2Diff-MinRatio, Sc-2Diff-MaxRatio |
The three-stage decision method
The first stage
The second stage
The third stage
Results
Data partitioning during the training process
Optimal features and classifier parameters identified during the training process
Decision results in each stage
Strategy | Recognition accuracy | ||||
---|---|---|---|---|---|
EQ1 | EQ2 | EQ3 | EQ4 | Average | |
Two pools: HA and LA | 86.67% | 80.00% | 30.56% | 58.33% | 77.57% |
Two pools: HV and LV | 33.33% | 83.33% | 50.55% | 50.00% | 43.57% |
Discussion
A comparison with other multiclass classification methods
Method | Parameters | Description | Accuracy | |
---|---|---|---|---|
1 |
k-NN |
k=1 | 44.30% | |
SVM | Linear kernel, cost C=200, tolerance E=0.001, epsilon for the loss function P=1.0E-12 | One sample from one trial | 51.10% | |
C4.5 | - | 47.24% | ||
Random forest | - | 46.69% | ||
2 | One-against-Rest |
k-NN, majority voting | 29 samples from one trial | 51.13% |
3 | One-against-One |
k-NN, majority voting | 52.16% | |
Three-stage decision | Two pools: HA and LA | 77.57% |