Citation: Hu, Z.; Chen, L.; Luo, Y.;
Zhou, J. EEG-Based Emotion
Recognition Using Convolutional
Recurrent Neural Network with
Multi-Head Self-Attention. Appl. Sci.
2022, 12, 11255. https://doi.org/
10.3390/app122111255
Academic Editors: Krzysztof
Ejsmont, Aamer Bilal Asghar,
Yong Wang and Rodolfo Haber
Received: 10 October 2022
Accepted: 28 October 2022
Published: 6 November 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
EEG-Based Emotion Recognition Using Convolutional
Recurrent Neural Network with Multi-Head Self-Attention
Zhangfang Hu
1
, Libujie Chen
1,2,
*, Yuan Luo
1,2
and Jingfan Zhou
1
1
Key Laboratory of Optoelectronic Information Sensing and Technology, Chongqing University of Posts and
Telecommunications, Chongqing 400065, China
2
School of Advanced Manufacturing Engineering, Chongqing University of Posts and Telecommunications,
Chongqing 400065, China
* Correspondence: s200431009@stu.cqupt.edu.cn
Featured Application: The proposed method in this study can be used in EEG emotion recognition
and achieve better results.
Abstract:
In recent years, deep learning has been widely used in emotion recognition, but the
models and algorithms in practical applications still have much room for improvement. With the
development of graph convolutional neural networks, new ideas for emotional recognition based on
EEG have arisen. In this paper, we propose a novel deep learning model-based emotion recognition
method. First, the EEG signal is spatially filtered by using the common spatial pattern (CSP), and
the filtered signal is converted into a time–frequency map by continuous wavelet transform (CWT).
This is used as the input data of the network; then the feature extraction and classification are
performed by the deep learning model. We called this model CNN-BiLSTM-MHSA, which consists
of a convolutional neural network (CNN), bi-directional long and short-term memory network
(BiLSTM), and multi-head self-attention (MHSA). This network is capable of learning the time series
and spatial information of EEG emotion signals in depth, smoothing EEG signals and extracting deep
features with CNN, learning emotion information of future and past time series with BiLSTM, and
improving recognition accuracy with MHSA by reassigning weights to emotion features. Finally,
we conducted experiments on the DEAP dataset for sentiment classification, and the experimental
results showed that the method has better results than the existing classification. The accuracy of
high and low valence, arousal, dominance, and liking state recognition is 98.10%, and the accuracy of
four classifications of high and low valence-arousal recognition is 89.33%.
Keywords:
EEG; emotion recognition; CNN; BiLSTM; multi-head self-attention; time–frequency map
1. Introduction
Emotion plays an important role in daily human life and influences all aspects. Emo-
tion is an indispensable and important role for humans that affects people all the time,
including human decision-making, speech, sleep, health, communication, and various
other characteristics. Emotion recognition is often implemented based on facial expressions,
speech, and physiological signals, while physiological signals more accurately reflect fluc-
tuations in human emotional states [
1
] and are often used in the field of human-computer
interaction. In recent years, emotion recognition based on electroencephalography (EEG) [
2
]
in physiological signals has had widespread applications because of its non-invasive, easy-
to-use, and inexpensive characteristics.
EEG is an electrical signal of the human brain epidermis, which has nonlinear and
non-smooth characteristics, while feature extraction and classification of such signals have
been a challenge for researchers. Many researchers have proposed their feature extraction
algorithms based on traditional methods [
3
–
6
] emotion recognition methods, and deep
learning emotion recognition methods based on convolutional neural networks [
7
], deep
Appl. Sci. 2022, 12, 11255. https://doi.org/10.3390/app122111255 https://www.mdpi.com/journal/applsci
