Citation: Shang, Y.; Yang, M.; Cui, J.;
Cui, L.; Huang, Z.; Li, X. Driver
Emotion and Fatigue State Detection
Based on Time Series Fusion.
Electronics 2023, 12, 26. https://
doi.org/10.3390/electronics12010026
Academic Editors: Silvia
Liberata Ullo and Donghyeon Cho
Received: 7 November 2022
Revised: 28 November 2022
Accepted: 16 December 2022
Published: 21 December 2022
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Driver Emotion and Fatigue State Detection Based on Time
Series Fusion
Yucheng Shang
1
, Mutian Yang
2
, Jianwei Cui
1,
*, Linwei Cui
1
, Zizheng Huang
1
and Xiang Li
1
1
Institute of Instrument Science and Engineering, Southeast University, Nanjing 210096, China
2
School of Information Science and Engineering, China University of Petroleum, Beijing 266580, China
* Correspondence: cjw@seu.edu.cn
Abstract:
Studies have shown that driver fatigue or unpleasant emotions significantly increase
driving risks. Detecting driver emotions and fatigue states and providing timely warnings can
effectively minimize the incidence of traffic accidents. However, existing models rarely combine
driver emotion and fatigue detection, and there is space to improve the accuracy of recognition.
In this paper, we propose a non-invasive and efficient detection method for driver fatigue and
emotional state, which is the first time to combine them in the detection of driver state. Firstly, the
captured video image sequences are preprocessed, and Dlib (image open source processing library)
is used to locate face regions and mark key points; secondly, facial features are extracted, and fatigue
indicators, such as driver eye closure time (PERCLOS) and yawn frequency are calculated using the
dual-threshold method and fused by mathematical methods; thirdly, an improved lightweight RM-
Xception convolutional neural network is introduced to identify the driver’s emotional state; finally,
the two indicators are fused based on time series to obtain a comprehensive score for evaluating the
driver’s state. The results show that the fatigue detection algorithm proposed in this paper has high
accuracy, and the accuracy of the emotion recognition network reaches an accuracy rate of 73.32% on
the Fer2013 dataset. The composite score calculated based on time series fusion can comprehensively
and accurately reflect the driver state in different environments and make a contribution to future
research in the field of assisted safe driving.
Keywords:
fatigue driving detection; time series; xception network; sentiment computing; metric fu-
sion
1. Introduction
According to statistics, the number of deaths caused by traffic accidents has reached
1.35 million worldwide [
1
], and among them, traffic accidents caused by fatigued driving
account for approximately 20%–30% of all traffic accidents [
2
]. Studies have shown that
uncontrollable emotions are one of the primary elements that raise driving risks [
3
], such
as anger that may cause road rage [
4
], and sadness and stress that can reduce driver
concentration [
5
]. Approximately 90% of traffic accidents can be avoided if drivers are
warned before they occur [
6
]. Therefore, in order to reduce and avoid traffic accidents, it is
significant to identify driver fatigue and emotional state and warn them with the help of
assisted driving systems.
The existing driver fatigue detection methods can be divided into three main cate-
gories: based on vehicle behavior [
7
], based on physiological signals [
8
–
11
], and based on
visual features [
12
–
14
]. Based on vehicle behavior, the driver’s fatigue can be indirectly
judged, such as whether the car is pressing the line, whether the distance between the
car and another car is too close, etc. However, due to the complex road conditions of the
actual scene and the great differences in drivers’ driving habits, it is difficult to develop
a unified standard to determine fatigue, and its main drawback is the low accuracy rate;
common physiological signals used to detect fatigue include electrocardiogram (ECG),
Electronics 2023, 12, 26. https://doi.org/10.3390/electronics12010026 https://www.mdpi.com/journal/electronics
