Citation: Lee, C.; Hong, J.; Jung, H.
N-Step Pre-Training and
Décalcomanie Data Augmentation
for Micro-Expression Recognition.
Sensors 2022, 22, 6671. https://
doi.org/10.3390/s22176671
Academic Editors: M. Jamal Deen,
Subhas Mukhopadhyay, Yangquan
Chen, Simone Morais, Nunzio
Cennamo and Junseop Lee
Received: 11 July 2022
Accepted: 28 August 2022
Published: 3 September 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
N-Step Pre-Training and Décalcomanie Data Augmentation for
Micro-Expression Recognition
Chaehyeon Lee , Jiuk Hong and Heechul Jung *
Department of Artificial Intelligence, Kyungpook National University, Daegu 41566, Korea
* Correspondence: heechul@knu.ac.kr; Tel.: +82-53-950-4558
Abstract:
Facial expressions are divided into micro- and macro-expressions. Micro-expressions are
low-intensity emotions presented for a short moment of about 0.25 s, whereas macro-expressions
last up to 4 s. To derive micro-expressions, participants are asked to suppress their emotions as
much as possible while watching emotion-inducing videos. However, it is a challenging process, and
the number of samples collected tends to be less than those of macro-expressions. Because training
models with insufficient data may lead to decreased performance, this study proposes two ways to
solve the problem of insufficient data for micro-expression training. The first method involves
N
-step
pre-training, which performs multiple transfer learning from action recognition datasets to those in
the facial domain. Second, we propose Décalcomanie data augmentation, which is based on facial
symmetry, to create a composite image by cutting and pasting both faces around their center lines.
The results show that the proposed methods can successfully overcome the data shortage problem
and achieve high performance.
Keywords:
deep learning; image processing; facial micro-expression; emotion recognition; convolu-
tional neural network (CNN)
1. Introduction
Humans reveal personal feelings, intentions, and emotional conditions through their
facial expressions. Generally, a person reveals emotions through explicit macro-expressions
that last between 0.25 and 4 s. During these periods, the emotions expressed on the face
and the actual feelings felt coincide. Conversely, when a person unconsciously reveals a
hidden emotion in fractional time (e.g., 0.25 s), it is considered to be a micro-expression.
These are likely to be missed or misinterpreted, even in laboratory settings. Figure 1 shows
a comparison between the micro- and macro-expressions.
Figure 1.
Examples of macro-expression (
left
) and micro-expression (
right
). Macro-expression
samples have high intensity, while micro-expression samples show little change in facial expression.
In 1966, Haggard et al. [
1
] first proposed the concept of micro-expressions. About three
years later, Ekman et al. [
2
] witnessed this phenomenon while researching lie detection
using interview videos of psychologists and patients. Robust micro-expression recogni-
tion systems are used in various fields, such as criminal recognition, lie detection, and
psychological diagnosis. Owing to the broad applicability of micro-expression recognition,
many studies have been conducted in recent years. These studies are primarily divided
Sensors 2022, 22, 6671. https://doi.org/10.3390/s22176671 https://www.mdpi.com/journal/sensors
