Citation: Shao, Z.; Zhou, Y.; Zhu, H.;
Du, W.-L.; Yao, R.; Chen, H. Facial
Action Unit Recognition by Prior and
Adaptive Attention. Electronics 2022,
11, 3047. https://doi.org/10.3390/
electronics11193047
Academic Editor: Silvia Liberata Ullo
Received: 31 August 2022
Accepted: 21 September 2022
Published: 24 September 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Facial Action Unit Recognition by Prior and Adaptive Attention
Zhiwen Shao
1,2
, Yong Zhou
1,2,
* , Hancheng Zhu
1,2
, Wen-Liang Du
1,2
, Rui Yao
1,2
and Hao Chen
3
1
School of Computer Science and Technology, China University of Mining and Technology,
Xuzhou 221116, China
2
Engineering Research Center of Mine Digitization, Ministry of Education of the People’s Republic of China,
Xuzhou 221116, China
3
Xuzhou Guanglian Technology Co., Ltd., Xuzhou 221116, China
* Correspondence: yzhou@cumt.edu.cn
Abstract:
Facial action unit (AU) recognition remains a challenging task, due to the subtlety and
non-rigidity of AUs. A typical solution is to localize the correlated regions of each AU. Current works
often predefine the region of interest (ROI) of each AU via prior knowledge, or try to capture the
ROI only by the supervision of AU recognition during training. However, the predefinition often
neglects important regions, while the supervision is insufficient to precisely localize ROIs. In this
paper, we propose a novel AU recognition method by prior and adaptive attention. Specifically, we
predefine a mask for each AU, in which the locations farther away from the AU centers specified by
prior knowledge have lower weights. A learnable parameter is adopted to control the importance of
different locations. Then, we element-wise multiply the mask by a learnable attention map, and use
the new attention map to extract the AU-related feature, in which AU recognition can supervise the
adaptive learning of a new attention map. Experimental results show that our method (i) outperforms
the state-of-the-art AU recognition approaches on challenging benchmark datasets, and (ii) can
accurately reason the regional attention distribution of each AU by combining the advantages of both
the predefinition and the supervision.
Keywords: facial AU recognition; prior knowledge; adaptive attention
1. Introduction
Facial action unit (AU) recognition involves the prediction for occurrence or non-
occurrence of each AU, which is an important task in the communities of computer vision
and affective computing [
1
–
4
]. As defined in the facial action coding system (FACS) [
5
,
6
],
each AU is a local facial action with one or more atomic muscle actions. Due to the
subtlety and non-rigidity, the appearance of AUs are diversely changed across persons
and expressions. For instance, as shown in Figure 1, AU 1 (inner brow raiser), AU 2 (outer
brow raiser), and AU 4 (brow lowerer) occur in brow regions with overlaps, in which it
is difficult to distinguish each AU from the fused appearance. In the literature, facial AU
recognition is still a challenging task.
Considering AUs appear in local facial regions, one intuitive solution is to localize
the correlated regions so as to extract features for AU recognition. Since the locations
of AU centers can be specified by correlated facial landmarks via prior knowledge, Li
et al. [
2
,
7
] predefined a regional attention map for each AU, in which a position with a
farther distance to the AU centers is given a lower attention value. However, different
AUs share the same attention distribution, which ignores the divergences across AUs.
Furthermore, correlated landmarks only can determine the central locations of AUs, while
a few potentially correlated regions very far away from the centers are neglected.
Electronics 2022, 11, 3047. https://doi.org/10.3390/electronics11193047 https://www.mdpi.com/journal/electronics
