Citation: Jia, J.; Wang, Z.; Xu, L.; Dai,
J.; Gu, M.; Huang, J. An
Interference-Resistant and
Low-Consumption Lip Recognition
Method. Electronics 2022, 11, 3066.
https://doi.org/10.3390/
electronics11193066
Academic Editor: Silvia Liberata Ullo
Received: 25 August 2022
Accepted: 20 September 2022
Published: 26 September 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
An Interference-Resistant and Low-Consumption Lip
Recognition Method
Junwei Jia
1
, Zhilu Wang
2
, Lianghui Xu
1
, Jiajia Dai
1
, Mingyi Gu
1
and Jing Huang
1,
*
1
School of Information and Electronic Engineering, Zhejiang Gongshang University (ZJSU),
Hangzhou 310018, China
2
College of Mechanical and Electrical Engineering, Hohai University (HHU), Changzhou 213022, China
* Correspondence: jhuang@mail.zjgsu.edu.cn
Abstract:
Lip movements contain essential linguistic information. It is an important medium for
studying the content of the dialogue. At present, there are many studies on how to improve the
accuracy of lip language recognition models. However, there are few studies on the robustness and
generalization performance of the model under various disturbances. Specific experiments show
that the current state-of-the-art lip recognition model significantly drops in accuracy when disturbed
and is particularly sensitive to adversarial examples. This paper substantially alleviates this problem
by using Mixup training. Taking the model subjected to negative attacks generated by FGSM as an
example, the model in this paper achieves 85.0% and 40.2% accuracy on the English dataset LRW
and the Mandarin dataset LRW-1000, respectively. The correct recognition rates are improved by
9.8% and 8.3%, compared with the current advanced lip recognition models. The positive impact of
Mixup training on the robustness and generalization of lip recognition models is demonstrated. In
addition, the performance of the lip recognition classification model depends more on the training
parameters, which increase the computational cost. The InvNet-18 network in this paper reduces the
consumption of GPU resources and the training time while improving the model accuracy. Compared
with the standard ResNet-18 network used in mainstream lip recognition models, the InvNet-18
network in this paper has more than three times lower GPU consumption and 32% fewer parameters.
After detailed analysis and comparison in various aspects, it is demonstrated that the model in
this paper can effectively improve the model’s anti-interference ability and reduce training resource
consumption. At the same time, the accuracy is comparable with the current state-of-the-art results.
Keywords:
lip recognition; visual speech recognition; data enhancement; inverse convolutional
neural network
1. Introduction
Lip recognition is also called visual speech recognition (VSR). It analyzes the dynamic
changes of the lips. The aim is to recognize the speech content in the video. This task
involves natural language processing, image classification, speech processing, and pat-
tern recognition. In recent years, there have been many applications of lip recognition in
real life such as in vivo detection [
1
], improved hearing aids [
2
], etc., with broad applica-
tion prospects.
The lip recognition model consists of two steps. The first is to extract the visual fea-
tures of the lips. The second is categorization. The extracted visual features should contain
sufficient representative information and robustness [
3
]. The traditional extraction method
is manual annotation. Such practices only ensure that the downstream task can be classified
and recognized without considering the effectiveness of the acquired features. Therefore,
the recognition accuracy is low. Although, there are corresponding methods [
4
,
5
] to solve
this problem. These methods rely on manual design, and the design process is complex.
Visual features obtained using manual annotation do not meet human expectations, and re-
searchers have begun to seek more effective visual elements. Deep learning techniques
Electronics 2022, 11, 3066. https://doi.org/10.3390/electronics11193066 https://www.mdpi.com/journal/electronics
