Citation: Zhang, J.; Peng, G.; Yang,
H.; Tan, C.; Tan, Y.; Bai, H. Real-Time
Finger-Writing Character Recognition
via ToF Sensors on Edge Deep
Learning. Electronics 2023, 12, 685.
https://doi.org/10.3390/
electronics12030685
Academic Editor: Jaime Lloret
Received: 28 December 2022
Revised: 18 January 2023
Accepted: 23 January 2023
Published: 30 January 2023
Copyright: © 2023 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Real-Time Finger-Writing Character Recognition via ToF
Sensors on Edge Deep Learning
Jiajin Zhang
1
, Guoying Peng
1
, Hongyu Yang
2
, Chao Tan
3
, Yaqing Tan
1,
* and Hui Bai
4,
*
1
College of Big Data, Yunnan Agricultural University, Kunming 650201, China
2
College of Mechanical and Electrical Engineering, Yunnan Agricultural University, Kunming 650201, China
3
Key Lab of Process Analysis and Control of Sichuan Universities, Yibin University, Yibin 644001, China
4
College of Architectural Engineering, Yunnan Agricultural University, Kunming 650201, China
* Correspondence: yatsingt@ynau.edu.cn (Y.T.); zjjclc@ynau.edu.cn (H.B.)
Abstract:
Human–computer interaction is demanded for natural and convenient approaches, in
which finger-writing recognition has aroused more and more attention. In this paper, a device-free
finger-writing character recognition system based on an array of time-of-flight (ToF) distance sensors
is presented. The ToF sensors acquire distance values between sensors to a writing finger within
a 9.5
×
15 cm square on a surface at specific time intervals and send distance data to a low-power
microcontroller STM32F401, equipped with deep learning algorithms for real-time inference and
recognition tasks. The proposed method enables one to distinguish 26 English lower-case letters
by users writing with their fingers and does not require one to wear additional devices. All data
used in this work were collected from 21 subjects (12 males and 9 females) to evaluate the proposed
system in a real scenario. In this work, the performance of different deep learning algorithms, such as
long short-term memory (LSTM), convolutional neural networks (CNNs) and bidirectional LSTM
(BiLSTM), was evaluated. Thus, these algorithms provide high accuracy, where the best result is
extracted from the LSTM, with 98.31% accuracy and 50 ms of maximum latency.
Keywords: finger-writing character recognition; time of flight; edge deep learning; distance sensor
1. Introduction
Text input, transferring words in mind into digital information, occurs frequently in
our day-to-day activities. Typing, the most used method for the majority of individuals in
human–computer interaction, requires dedicated writing instruments, such as keyboards
or touch screens, which are not always available in everyday life [
1
]. W. Chen et al. [
2
]
presented an on-body typing system that allowed users to type on the backs of their hands
on a T9-shaped keyboard. It is an advance in typing systems, but typing with one hand
on a T9 keyboard is still awkward and slow. As a replacement for typing, speech/voice
recognition is more friendly for those who struggle with keyboards and has been developed
over many years [
3
,
4
]. However, it is sensitive to noise levels in the surroundings and could
disclose sensitive information [
5
]. To fill this research gap, Silent Speech Interface (SSI)
using deep neural network models and ultra-sound images to monitor a user’s unvoiced
utterance and convert it into speech signals was proposed [
6
,
7
]. Unfortunately, real-time
articulatory-to-acoustic mapping has not been accomplished by SSI.
Handwriting recognition is currently receiving more and more attention as a way to
go beyond the constraints of typing and speech recognition. As a part of our body, hands
and fingers are the earliest and most frequently used tool, and they are not an extra burden
for us. In early human history, our ancestors painted on cave walls with their hands and
fingers [8]. Various novel sensors have been applied in handwriting recognition to enable
users to write freely and naturally like our predecessors did, as well as to accurately read
what has been written. Inertial-based sensors are often used in wearable devices attached
to hands or fingers to gather precise body motion data [
9
–
11
]. Optical sensors provided a
Electronics 2023, 12, 685. https://doi.org/10.3390/electronics12030685 https://www.mdpi.com/journal/electronics
