Citation: Cao, Z.; Bao, T.; Ren, Z.;
Fan, Y.; Deng, K.; Jia, W. Real-Time
Stylized Humanoid Behavior Control
through Interaction and
Synchronization. Sensors 2022, 22,
1457. https://doi.org/10.3390/
s22041457
Academic Editors: Abolfazl Zaraki
and Hamed Rahimi Nohooji
Received: 25 December 2021
Accepted: 9 February 2022
Published: 14 February 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Real-Time Stylized Humanoid Behavior Control through
Interaction and Synchronization
Zhiyan Cao
1
, Tianxu Bao
1
, Zeyu Ren
1
, Yunxin Fan
1
, Ken Deng
2
and Wenchuan Jia
1,
*
1
School of Mechatronic Engineering and Automation, Shanghai University, Shanghai 200444, China;
caozhiyan@shu.edu.cn (Z.C.); baotianxu@shu.edu.cn (T.B.); rzy007@shu.edu.cn (Z.R.);
fanyunxin@shu.edu.cn (Y.F.)
2
Institute of Wireless Theories and Technologies Lab, Beijing University of Posts and Telecommunications,
Beijing 100876, China; arieldeng@bupt.edu.cn
* Correspondence: lovvchris@shu.edu.cn
Abstract:
Restricted by the diversity and complexity of human behaviors, simulating a character to
achieve human-level perception and motion control is still an active as well as a challenging area.
We present a style-based teleoperation framework with the help of human perceptions and analyses
to understand the tasks being handled and the unknown environment to control the character. In
this framework, the motion optimization and body controller with center-of-mass and root virtual
control (CR-VC) method are designed to achieve motion synchronization and style mimicking while
maintaining the balance of the character. The motion optimization synthesizes the human high-level
style features with the balance strategy to create a feasible, stylized, and stable pose for the character.
The CR-VC method including the model-based torque compensation synchronizes the motion rhythm
of the human and character. Without any inverse dynamics knowledge or offline preprocessing, our
framework is generalized to various scenarios and robust to human behavior changes in real-time.
We demonstrate the effectiveness of this framework through the teleoperation experiments with
different tasks, motion styles, and operators. This study is a step toward building a human-robot
interaction that uses humans to help characters understand and achieve the tasks.
Keywords:
character animation; human-robot interaction; teleoperation control; human-in-the-
loop control
1. Introduction
In character animation and humanoid robotics, analyzing and rebuilding different
interactions between humans and the environment offers the opportunity to create alterna-
tive solutions for humanoid activities. In the last two decades, research on physics-based
humanoid animation and robotics improved rapidly, resulting in highly realistic and adap-
tive control achievements [
1
]. Despite this promising progress, the diversity and complexity
of human behaviors, which includes a great number of behavior patterns and uncertain
personal style preferences, have restricted the applications of previously proposed active
controllers. The schematic of interaction based on humanoid active intention is shown in
Figure 1. The human body’s central nervous system, sensing system, and musculoskeletal
energy system couple together to determine behavior performance. On the one hand, peo-
ple tend to make a general decision with a typical skeletal composition reacting to a regular
stimulus from the environment, which therefore performs a behavior pattern, that is, for
example, walking slowly after a meal. Rebuilding this pattern requires a comprehensive
analysis of the environment and social habits. On the other hand, human personality styles
generate tiny distinctions of this behavior pattern, such as one would like to walk in a
sneaky manner while another would like to swagger down the street. These distinctions
will cause an unstable disturbance for the pattern-behavior rebuilding. Not only that, but
with different characteristic body parameters between humanity and controlled characters,
Sensors 2022, 22, 1457. https://doi.org/10.3390/s22041457 https://www.mdpi.com/journal/sensors
