Citation: Su, Y.-P.; Chen, X.-Q.; Zhou,
T.; Pretty, C.; Chase, G.
Mixed-Reality-Enhanced
Human–Robot Interaction with an
Imitation-Based Mapping Approach
for Intuitive Teleoperation of a
Robotic Arm-Hand System. Appl. Sci.
2022, 12, 4740. https://doi.org/
10.3390/app12094740
Academic Editor: Jorge
Martin-Gutierrez
Received: 1 April 2022
Accepted: 5 May 2022
Published: 8 May 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Mixed-Reality-Enhanced Human–Robot Interaction with an
Imitation-Based Mapping Approach for Intuitive Teleoperation
of a Robotic Arm-Hand System
Yun-Peng Su
1
, Xiao-Qi Chen
2
, Tony Zhou
1
, Christopher Pretty
1
and Geoffrey Chase
1,
*
1
Mechanical Engineering Deprtment, College of Engineering, University of Canterbury,
Christchurch 8041, New Zealand; yunpeng.su@pg.canterbury.ac.nz (Y.-P.S.);
tony.zhou@canterbury.ac.nz (T.Z.); chris.pretty@canterbury.ac.nz (C.P.)
2
Manufacturing Futures Research Institute (MFRI), Swinburne University of Technology,
Melbourne 3122, Australia; xiaoqichen@swin.edu.au
* Correspondence: geoff.chase@canterbury.ac.nz; Tel.: +64-336-92-182
Abstract:
This paper presents an integrated mapping of motion and visualization scheme based on a
Mixed Reality (MR) subspace approach for the intuitive and immersive telemanipulation of robotic
arm-hand systems. The effectiveness of different control-feedback methods for the teleoperation
system is validated and compared. The robotic arm-hand system consists of a 6 Degrees-of-Freedom
(DOF) industrial manipulator and a low-cost 2-finger gripper, which can be manipulated in a natural
manner by novice users physically distant from the working site. By incorporating MR technology, the
user is fully immersed in a virtual operating space augmented by real-time 3D visual feedback from
the robot working site. Imitation-based velocity-centric motion mapping is implemented via the MR
subspace to accurately track operator hand movements for robot motion control and enables spatial
velocity-based control of the robot Tool Center Point (TCP). The user control space and robot working
space are overlaid through the MR subspace, and the local user and a digital twin of the remote robot
share the same environment in the MR subspace. The MR-based motion and visualization mapping
scheme for telerobotics is compared to conventional 2D Baseline and MR tele-control paradigms over
two tabletop object manipulation experiments. A user survey of 24 participants was conducted to
demonstrate the effectiveness and performance enhancements enabled by the proposed system. The
MR-subspace-integrated 3D mapping of motion and visualization scheme reduced the aggregate
task completion time by 48% compared to the 2D Baseline module and 29%, compared to the MR
SpaceMouse module. The perceived workload decreased by 32% and 22%, compared to the 2D
Baseline and MR SpaceMouse approaches.
Keywords: mixed reality; teleoperation; 3D mapping; robotic manipulation
1. Introduction
With the rapid development of space exploration, deep-sea discovery, nuclear rescue,
radiation detection, and robot-assisted medical equipment in recent years, humans urgently
need interactive control of slave robots to complete remote operations [
1
,
2
]. More recently,
medical robotic applications during the coronavirus pandemic have proven valuable [
3
,
4
].
Due to the highly contagious nature of the novel coronavirus, surgeons are at high risk of
infection when examining and sampling patients face-to-face [
5
]. However, oropharyngeal
swabbing is a commonly used technique for COVID-19 sampling and diagnosis in the
pandemic worldwide [
6
,
7
]. One application scenario of medical telerobotic systems is to
teleoperate robots to conduct COVID-19 swab testing and provide other healthcare services,
such as (1) robotics-assisted telesurgery, (2) tele-examination of patients before and after
treatment, and (3) tele-training for surgical procedures [
8
,
9
]. On the user side of biomedical
telerobotic systems, surgeons can operate a Human–Robot Interaction (HRI) system with an
Appl. Sci. 2022, 12, 4740. https://doi.org/10.3390/app12094740 https://www.mdpi.com/journal/applsci
