Citation: Arif, Z.; Fu, Y. Mix Frame
Visual Servo Control Framework for
Autonomous Assistive Robotic Arms.
Sensors 2022, 22, 642. https://
doi.org/10.3390/s22020642
Academic Editors: Abolfazl Zaraki
and Hamed Rahimi Nohooji
Received: 4 November 2021
Accepted: 4 January 2022
Published: 14 January 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Mix Frame Visual Servo Control Framework for Autonomous
Assistive Robotic Arms
Zubair Arif and Yili Fu *
State Key Laboratory of Robotics and Systems, Harbin Institute of Technology, Harbin 150001, China;
zub@hit.edu.cn or zubair_97@yahoo.com
* Correspondence: meylfu@hit.edu.cn
Abstract:
Assistive robotic arms (ARAs) that provide care to the elderly and people with disabilities,
are a significant part of Human-Robot Interaction (HRI). Presently available ARAs provide non-
intuitive interfaces such as joysticks for control and thus, lacks the autonomy to perform daily
activities. This study proposes that, for inducing autonomous behavior in ARAs, visual sensors
integration is vital, and visual servoing in the direct Cartesian control mode is the preferred method.
Generally, ARAs are designed in a configuration where its end-effector’s position is defined in
the fixed base frame while orientation is expressed in the end-effector frame. We denoted this
configuration as ‘mixed frame robotic arms’. Consequently, conventional visual servo controllers
which operate in a single frame of reference are incompatible with mixed frame ARAs. Therefore,
we propose a mixed-frame visual servo control framework for ARAs. Moreover, we enlightened
the task space kinematics of a mixed frame ARAs, which led us to the development of a novel
“mixed frame Jacobian matrix”. The proposed framework was validated on a mixed frame JACO-2
7 DoF ARA using an adaptive proportional derivative controller for achieving image-based visual
servoing (IBVS), which showed a significant increase of 31% in the convergence rate, outperforming
conventional IBVS joint controllers, especially in the outstretched arm positions and near the base
frame. Our Results determine the need for the mixed frame controller for deploying visual servo
control on modern ARAs, that can inherently cater to the robotic arm’s joint limits, singularities, and
self-collision problems.
Keywords:
visual servo control; assistive robotic arms; IBVS; autonomous robots; mix frame
Jacobian matrix
1. Introduction
The ultimate goal of science and engineering is to serve humanity and humans by
creating ease in their daily lives. Robotics is an innovative engineering discipline that does
so by automatically performing repetitive, laborious, and complex tasks, providing relief to
humans. Recently, with the advancement in robotic technologies, the acceptance of robots
in society has improved considerably, resulting in an increased human-robot interaction
application [
1
]. Large volumes of robots are not only brought into the industry, but are also
introduced in dynamic environments that were originally designed for humans, such as
in homes, schools, and hospitals [
2
]. As these environments are dynamic in nature, such
workplaces demand a high level of autonomy and dexterity, which needs to be developed
in robots to perform their task autonomously [3].
Vision is the fundamental sensor that humans use to perceive, adapt, and work in
dynamic environments. Henceforth, robotic vision has emerged as the vital tool for robots
to perceive the environment and acquire autonomy to perform their tasks in human-centric
environments in human-robot interactions (HRIs) [4].
One key area of HRI is providing care to patients, the elderly, and people with disabili-
ties. Recently, assistive robotic arms (ARAs), a form of robotic assistive care have gained
Sensors 2022, 22, 642. https://doi.org/10.3390/s22020642 https://www.mdpi.com/journal/sensors
