Article
A “Global–Local” Visual Servo System for
Picking Manipulators
Yinggang Shi
1,2,3
, Wei Zhang
1
, Zhiwen Li
1
, Yong Wang
1
, Li Liu
1,2,3
and Yongjie Cui
1,2,3,
*
1
College of Mechanical and Electronic Engineering, Northwest A&F University, Yangling 712100, China;
syg9696@nwsuaf.edu.cn (Y.S.); zx1314@nwafu.edu.cn (W.Z.); lizhiwen@nwafu.edu.cn (Z.L.);
yongwang@nwafu.edu.cn (Y.W.); liuli_ren_79@nwsuaf.edu.cn (L.L.)
2
Key Laboratory of Agricultural Internet of Things, Ministry of Agriculture and Rural, Yangling 712100, China
3
Shaanxi Key Laboratory of Agricultural Information Perception and Intelligent Service,
Yangling 712100, China
* Correspondence: agriculturalrobot@nwafu.edu.cn
Received: 1 May 2020; Accepted: 12 June 2020; Published: 14 June 2020
Abstract:
During the process of automated crop picking, the two hand–eye coordination operation
systems, namely “eye to hand” and “eye in hand” have their respective advantages and disadvantages.
It is challenging to simultaneously consider both the operational accuracy and the speed of a
manipulator. In response to this problem, this study constructs a “global–local” visual servo picking
system based on a prototype of a picking robot to provide a global field of vision (through binocular
vision) and carry out the picking operation using the monocular visual servo. Using tomato picking
as an example, experiments were conducted to obtain the accuracies of judgment and range of fruit
maturity, and the scenario of fruit-bearing was simulated over an area where the operation was
ongoing to examine the rate of success of the system in terms of continuous fruit picking. The results
show that the global–local visual servo picking system had an average accuracy of correctly judging
fruit maturity of 92.8%, average error of fruit distance measurement in the range 0.485 cm, average
time for continuous fruit picking of 20.06 s, and average success rate of picking of 92.45%.
Keywords: tomato picking; visual servo; manipulator; hand–eye coordination
1. Introduction
Selective fruit harvesting is among the most time-consuming and labor-intensive agricultural
operations. Over the past four decades, humans have been trying to develop robots to do this work [
1
–
4
].
However, owing to the complex operating environment and unstructured physical parameters of the
operational objects, several key factors affect the smooth operation of fruit-harvesting robots. One of
them is the precise collaborative operation of the target positioning unit of fruit recognition and the
picking execution component, also known as the “hand–eye collaborative operation” system [5–7].
In the hand–eye coordination operation system of the manipulator, there are two major ways
to install cameras, “eye to hand” and “eye in hand” [
8
]. In the “eye to hand”-based hand–eye
coordination operation system, the camera and the manipulator are installed separately, which can
help obtain image-related information of the fruit over a larger field of view, where it is easy to
realize visual feedback control, but in this system, the movement of the manipulator causes the target
object to be occluded. When the environment changes or the camera moves, the latter needs to be
recalibrated [
9
,
10
]. The accuracy of the visual system and the mechanical system also affect the rate
of success of operation, operational accuracy, and cost. In the “eye in hand” hand–eye coordination
operation system, the camera is fixed at the end of the manipulator, and the target object is close to it to
prevent the manipulator from occluding the target object and to achieve a high-resolution image [
11
].
Sensors 2020, 20, 3366; doi:10.3390/s20123366 www.mdpi.com/journal/sensors
