Citation: Han, S.; Liu, X.; Wang, G.
Visual Sorting Method Based on
Multi-Modal Information Fusion.
Appl. Sci. 2022, 12, 2946.
https://doi.org/10.3390/
app12062946
Academic Editor: Giovanni Boschetti
and João Miguel da Costa Sousa
Received: 12 February 2022
Accepted: 10 March 2022
Published: 14 March 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Visual Sorting Method Based on Multi-Modal
Information Fusion
Song Han , Xiaoping Liu * and Gang Wang
School of Modern Post, Beijing University of Posts and Telecommunications, Beijing 100876, China;
hansong@bupt.edu.cn (S.H.); wg58977@bupt.edu.cn (G.W.)
* Correspondence: liuxp@bupt.edu.cn
Abstract:
Visual sorting of stacked parcels is a key issue in intelligent logistics sorting systems. In order
to improve the sorting success rate of express parcels and effectively obtain the sorting order of express
parcels, a visual sorting method based on multi-modal information fusion (VS-MF) is proposed in
this paper. Firstly, an object detection network based on multi-modal information fusion (OD-MF) is
proposed. The global gradient feature is extracted from depth information as a self-attention module.
More spatial features are learned by the network, and the detection accuracy is improved significantly.
Secondly, a multi-modal segmentation network based on Swin Transformer (MS-ST) is proposed
to detect the optimal sorting positions and poses of parcels. More fine-grained information of the
sorting parcels and the relationships between them are gained by adding Swin Transformer models.
Frequency domain information and depth information are used as supervision signals to obtain the
pickable areas and infer the occlusion degrees of parcels. A strategy for the optimal sorting order is
also proposed to ensure the stability of the system. Finally, a sorting system with a 6-DOF robot is
constructed to complete the sorting task of stacked parcels. The accuracy and stability the system are
verified by sorting experiments.
Keywords: multi-modal; self-attention; Swin Transformer; depth estimation; robot sorting
1. Introduction
With the vigorous development of e-commerce and the rising labor costs, more and
more e-commerce and logistics companies are building automated logistics sorting centers.
The degree of automation is getting higher and higher. However, there are still some defects
in the automated sorting process where human assistance is required. For example, the
disorderly stacking of express parcels requires manual sorting and handling, which greatly
limits the efficiency of express sorting and transportation. For such scenarios, industrial
robots are used to replace manpower in our solution, and a visual sorting method based
on multi-modal information fusion (VS-MF) is proposed. The proposed strategy could
realize the detection of the optimal sorting position, pose and order for disorderly stacked
express parcels.
Robotic visual sorting systems [
1
–
3
] generally use the environmental information
collected by 3D vision systems or RGB-D cameras as the original input source. Then, one or
more potential sorting positions are predicted by visual algorithms. The grasping posture
of the robot is gained according to the object posture or the image features. Finally, the
robot achieves the grasping through trajectory planning. The key tasks in the whole process
are improving the detection accuracy of sorting position and pose for each parcel, and
determining the optimal sorting order of multi-object scene. The completion effect of these
two tasks will influence the accuracy safety of the final sorting and the efficiency of the
sorting system.
Deep learning methods have achieved great success in various vision tasks. Many re-
searchers have introduced deep learning frameworks into the field of robot sorting, forming
methods based on two different types of vision tasks. One is the visual sorting system based
Appl. Sci. 2022, 12, 2946. https://doi.org/10.3390/app12062946 https://www.mdpi.com/journal/applsci
