Article
Data-Driven Reinforcement-Learning-Based Automatic
Bucket-Filling for Wheel Loaders
Jianfei Huang , Dewen Kong, Guangzong Gao , Xinchun Cheng and Jinshi Chen *
Citation: Huang, J.; Kong, D.; Gao,
G.; Cheng, X.; Chen, J. Data-Driven
Reinforcement-Learning-Based
Automatic Bucket-Filling for Wheel
Loaders. Appl. Sci. 2021, 11, 9191.
https://doi.org/10.3390/app11199191
Academic Editors: Nikos D. Lagaros
and Vagelis Plevris
Received: 12 July 2021
Accepted: 24 September 2021
Published: 2 October 2021
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Key Laboratory of CNC Equipment Reliability, Ministry of Education, School of Mechanical and Aerospace
Engineering, Jilin University, Changchun 130022, China; jfhuang19@mails.jlu.edu.cn (J.H.);
dwkong@jlu.edu.cn (D.K.); gaoguangzong_jlu@163.com (G.G.); chengxc19@mails.jlu.edu.cn (X.C.)
* Correspondence: chenjinshi8304@163.com
Abstract:
Automation of bucket-filling is of crucial significance to the fully automated systems for
wheel loaders. Most previous works are based on a physical model, which cannot adapt to the
changeable and complicated working environment. Thus, in this paper, a data-driven reinforcement-
learning (RL)-based approach is proposed to achieve automatic bucket-filling. An automatic bucket-
filling algorithm based on Q-learning is developed to enhance the adaptability of the autonomous
scooping system. A nonlinear, non-parametric statistical model is also built to approximate the real
working environment using the actual data obtained from tests. The statistical model is used for
predicting the state of wheel loaders in the bucket-filling process. Then, the proposed algorithm
is trained on the prediction model. Finally, the results of the training confirm that the proposed
algorithm has good performance in adaptability, convergence, and fuel consumption in the absence
of a physical model. The results also demonstrate the transfer learning capability of the proposed
approach. The proposed method can be applied to different machine-pile environments.
Keywords: data-driven model; reinforcement learning; wheel loaders; automatic bucket-filling
1. Introduction
Construction machinery has a pivotal role in the building and mining industry, which
makes a great contribution to the world economy [
1
]. The wheel loader is one of the most
common mobile construction machinery and is often used to transport different materials
at production sites [2].
The automation of wheel loaders, which has received great attention over the past
three decades, can improve safety and reduce costs. Dadhich et al. [
3
] proposed five
steps to full automation of wheel loaders: manual operation, in-sight tele-operation, tele-
remote operation, assisted tele-remote operation, and fully autonomous operation. Despite
extensive research in this field, fully automated systems for wheel loaders have never been
demonstrated. Remote operation is considered a step towards fully automated equipment,
but it has led to a reduction in productivity and fuel efficiency [4].
In the working process of wheel loaders, bucket-filling is a crucial part, as it determines
the weight of the loaded materials. Bucket-filling is a relatively repetitive task for the
operators of wheel-loaders and is suitable for automation. Automatic bucket-filling is
also required for efficient remote operation and the development of fully autonomous
solutions [5]
. The interaction condition between the bucket and the pile strongly affects the
bucket-filling. However, due to the complexity of the working environment, the interaction
condition is unknown and constantly changing. The difference in working materials also
influences the bucket-filling. A general automatic bucket-filling solution is still a challenge
for different piles.
In this paper, a data-driven RL-based approach is proposed for automatic bucket-filling
of wheel loaders to achieve low costs and adapt to changing conditions. The Q-learning
algorithm can learn from different conditions and is used to learn the optimal action in
Appl. Sci. 2021, 11, 9191. https://doi.org/10.3390/app11199191 https://www.mdpi.com/journal/applsci
