Citation: Zhai, P.; Zhang, Y.; Shaobo,
W. Intelligent Ship Collision
Avoidance Algorithm Based on
DDQN with Prioritized Experience
Replay under COLREGs. J. Mar. Sci.
Eng. 2022, 10, 585. https://doi.org/
10.3390/jmse10050585
Academic Editors: Jacopo Aguzzi
and Daniel Mihai Toma
Received: 4 April 2022
Accepted: 22 April 2022
Published: 26 April 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Journal of
Marine Science
and Engineering
Article
Intelligent Ship Collision Avoidance Algorithm Based on
DDQN with Prioritized Experience Replay under COLREGs
Pengyu Zhai, Yingjun Zhang * and Wang Shaobo
Navigation College, Dalian Maritime University, Dalian 116026, China; zhai_pengyu@dlmu.edu.cn (P.Z.);
wangshaobo1@163.com (W.S.)
* Correspondence: zhangyj@dlmu.edu.cn
Abstract:
Ship collisions often result in huge losses of life, cargo and ships, as well as serious pollution
of the water environment. Meanwhile, it is estimated that between 75% and 86% of maritime accidents
are related to human factors. Thus, it is necessary to enhance the intelligence of ships to partially or
fully replace the traditional piloting mode and eventually achieve autonomous collision avoidance to
reduce the influence of human factors. In this paper, we propose a multi-ship automatic collision
avoidance method based on a double deep Q network (DDQN) with prioritized experience replay.
Firstly, we vectorize the predicted hazardous areas as the observation states of the agent so that similar
ship encounter scenarios can be clustered and the input dimension of the neural network can be fixed.
The reward function is designed based on the International Regulations for Preventing Collision at Sea
(COLREGs) and human experience. Different from the architecture of previous collision avoidance
methods based on deep reinforcement learning (DRL), in this paper, the interaction between the
agent and the environment occurs only in the collision avoidance decision-making phase, which
greatly reduces the number of state transitions in the Markov decision process (MDP). The prioritized
experience replay method is also used to make the model converge more quickly. Finally, 19 single-
vessel collision avoidance scenarios were constructed based on the encounter situations classified by
the COLREGs, which were arranged and combined as the training set for the agent. The effectiveness
of the proposed method in close-quarters situation was verified using the Imazu problem. The
simulation results show that the method can achieve multi-ship collision avoidance in crowded
waters, and the decisions generated by this method conform to the COLREGs and are close to the
level of human ship handling.
Keywords:
collision avoidance; reinforcement learning; DDQN with prioritized experience replay;
COLREGs; intelligent ship
1. Introduction
As global economic activities become more interconnected, the density of maritime
traffic flows is increasing, especially in inshore navigation and in fishing areas. Clearly,
this situation increases the risk of collisions between vessels. Ship collisions often result in
significant casualties and economic damage. At present, the officer on watch (OOW) can
obtain navigational data on surrounding vessels through navigation aids, such as radar
and automatic identification systems (AIS). However, contrary to expectations, misinter-
pretation or omission of information by the pilot can sometimes lead to incorrect decisions
or untimely action. Surveys show that approximately 94.7% of ship-to-ship collisions in the
last 43 years have been caused by human error on the part of the crew, and at least 56% of
collisions are caused by violations of the International Regulations for Collision Avoidance
at Sea (COLREGs) established by the International Maritime Organization (IMO) [
1
,
2
].
Therefore, the development of autonomous collision avoidance systems that comply with
navigation rules is one of the most effective ways to reduce the human factor and the
incidence of collisions by improving the intelligence of ships.
J. Mar. Sci. Eng. 2022, 10, 585. https://doi.org/10.3390/jmse10050585 https://www.mdpi.com/journal/jmse
