Citation: Yu, J.; Sun, H.; Sun, J.
Improved Twin Delayed Deep
Deterministic Policy Gradient
Algorithm Based Real-Time
Trajectory Planning for Parafoil
under Complicated Constraints. Appl.
Sci. 2022, 12, 8189. https://
doi.org/10.3390/app12168189
Academic Editor: Giancarlo Mauri
Received: 3 July 2022
Accepted: 14 August 2022
Published: 16 August 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Improved Twin Delayed Deep Deterministic Policy Gradient
Algorithm Based Real-Time Trajectory Planning for Parafoil
under Complicated Constraints
Jiaming Yu
1
, Hao Sun
2,
* and Junqing Sun
1
1
School of Computer Science and Engineering, Tianjin University of Technology, Tianjin 300384, China
2
College of Artificial Intelligence, Nankai University, Tianjin 300350, China
* Correspondence: sunh@nankai.edu.cn; Tel.: +86-18502256348
Abstract:
A parafoil delivery system has usually been used in the fields of military and civilian
airdrop supply and aircraft recovery in recent years. However, since the altitude of the unpowered
parafoil is monotonically decreasing, it is limited by the initial flight altitude. Thus, combining
the multiple constraints, such as the ground obstacle avoidance and flight time, it puts forward a
more stringent standard for the real-time performance of trajectory planning of the parafoil delivery
system. Thus, to enhance the real-time performance, we propose a new parafoil trajectory planning
method based on an improved twin delayed deep deterministic policy gradient. In this method, by
pre-evaluating the value of the action, a scale of noise will be dynamically selected for improving
the globality and randomness, especially for the actions with a low value. Furthermore, not like the
traditional numerical computation algorithm, by building the planning model in advance, the deep
reinforcement learning method does not recalculate the optimal flight trajectory of the system when
the parafoil delivery system is launched at different initial positions. In this condition, the trajectory
planning method of deep reinforcement learning has greatly improved in real-time performance.
Finally, several groups of simulation data show that the trajectory planning theory in this paper is
feasible and correct. Compared with the traditional twin delayed deep deterministic policy gradient
and deep deterministic policy gradient, the landing accuracy and success rate of the proposed method
are improved greatly.
Keywords:
parafoil delivery system; trajectory planning; homing control; twin delayed deep
deterministic policy gradient
1. Introduction
A parafoil delivery system is a special precision air conveying system [
1
–
4
]. By
controlling the shape of the parafoil canopy, this system can change the flight direction
and achieve precise landing, which is hard to realize by the traditional parachute system.
Based on this advantage, parafoil has broad prospects in the fields of military and civilian
airdrop supply and aircraft recovery. For example, NASA applied a 689 m
2
parafoil to
recovery X-38 aircraft [
5
,
6
]. The German Aerospace Center (DLR) has also developed a
small and widely instrumented aircraft Alex (a parafoil delivery system), and proposed
the application of the GNC algorithm in autonomous landing [
7
,
8
]. Other research, such
as [
9
,
10
], explored the modeling approach for ram-air parachutes. Ref. [
11
] proposed pods
that provide distributed sensors in the whole parachute canopy and a fusion algorithm to
merge the pod data into useful canopy state estimation, which improves the accuracy of
canopy state estimation. Ref. [
12
] presented a feasibility study on the engine and engine
frame recovery system of an existing expendable heavy launch vehicle.
By analyzing the dynamic features of parafoil, it can be observed that its vertical
velocity is nearly uncontrollable for the unpowered parafoil. Due to this feature, the flight
time of parafoil trajectory planning depends mainly on the initial launching altitude. The
Appl. Sci. 2022, 12, 8189. https://doi.org/10.3390/app12168189 https://www.mdpi.com/journal/applsci
