Article
Traffic Signal Optimization for Multiple Intersections Based on
Reinforcement Learning
Jaun Gu
1
, Minhyuck Lee
1
, Chulmin Jun
1,
* , Yohee Han
2
, Youngchan Kim
2
and Junwon Kim
2
Citation: Gu, J.; Lee, M.; Jun, C.; Han,
Y.; Kim, Y.; Kim, J. Traffic Signal
Optimization for Multiple
Intersections Based on Reinforcement
Learning. Appl. Sci. 2021, 11, 10688.
https://doi.org/10.3390/app112210688
Academic Editors: Nikos D. Lagaros,
Vagelis Plevris and Paola Pellegrini
Received: 20 August 2021
Accepted: 10 November 2021
Published: 12 November 2021
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
1
Department of Geoinformatics, University of Seoul, Seoul 02504, Korea; umseakind2@uos.ac.kr (J.G.);
lmhll123@uos.ac.kr (M.L.)
2
Department of Transportation Engineering, University of Seoul, Seoul 02504, Korea;
yeohee@gmail.com (Y.H.); yckimm@uos.ac.kr (Y.K.); mirageno21@uos.ac.kr (J.K.)
* Correspondence: cmjun@uos.ac.kr
Abstract:
In order to deal with dynamic traffic flow, adaptive traffic signal controls using reinforce-
ment learning are being studied. However, most of the related studies are difficult to apply to the
real field considering only mathematical optimization. In this study, we propose a reinforcement
learning-based signal optimization model with constraints. The proposed model maintains the
sequence of typical signal phases and considers the minimum green time. The model was trained
using Simulation of Urban MObility (SUMO), a microscopic traffic simulator. The model was eval-
uated in the virtual environment similar to a real road with multiple intersections connected. The
performance of the proposed model was analyzed by comparing the delay and number of stops with
a reinforcement learning model that did not consider constraints and a fixed-time model. In a peak
hour, the proposed model reduced the delay from 3 min 15 s to 2 min 15 s and the number of stops
from 11 to 4.7 compared to the fixed-time model.
Keywords:
traffic signal optimization; reinforcement learning; adaptive traffic signal control; multiple
intersections; Deep Q-network
1. Introduction
Traffic signal control plays an essential role in city management because traffic conges-
tion brings economic, environmental, and social disadvantages. Traffic signal control aims
to minimize congestion by determining the optimal values of parameters such as the cycle
length and phases duration [
1
,
2
]. In many areas, the traffic signal control systems based on
a fixed-time model are still in use [
3
–
5
]. While these systems are easy to implement, they
cannot respond flexibly to dynamic traffic flows [6,7].
To quickly respond to variety in the traffic environment, signal control systems should
be able to choose their own actions without waiting for instructions from a central com-
puter [
8
]. Therefore, reinforcement learning models are being studied that allow the traffic
signal controller to receive realtime data around the intersection, such as traffic volume
and vehicle speed, and change signal appropriately for the given traffic situation [
9
,
10
].
If the above sentence is expressed in reinforcement learning terms, the controller is the
agent, the data input to the controller is the state, the controller’s decision is the action,
and the benefit provided to the agent according to the action is called reward. The goal of
reinforcement learning is to maximize the future reward that an agent can obtain [11].
2. Literature Review
Reinforcement learning is the most recently used algorithm in the field of signal
control research. However, most studies have not considered the constraints applied to a
real-world intersection or tested in a local area such as a single intersection.
Touhbi et al.
(2017) analyzed the possibility of using the Q-Learning algorithm for adaptive traffic
signal control [
12
]. The Q-Learning algorithm was helpful in resolving traffic congestion
Appl. Sci. 2021, 11, 10688. https://doi.org/10.3390/app112210688 https://www.mdpi.com/journal/applsci
