Article
Transmission Network Expansion Planning Considering Wind
Power and Load Uncertainties Based on Multi-Agent DDQN
Yuhong Wang
1
, Xu Zhou
1
, Yunxiang Shi
1
, Zongsheng Zheng
1,
*, Qi Zeng
1
, Lei Chen
1
, Bo Xiang
2
and Rui Huang
2
Citation: Wang, Y.; Zhou, X.; Shi, Y.;
Zheng, Z.; Zeng, Q.; Chen, L.; Xiang,
B.; Huang, R. Transmission Network
Expansion Planning Considering
Wind Power and Load Uncertainties
Based on Multi-Agent DDQN.
Energies 2021, 14, 6073. https://
doi.org/10.3390/en14196073
Academic Editors: Pierluigi Siano
and Hassan Haes Alhelou
Received: 10 August 2021
Accepted: 13 September 2021
Published: 24 September 2021
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
1
College of Electrical Engineering, Sichuan University, Chengdu 610065, China;
yuhongwang@scu.edu.cn (Y.W.); zhouxu@stu.ecu.edu.cn (X.Z.); shiyunxiang@stu.scu.edu.cn (Y.S.);
zengqi@scu.edu.cn (Q.Z.); chen_lei@stu.scu.edu.cn (L.C.)
2
State Grid Sichuan Comprehensive Energy Service Co., Ltd., Chengdu 610031, China;
771771973@163.com (B.X.); 12687309@163.com (R.H.)
* Correspondence: zongshengzheng@scu.edu.cn; Tel.: +86-1528-106-498
Abstract:
This paper presents a multi-agent Double Deep Q Network (DDQN) based on deep
reinforcement learning for solving the transmission network expansion planning (TNEP) of a high-
penetration renewable energy source (RES) system considering uncertainty. First, a K-means algo-
rithm that enhances the extraction quality of variable wind and load power uncertain characteristics
is proposed. Its clustering objective function considers the cumulation and change rate of oper-
ation data. Then, based on the typical scenarios, we build a bi-level TNEP model that includes
comprehensive cost, electrical betweenness, wind curtailment and load shedding to evaluate the
stability and economy of the network. Finally, we propose a multi-agent DDQN that predicts the
construction value of each line through interaction with the TNEP model, and then optimizes the
line construction sequence. This training mechanism is more traceable and interpretable than the
heuristic-based methods. Simultaneously, the experience reuse characteristic of multi-agent DDQN
can be implemented in multi-scenario TNEP tasks without repeated training. Simulation results
obtained in the modified IEEE 24-bus system and New England 39-bus system verify the effectiveness
of the proposed method.
Keywords:
transmission network expansion planning (TNEP); deep reinforcement learning; uncer-
tainty; wind power; multi-agent DDQN
1. Introduction
Although countries have actively implemented Nationally Determined Contributions
(NDCs) to alleviate climate deterioration in recent years, global greenhouse gas emissions
are still in the process of continuous growth, and there has not yet been a peak phenomenon.
In order to control the future temperature rise within 1.5
◦
C, the United Nations Environ-
ment Programme advocates that countries around the world should reduce the emissions
to fill the gap between the current greenhouse gas emissions level and the Paris Agreement
provisions [
1
]. The transformation of energy structure is regarded as the primary way
for emissions reduction by all countries. Many countries have formulated plans to build
a high-penetration renewable energy source (RES) system, which fully releases the high
environmental and economic value of renewable energy by replacing fossil energy [
2
,
3
].
There are two main challenges in the RES system construction. One is to solve the time
and space uncertainties caused by the intermittency of renewable energy [
4
], and the other
is to optimize the network structure for large-scale renewable energy integration [
5
]. The
transmission network expansion planning (TNEP) is the crucial task of power system
construction, which determines the basic structure and system characteristic. Therefore,
the characteristics of system with high-penetration of RES should be fully considered in
the TNEP task on the basis of ensuring system stability and economy.
Energies 2021, 14, 6073. https://doi.org/10.3390/en14196073 https://www.mdpi.com/journal/energies
