Citation: Chen, R.; Liu, H.; Liu, C.;
Yu, G.; Yang, X.; Zhou, Y. System
Frequency Control Method Driven by
Deep Reinforcement Learning and
Customer Satisfaction for
Thermostatically Controlled Load.
Energies 2022, 15, 7866. https://
doi.org/10.3390/en15217866
Academic Editors: Luis
Hernández-Callejo,
Sergio Nesmachnow and
Sara Gallardo Saavedra
Received: 18 September 2022
Accepted: 18 October 2022
Published: 24 October 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
System Frequency Control Method Driven by Deep
Reinforcement Learning and Customer Satisfaction for
Thermostatically Controlled Load
Rusi Chen
1
, Haiguang Liu
1
, Chengquan Liu
2,
*, Guangzheng Yu
2
, Xuan Yang
3
and Yue Zhou
3
1
State Grid Hubei Electric Power Research Institute, Wuhan 430077, China
2
Department of Electrical Engineering, Shanghai University of Electric Power, Shanghai 201306, China
3
State Grid Hubei Electric Power Co., Ltd., Wuhan 430072, China
* Correspondence: y20103028@mail.shiep.edu.cn
Abstract:
The intermittence and fluctuation of renewable energy aggravate the power fluctuation of
the power grid and pose a severe challenge to the frequency stability of the power system. Thermo-
statically controlled loads can participate in the frequency regulation of the power grid due to their
flexibility. Aiming to solve the problem of the traditional control methods, which have limited adjust-
ment ability, and to have a positive influence on customers, a deep reinforcement learning control
strategy based on the framework of soft actor–critic is proposed, considering customer satisfaction.
Firstly, the energy storage index and the discomfort index of different users are defined. Secondly,
the fuzzy comprehensive evaluation method is applied to evaluate customer satisfaction. Then, the
multi-agent models of thermostatically controlled loads are established based on the soft actor–critic
algorithm. The models are trained by using the local information of thermostatically controlled loads,
and the comprehensive evaluation index fed back by users and the frequency deviation. After train-
ing, each agent can realize the cooperative response of thermostatically controlled loads to the system
frequency only by relying on the local information. The simulation results show that the proposed
strategy can not only reduce the frequency fluctuation, but also improve customer satisfaction.
Keywords:
thermostatically controlled load; frequency regulation; customer satisfaction; soft
actor–critic
;
energy storage index; discomfort index
1. Introduction
With the increasing proportion of renewable energy in the power grid, the charac-
teristics of intermittence and fluctuation will bring considerable challenges to the active
power balance and frequency stability of the power grid [
1
]. The traditional power system
maintains the balance of the system by adjusting the output of the generating side units.
The regulation method is relatively simple and will generate additional economic and
environmental costs [
2
]. In addition, with the increase in power load and the extensive
access to renewable energy, the regulation capacity of the power generation side gradually
decreases [
3
]. The power system with renewable energy as the main body can utilize
advanced information technology to integrate and dispatch demand-side resources to
provide a variety of auxiliary services [
4
,
5
]. Therefore, reasonable control of demand-side
resources can supplement the traditional system frequency regulation, and thus enhance
the stability of the power system [6].
In the demand-side resources, the thermostatically controlled load (TCL) is a kind of
electric equipment controlled by a thermostat, which can realize electric heating conver-
sion and adjustable temperature, including in heat pumps, electric storage water heaters
(ESWHs), refrigerators, and heating, ventilation and air conditioning (HVAC) systems [7].
TCL can be used to provide frequency regulation services, and is mainly based on the
Energies 2022, 15, 7866. https://doi.org/10.3390/en15217866 https://www.mdpi.com/journal/energies
