Citation: Sheraz, M.; Shafique, S.;
Imran, S.; Asif, M.; Ullah, R.;
Ibrar, M.; Khan, J.; Wuttisittikulkij, L.
A Reinforcement Learning Based
Data Caching in Wireless Networks.
Appl. Sci. 2022, 12, 5692. https://
doi.org/10.3390/app12115692
Academic Editors: Alexandros-
Apostolos Boulogeorgos, Thomas
Lagkas, Panagiotis Sarigiannidis,
Vasileios Argyriou, Pantelis
Angelidis and Christos Bouras
Received: 22 April 2022
Accepted: 31 May 2022
Published: 3 June 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
A Reinforcement Learning Based Data Caching in
Wireless Networks
Muhammad Sheraz
1
, Shahryar Shafique
1,
*, Sohail Imran
1
, Muhammad Asif
2,
*, Rizwan Ullah
3,
*,
Muhammad Ibrar
4
, Jahanzeb Khan
1
and Lunchakorn Wuttisittikulkij
3,
*
1
Department of Electrical Engineering, Iqra National University, Peshawar 25000, Pakistan;
mshpk2@gmail.com (M.S.); sohail.imran@inu.edu.pk (S.I.); jehanzeb.khan@inu.edu.pk (J.K.)
2
Department of Electrical Engineering, Main Campus, University of Science & Technology,
Bannu 28100, Pakistan
3
Wireless Communication Ecosystem Research Unit, Department of Electrical Engineering,
Chulalongkorn University, Bangkok 10330, Thailand
4
Department of Physics, Islamia College Peshawar, Peshawar 25000, Pakistan; ibrar@icp.edu.pk
* Correspondence: shahryar.shafique@inu.edu.pk (S.S.); masifeed@ustb.edu.pk (M.A.);
eageleyes_2009@yahoo.com (R.U.); wlunchak@chula.ac.th (L.W.)
Abstract:
Data caching has emerged as a promising technique to handle growing data traffic and
backhaul congestion of wireless networks. However, there is a concern regarding how and where to
place contents to optimize data access by the users. Data caching can be exploited close to users by
deploying cache entities at Small Base Stations (SBSs). In this approach, SBSs cache contents through
the core network during off-peak traffic hours. Then, SBSs provide cached contents to content-
demanding users during peak traffic hours with low latency. In this paper, we exploit the potential
of data caching at the SBS level to minimize data access delay. We propose an intelligence-based
data caching mechanism inspired by an artificial intelligence approach known as Reinforcement
Learning (RL). Our proposed RL-based data caching mechanism is adaptive to dynamic learning and
tracks network states to capture users’ diverse and varying data demands. Our proposed approach
optimizes data caching at the SBS level by observing users’ data demands and locations to efficiently
utilize the limited cache resources of SBS. Extensive simulations are performed to evaluate the
performance of proposed caching mechanism based on various factors such as caching capacity, data
library size, etc. The obtained results demonstrate that our proposed caching mechanism achieves
4% performance gain in terms of delay vs. contents, 3.5% performance gain in terms of delay vs.
users, 2.6% performance gain in terms of delay vs. cache capacity, 18% performance gain in terms
of percentage traffic offloading vs. popularity skewness (
γ
), and 6% performance gain in terms of
backhaul saving vs. cache capacity.
Keywords: caching; network delay; small base station; 5G; dynamic data popularity; reinforcement
learning; Q-learning
1. Introduction
There is an unprecedented increase in the growth of mobile data traffic contributed
by the advent of inexpensive smart electronic gadgets that are overburdening the scarce
spectrum resources below 6 GHz [
1
]. This tremendous burst of users’ data requests imposes
network overhead to provide data to the users during peak-traffic hours. To handle high
data demands, it is imperative to develop new communication techniques such as internet
of things (IoT), data caching, and cloud computing [2].
A hierarchical heterogeneous network comprised of macro and small cells is envi-
sioned for 4G by 3GPP to ameliorate network density [
3
]. Under the coverage of macro
cells, multiple low-power and small coverage Small Base Stations (SBSs) are deployed and
connected to the core network through a wired backhaul link [
4
]. Conventionally, users
Appl. Sci. 2022, 12, 5692. https://doi.org/10.3390/app12115692 https://www.mdpi.com/journal/applsci
