Citation: Thomos, N.; Maugey, T.;
Toni, L. Machine Learning for
Multimedia Communications.
Sensors 2022, 22, 819. https://
doi.org/10.3390/s22030819
Academic Editor: Lei Shu
Received: 17 December 2021
Accepted: 14 January 2022
Published: 21 January 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Review
Machine Learning for Multimedia Communications
Nikolaos Thomos
1,
*
,†
, Thomas Maugey
2,†
and Laura Toni
3,†
1
School of Computer Science and Electronic Engineering, University of Essex, Colchester CO4 3SQ, UK
2
Inria, 35042 Rennes, France; thomas.maugey@inria.fr
3
Department of Electrical & Electrical Engineering, University College London (UCL), London WC1E 6AE, UK;
l.toni@ucl.ac.uk
* Correspondence: nthomos@essex.ac.uk
† These authors contributed equally to this work.
Abstract:
Machine learning is revolutionizing the way multimedia information is processed and
transmitted to users. After intensive and powerful training, some impressive efficiency/accuracy
improvements have been made all over the transmission pipeline. For example, the high model
capacity of the learning-based architectures enables us to accurately model the image and video
behavior such that tremendous compression gains can be achieved. Similarly, error concealment,
streaming strategy or even user perception modeling have widely benefited from the recent learning-
oriented developments. However, learning-based algorithms often imply drastic changes to the way
data are represented or consumed, meaning that the overall pipeline can be affected even though
a subpart of it is optimized. In this paper, we review the recent major advances that have been
proposed all across the transmission chain, and we discuss their potential impact and the research
challenges that they raise.
Keywords:
multimedia communications; machine learning; video coding; image coding; error
concealment; video streaming; QoE assessment; content consumption; channel coding; caching
1. Introduction
During the past few years, we have witnessed an unprecedented change in the way
multimedia data are generated and consumed as well as the wide adaptation of im-
age/video in an increasing number of driving applications. For example, Augmented
Reality/Virtual Reality/eXtended Reality (AR/VR/XR) is now widely used in education,
entertainment, military training, and so forth, although this was considered a utopia only a
few years ago. AR/VR/XR systems have transformed the way we interact with the data
and will soon become the main means of communication. Image and video data in various
formats is an essential component of numerous future use cases. An important example is
intelligent transportation systems (ITS), where visual sensors are installed in vehicles to
improve safety through autonomous driving. Another example is visual communication
systems that are commonly deployed in smart cities mainly for surveillance, improving the
quality of life, and environmental monitoring. The above use cases face unique challenges
as they involve not only the communication of huge amounts of data, for example, an intel-
ligent vehicle may require the communication of 750 MB of data per second [
1
], with the
vast majority of them being visual data, but they also have ultra-low latency requirements.
Further, most of these visual data are not expected to be watched, but will be processed by
a machine, necessitating the consideration of goal-oriented coding and communication.
These undergoing transformative changes have been the driving force of research in
both multimedia coding and multimedia communication. This has led to new video coding
standards for encoding visual data for humans (HEVC, VVC) or machines (MPEG activity
on Video Coding for Machines), novel multimedia formats like point clouds, and support
of higher resolutions by the latest displays (digital theatre). Various quality of experience
Sensors 2022, 22, 819. https://doi.org/10.3390/s22030819 https://www.mdpi.com/journal/sensors
