Citation: Quiroz, M.; Patiño, R.;
Diaz-Amado, J.; Cardinale, Y. Group
Emotion Detection Based on Social
Robot Perception. Sensors 2022, 22,
3749. https://doi.org/10.3390/
s22103749
Academic Editors: Enrico Vezzetti,
Andrea Luigi Guerra, Gabriele
Baronio, Domenico Speranza and
Luca Ulrich
Received: 1 April 2022
Accepted: 5 May 2022
Published: 14 May 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Group Emotion Detection Based on Social Robot Perception
Marco Quiroz
1,†
, Raquel Patiño
1
, José Diaz-Amado
1,2
and Yudith Cardinale
1,3,
*
,†
1
Electrical and Electronics Engineering Department, School of Electronics and Telecommunications
Engineering, Universidad Católica San Pablo, Arequipa 04001, Peru; marco.quiroz@ucsp.edu.pe (M.Q.);
rpatino@ucsp.edu.pe (R.P.); jose_diaz@ifba.edu.br (J.D.-A.)
2
Instituto Federal da Bahia, Vitoria da Conquista 45078-300, Brazil
3
Higher School of Engineering, Science and Technology, Universidad Internacional de Valencia,
46002 Valencia, Spain
* Correspondence: yudith.cardinale@campusviu.es; Tel.: +58-412-021-5500
† These authors contributed equally to this work.
Abstract:
Social robotics is an emerging area that is becoming present in social spaces, by introducing
autonomous social robots. Social robots offer services, perform tasks, and interact with people in
such social environments, demanding more efficient and complex Human–Robot Interaction (HRI)
designs. A strategy to improve HRI is to provide robots with the capacity of detecting the emotions
of the people around them to plan a trajectory, modify their behaviour, and generate an appropriate
interaction with people based on the analysed information. However, in social environments in
which it is common to find a group of persons, new approaches are needed in order to make robots
able to recognise groups of people and the emotion of the groups, which can be also associated
with a scene in which the group is participating. Some existing studies are focused on detecting
group cohesion and the recognition of group emotions; nevertheless, these works do not focus on
performing the recognition tasks from a robocentric perspective, considering the sensory capacity
of robots. In this context, a system to recognise scenes in terms of groups of people, to then detect
global (prevailing) emotions in a scene, is presented. The approach proposed to visualise and
recognise emotions in typical HRI is based on the face size of people recognised by the robot during
its navigation (face sizes decrease when the robot moves away from a group of people). On each
frame of the video stream of the visual sensor, individual emotions are recognised based on the
Visual Geometry Group (VGG) neural network pre-trained to recognise faces (VGGFace); then, to
detect the emotion of the frame, individual emotions are aggregated with a fusion method, and
consequently, to detect global (prevalent) emotion in the scene (group of people), the emotions of
its constituent frames are also aggregated. Additionally, this work proposes a strategy to create
datasets with images/videos in order to validate the estimation of emotions in scenes and personal
emotions. Both datasets are generated in a simulated environment based on the Robot Operating
System (ROS) from videos captured by robots through their sensory capabilities. Tests are performed
in two simulated environments in ROS/Gazebo: a museum and a cafeteria. Results show that the
accuracy in the detection of individual emotions is 99.79% and the detection of group emotion (scene
emotion) in each frame is 90.84% and 89.78% in the cafeteria and the museum scenarios, respectively.
Keywords:
social robots; emotion detection; group emotion; group detection; facial expression
recognition; group behaviour recognition; human–robot interaction
1. Introduction
Social robots are increasingly being incorporated into crowded human spaces, such
as museums, hospitals, and restaurants, in order to offer services, perform tasks, and
interact with people. Social robots are considered as physical agents with the abilities to
act in complex social environments [
1
]. They must imitate the socio-cognitive abilities of
humans and explore behaviours to be empathic and aid with the interactions between
Sensors 2022, 22, 3749. https://doi.org/10.3390/s22103749 https://www.mdpi.com/journal/sensors
