Citation: Wang, J.; Wang, C.; Liu, Y.;
Yue, T.; Wang, Y.; You, F. Multimodal
Warnings Design for In-Vehicle Robots
under Driving Safety Scenarios.
Sensors 2023, 23, 156. https://
doi.org/10.3390/s23010156
Academic Editors: Enrico Vezzetti,
Gabriele Baronio, Domenico
Speranza, Luca Ulrich and Andrea
Luigi Guerra
Received: 4 December 2022
Revised: 18 December 2022
Accepted: 19 December 2022
Published: 23 December 2022
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Multimodal Warnings Design for In-Vehicle Robots under
Driving Safety Scenarios
Jianmin Wang
1,2,3
, Chengji Wang
1
, Yujia Liu
1
, Tianyang Yue
1
, Yuxi Wang
1
and Fang You
1,2,
*
1
Car Interaction Design Lab, College of Arts and Media, Tongji University, Shanghai 201804, China
2
Shenzhen Research Institute, Sun Yat-Sen University, Shenzhen 518057, China
3
Nanchang Research Institute, Sun Yat-Sen University, Nanchang 330224, China
* Correspondence: youfang@tongji.edu.cn; Tel.: +86-21-6958-4745
Abstract:
In case of dangerous driving, the in-vehicle robot can provide multimodal warnings to
help the driver correct the wrong operation, so the impact of the warning signal itself on driving
safety needs to be reduced. This study investigates the design of multimodal warnings for in-vehicle
robots under driving safety warning scenarios. Based on transparency theory, this study addressed
the content and timing of visual and auditory modality warning outputs and discussed the effects
of different robot speech and facial expressions on driving safety. Two rounds of experiments were
conducted on a driving simulator to collect vehicle data, subjective data, and behavioral data. The
results showed that driving safety and workload were optimal when the robot was designed to use
negative expressions for the visual modality during the comprehension (SAT 2) phase and speech
at a rate of 345 words/minute for the auditory modality during the comprehension (SAT 2) and
prediction (SAT 3) phases. The design guideline obtained from the study provides a reference for the
interaction design of driver assistance systems with robots as the interface.
Keywords: multimodal warnings; interaction design; transparency; in-vehicle robots
1. Introduction
Driving safety is critical for vehicle drivers and other road users (e.g., pedestrians,
cyclists, bicyclists, motorcycles, etc.). Human factors play a significant role in automotive
safety. According to a survey conducted by the National Highway Traffic Safety Admin-
istration (NHTSA), human-caused incidents account for 94% of all vehicle crashes [
1
].
Today, cars are equipped with a variety of driver assistance systems to help drivers drive
more safely. These driver assistance systems have become one of the most active areas
of Intelligent Traffic System (ITS) research [
2
,
3
]. In recent years, with the development
of automation technology, natural language processing, and emotional computing, many
intelligent assistive systems have been equipped with anthropomorphic robotic bodies.
Driver assistance systems have evolved from human–machine interaction to human–robot
interaction (Figure 1), and automakers have begun using robots as the interface of driver
assistance systems. The research of Williams et al. showed that dynamic robots (Figure 2)
had a significant impact on reducing the user’s cognitive load and distractions [
4
]. These
robots are generally anthropomorphic and, thus, more like human passengers, which
enhances the driver’s concern for safe driving [
5
–
7
]. Since robots generally have displays
and speakers, vehicle-mounted robots can provide multimodal warnings with combined
visual modality and auditory modality by voice and expressions. Outputting facial ex-
pressions and speech is a fundamental capability of in-vehicle robots. Several studies
have shown that robot facial expressions and speech have additional positive effects on
driving safety [
8
,
9
]. Many studies have shown that multimodal warnings in cars are more
beneficial for driving safety than unimodal warnings [
10
,
11
], both for manual [
11
,
12
] and
highly automated [13] vehicles.
Sensors 2023, 23, 156. https://doi.org/10.3390/s23010156 https://www.mdpi.com/journal/sensors
