Citation: Lee, J.; Park, S.; Kim, T.;
Kim, H. Time-Invariant
Features-Based Online Learning for
Long-Term Notification Management:
A Longitudinal Study. Appl. Sci. 2022,
12, 5432. https://doi.org/10.3390/
app12115432
Academic Editors: Enrico Vezzetti,
Andrea Luigi Guerra, Gabriele
Baronio, Domenico Speranza
and Luca Ulrich
Received: 23 April 2022
Accepted: 26 May 2022
Published: 27 May 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Time-Invariant Features-Based Online Learning for Long-Term
Notification Management: A Longitudinal Study
Jemin Lee
1
, Sihyeong Park
2
, Taeho Kim
1
and Hyungshin Kim
3,
*
1
AI SoC Research Division, Artificial Intelligence Research Laboratory, Electronics and Telecommunications
Research Institute (ETRI), Daejeon 34129, Korea; leejaymin@etri.re.kr (J.L.); taehokim@etri.re.kr (T.K.)
2
SoC Platform Research Center, Korea Electronics Technology Institute (KETI), Seongnam-si 13509, Korea;
sihyeong@keti.re.kr
3
The Division of Computer Convergence, Chungnam National University, Daejeon 34134, Korea
* Correspondence: hyungshin@cnu.ac.kr
Abstract:
The increasing number of daily notifications generated by smartphones and wearable
devices increases mental burdens, deteriorates productivity, and results in energy waste. These
phenomena are exacerbated by emerging use cases in which users are wearing and using an increas-
ing number of personal mobile devices, such as smartphones, smartwatches, AirPods, or tablets
because all the devices can generate redundant notifications simultaneously. Therefore, in addition
to distraction, redundant notifications triggered by multiple devices result in energy waste. Prior
work proposed a notification management system called PASS, which automatically manipulates the
occurrence of notifications based on personalized models. However, machine-learning-based models
work poorly against new incoming notifications because prior work has not investigated behavior
changes over time. To reduce the gap between modeling and real deployment when the model is to be
used long-term, we conducted a longitudinal study with data collection over long-term periods. We
collected an additional 11,258 notifications and analyzed 18,407 notifications, including the original
dataset. The total study spans two years. Through a statistical test, we identified time-invariant
features that can be fully used for training. To overcome the accuracy drop caused by newly occurring
data, we design windowing time-invariant online learning (WTOL). In the newly collected dataset,
WTOL improves the F-score of the original models based on batch learning from 44.3% to 69.0% by
combining online learning and windowing features depending on time sensitivity.
Keywords: notification; machine learning; smartphone; smartwatch
1. Introduction
Notifications are an essential function to inform users of urgent or useful information.
In spite of their usefulness, frequently sending notifications to users at inopportune mo-
ments could harm their engagement with the task at hand and productivity. Prior studies
have investigated the negative effects of notifications [
1
–
3
] and proposed a variety of in-
telligent notification systems that manage notification delivery at an opportune moment
when users can conveniently deal with new incoming messages [
4
–
9
]. To help a system
recognize when users can tolerate an incoming notification, interruptibility models were
proposed and trained with notification contents [
4
,
5
], breakpoints [
6
], or important usage
information [
7
]. Furthermore, Mehrotra et al. [
8
] and Pradhan et al. [
9
] proposed a smart
notification manager that directly manipulates notification occurrences based on machine-
learning models. In addition to studies on notifications by smartphones, recent studies
have analyzed and managed notifications triggered by smartwatches [
10
–
12
]. Meanwhile,
interruptibility research has been extended to multi-device environments [
12
–
16
]. In multi-
device environments, in addition to user distraction, energy waste caused by redundant
notifications increases. To suppress redundant notifications between smartphones and
smartwatches, PASS [
17
] proposed a matching-learning-based notification manager that
Appl. Sci. 2022, 12, 5432. https://doi.org/10.3390/app12115432 https://www.mdpi.com/journal/applsci
