Citation: Yuan, Y.; Yang, W.; Luo, Z.;
Gou, R. Temporal Context Modeling
Network with Local-Global
Complementary Architecture for
Temporal Proposal Generation.
Electronics 2022, 11, 2674. https://
doi.org/10.3390/electronics11172674
Academic Editor: Silvia Liberata
Ullo
Received: 20 July 2022
Accepted: 23 August 2022
Published: 26 August 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Temporal Context Modeling Network with Local-Global
Complementary Architecture for Temporal Proposal Generation
Yunfeng Yuan
1,2
, Wenzhu Yang
1,2,
*, Zifei Luo
1,2
and Ruru Gou
1,2
1
School of Cyber Security and Computer, Hebei University, Baoding 071002, China
2
Hebei Machine Vision Engineering Research Center, Hebei University, Baoding 071002, China
* Correspondence: wenzhuyang@hbu.edu.cn; Tel.: +86-15720127565
Abstract:
Temporal Action Proposal Generation (TAPG) is a promising but challenging task with
a wide range of practical applications. Although state-of-the-art methods have made significant
progress in TAPG, most ignore the impact of the temporal scales of action and lack the exploitation
of effective boundary contexts. In this paper, we propose a simple but effective unified framework
named Temporal Context Modeling Network (TCMNet) that generates temporal action proposals.
TCMNet innovatively uses convolutional filters with different dilation rates to address the temporal
scale issue. Specifically, TCMNet contains a BaseNet with dilated convolutions (DBNet), an Action
Completeness Module (ACM), and a Temporal Boundary Generator (TBG). The DBNet aims to
model temporal information. It handles input video features through different dilated convolutional
layers and outputs a feature sequence as the input of ACM and TBG. The ACM aims to evaluate
the confidence scores of densely distributed proposals. The TBG is designed to enrich the boundary
context of an action instance. The TBG can generate action boundaries with high precision and high
recall through a local–global complementary structure. We conduct comprehensive evaluations on
two challenging video benchmarks: ActivityNet-1.3 and THUMOS14. Extensive experiments demon-
strate the effectiveness of the proposed TCMNet on tasks of temporal action proposal generation and
temporal action detection.
Keywords:
temporal action proposal generation; temporal action detection; boundary context; action
completeness module; temporal boundary generator
1. Introduction
Temporal action detection is one of the most fundamental tasks in video understanding,
which is aimed at not only classifying every action instance in each video, but also looking
for their accurate temporal locations. In general, the temporal action detection task is
composed of two subtasks: temporal action proposal generation and action classification.
Although current action recognition methods [
1
,
2
] can achieve convincing classification
accuracy, the performance of temporal action detection is still unsatisfactory on mainstream
benchmarks. Object detection aims to find as many tight bounding box locations and
classes of objects as possible. With the continuous in-depth research of many works, a
quite number of recent methods [
3
–
5
] have achieved remarkable progress and superior
performance. Akin to object proposals for object detection in images, temporal action
proposal has a crucial impact on the quality of action detection. As a result, more and
more works are therefore devoted to improving the quality of temporal action proposals.
Temporal Action Proposal Generation (TAPG) gradually became a research focus in video
understanding tasks. TAPG is not only used for temporal action detection, but is also
the core of several downstream tasks such as video recommendation, video captioning,
and summarization.
Proposals generated by a robust TAPG method usually have two essential properties:
(1) The generated temporal proposals should cover ground-truth action instances accurately
Electronics 2022, 11, 2674. https://doi.org/10.3390/electronics11172674 https://www.mdpi.com/journal/electronics
