Citation: Liu, M.; Xie, T.; Cheng, X.;
Deng, J.; Yang, M.; Wang, X.; Liu, M.
FocusedDropout for Convolutional
Neural Network. Appl. Sci. 2022, 12,
7682. https://doi.org/10.3390/
app12157682
Academic Editor: Krzysztof Koszela
Received: 23 June 2022
Accepted: 27 July 2022
Published: 30 July 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
FocusedDropout for Convolutional Neural Network
Minghui Liu
1
, Tianshu Xie
1
, Xuan Cheng
1
, Jiali Deng
1
, Meiyi Yang
2
, Xiaomin Wang
2,
* and Ming Liu
1
1
School of Computer Science and Engineering, University of Electronic Science and Technology of China,
Chengdu 611731, China; minghuiliu@std.uestc.edu.cn (M.L.); tianshuxie@std.uestc.edu.cn (T.X.);
cs_xuancheng@std.uestc.edu.cn (X.C.); dengjiali@std.uestc.edu.cn (J.D.); csmliu@uestc.edu.cn (M.L.)
2
Yangtze Delta Region Institute (Quzhou), University of Electronic Science and Technology of China,
Quzhou 324003, China; meiyiyang@std.uestc.edu.cn
* Correspondence: xmwang@uestc.edu.cn
Featured Application: We propose a non-random dropout method named FocusedDropout, aiming
to make the network focus more on the target. It can effectively improve the performance of feature
learning in deep learning that can be used for any applications with deep learning technology.
Abstract:
In a convolutional neural network (CNN), dropout cannot work well because dropped
information is not entirely obscured in convolutional layers where features are correlated spatially.
Except for randomly discarding regions or channels, many approaches try to overcome this defect
by dropping influential units. In this paper, we propose a non-random dropout method named
FocusedDropout, aiming to make the network focus more on the target. In FocusedDropout, we
use a simple but effective method to search for the target-related features, retain these features and
discard others, which is contrary to the existing methods. We find that this novel method can improve
network performance by making the network more target focused. Additionally, increasing the
weight decay while using FocusedDropout can avoid overfitting and increase accuracy. Experimental
results show that with a slight cost, 10% of batches employing FocusedDropout, can produce a
nice performance boost over the baselines on multiple datasets of classification, including CIFAR10,
CIFAR100 and Tiny ImageNet, and has a good versatility for different CNN models.
Keywords: classification; convolutional neural network; dropout; regularization
1. Introduction
In recent years, deep neural networks have made significant achievements in many
computer vision tasks such as image classification [
1
–
4
], object detection [
5
–
7
], and semantic
segmentation [
8
,
9
]. However, deep layers and millions of neurons also lead to inadequate
training of CNN. Dropout [
10
] is proposed as a regularization method widely used to fight
against overfitting, which stochastically sets the activations of hidden units to zero during
training. For deep CNN, dropout works well in fully connected layers, but its effect is still
not apparent in convolutional layers, where features are correlated spatially. When the
features are strongly correlated between adjacent neurons, the information of discarded
neurons cannot be completely obscured.
Many researchers have observed this defect and tried to make dropout better regular-
ize CNN. As shown in Figure 1, SpatialDropout [
11
] randomly discards entire channels
from whole feature maps. DropBlock [
12
] randomly discards units in a contiguous region
of a channel instead of substantive units. Guided dropout [
13
], AttentionDrop [
14
], and
CamDrop [
15
] search the influential units in the network through different methods and
drop them to enhance the generalization performance of the network. Furthermore, Auto
Dropout [
16
] is proposed to learn the dropping patterns of SpatialDropout and DropBlock
via reinforcement learning. Although it achieves state-of-the-art results, it requires a huge
computational cost and is more like an extension of the mentioned approaches.
Appl. Sci. 2022, 12, 7682. https://doi.org/10.3390/app12157682 https://www.mdpi.com/journal/applsci
