Citation: Cai, G.; Li, J.; Liu, X.; Chen,
Z.; Zhang, H. Learning and
Compressing: Low-Rank Matrix
Factorization for Deep Neural
Network Compression. Appl. Sci.
2023, 13, 2704. https://doi.org/
10.3390/app13042704
Academic Editors: Phivos Mylonas,
Katia Lida Kermanidis
and Manolis Maragoudakis
Received: 5 January 2023
Revised: 15 February 2023
Accepted: 16 February 2023
Published: 20 February 2023
Copyright: © 2023 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Learning and Compressing: Low-Rank Matrix Factorization for
Deep Neural Network Compression
Gaoyuan Cai
1,2
, Juhu Li
1,2,
*, Xuanxin Liu
1,2
, Zhibo Chen
1,2
and Haiyan Zhang
1,2
1
School of Information Science and Technology, Beijing Forestry University, Beijing 100083, China
2
Engineering Research Center for Forestry-Oriented Intelligent Information Processing of National Forestry
and Grassland Administration, Beijing 100083, China
* Correspondence: lijuhu@bjfu.edu.cn
Abstract:
Recently, the deep neural network (DNN) has become one of the most advanced and
powerful methods used in classification tasks. However, the cost of DNN models is sometimes
considerable due to the huge sets of parameters. Therefore, it is necessary to compress these models
in order to reduce the parameters in weight matrices and decrease computational consumption,
while maintaining the same level of accuracy. In this paper, in order to deal with the compression
problem, we first combine the loss function and the compression cost function into a joint function,
and optimize it as an optimization framework. Then we combine the CUR decomposition method
with this joint optimization framework to obtain the low-rank approximation matrices. Finally, we
narrow the gap between the weight matrices and the low-rank approximations to compress the
DNN models on the image classification task. In this algorithm, we not only solve the optimal ranks
by enumeration, but also obtain the compression result with low-rank characteristics iteratively.
Experiments were carried out on three public datasets under classification tasks. Comparisons
with baselines and current state-of-the-art results can conclude that our proposed low-rank joint
optimization compression algorithm can achieve higher accuracy and compression ratios.
Keywords:
deep neural network compression; low-rank matrix factorization; truncated singular
value decomposition; CUR decomposition; joint optimization; optimal rank
1. Introduction
In recent years, DNNs have become prevalent in the machine learning area, applied in
various fields such as computer vision (CV) [
1
,
2
], natural language processing (NLP) [
3
,
4
],
and speech recognition [
5
,
6
]. However, with the increase in the accuracy performance
of real-world applications, the DNN model needs more neurons, thus the increasing
number of layers in DNN induce massive weight parameters, which makes the calculation
resources large. Therefore, it is significant to compress the DNNs while keeping their
accuracy performance when running these compressed models on resource-constrained
embedded devices.
There exist many DNN compressing techniques so far, such as pruning [
7
–
10
], weight
quantization [11–14], and low-rank matrix factorization (MF) [15–18].
Weight pruning is proposed to reduce the weight parameters of the DNN models
while retaining original precision, including removing neurons and automatically learning
the correct number of neurons and weights. However, all the pruning criteria require
the manual setup of sensitivity for each layer and require the fine-tuning of the weight
parameters. Weight quantization intends to compress the DNN model by re-representing
the number of bits required to represent each weight parameter. In fact, the main difficulty
of the weight quantization is that processing low-precision weights and excessive quanti-
zation will make the parameters of DNN model lose important content and information.
The low-rank matrix factorization is to find an approximate matrix with low-rank property
Appl. Sci. 2023, 13, 2704. https://doi.org/10.3390/app13042704 https://www.mdpi.com/journal/applsci
