Citation: Vásquez-Coronel, J.A.;
Mora, M.; Vilches, K. Fast Training of
Extreme Learning Machine
Autoencoder based on Iterative
Shrinkage-Thresholding
Optimization Algorithm. Appl. Sci.
2022, 12, 9021. https://doi.org/
10.3390/app12189021
Academic Editors: Krzysztof
Ejsmont, Aamer Bilal Asghar, Yong
Wang and Rodolfo Haber
Received: 3 August 2022
Accepted: 5 September 2022
Published: 8 September 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Training of an Extreme Learning Machine Autoencoder Based on
an Iterative Shrinkage-Thresholding Optimization Algorithm
José A. Vásquez-Coronel
1,†
, Marco Mora
2,3,
*
,†
and Karina Vilches
2,4,†
1
Doctorado en Modelamiento Matemático Aplicado, Universidad Católica del Maule, Talca 3480112, Chile
2
Laboratory of Technological Research in Pattern Recognition (LITRP), Universidad Católica del Maule,
Talca 3480112, Chile
3
Departamento de Ciencias de la Computación e Industrias, Universidad Católica del Maule,
Talca 3480112, Chile
4
Departamento de Matemáticas, Física y Estadística, Universidad Católica del Maule, Talca 3480112, Chile
* Correspondence: mmora@ucm.cl
† These authors contributed equally to this work.
Abstract:
Orthogonal transformations, proper decomposition, and the Moore–Penrose inverse are
traditional methods of obtaining the output layer weights for an extreme learning machine autoen-
coder. However, an increase in the number of hidden neurons causes higher convergence times and
computational complexity, whereas the generalization capability is low when the number of neurons
is small. One way to address this issue is to use the fast iterative shrinkage-thresholding algorithm
(FISTA) to minimize the output weights of the extreme learning machine. In this work, we aim to
improve the convergence speed of FISTA by using two fast algorithms of the shrinkage-thresholding
class, called greedy FISTA (G-FISTA) and linearly convergent FISTA (LC-FISTA). Our method is
an exciting proposal for decision-making involving the resolution of many application problems,
especially those requiring longer computational times. In our experiments, we adopt six public
datasets that are frequently used in machine learning: MNIST, NORB, CIFAR10, UMist, Caltech256,
and Stanford Cars. We apply several metrics to evaluate the performance of our method, and the
object of comparison is the FISTA algorithm due to its popularity for neural network training. The
experimental results show that G-FISTA and LC-FISTA achieve higher convergence speeds in the
autoencoder training process; for example, in the Stanford Cars dataset, G-FISTA and LC-FISTA are
faster than FISTA by 48.42% and 47.32%, respectively. Overall, all three algorithms maintain good
values of the performance metrics on all databases.
Keywords:
autoencoder; feature extraction; extreme learning machine; shrinkage-thresholding algorithms
1. Introduction
In pattern recognition systems, efficient methods of feature selection and extraction
can reduce the dimensionality problem, thus reducing both the computation time and the
memory requirements of the training algorithms [
1
]. An autoencoder is a feed-forward
neural network that builds a compact representation of the input data, and is mainly used
for unsupervised learning [
2
]. It is composed of an encoder and a decoder: the encoder
reads the input data and maps them to a lower-dimensionality space, while the decoder
reads the compact representation and reconstructs the neural network input. In the same
way as for all supervised learning neural networks [
3
,
4
], the core aspect of the training of an
autoencoder is the backpropagation algorithm. This algorithm iteratively tunes the weights
and biases of the neural network by applying the gradient descent method. Autoencoder
networks are in great demand in multiple applications in modern society, for example, for
dimensionality reduction, image retrieval, denoising, and data augmentation [2].
Several works in the literature have developed feature extraction methods using au-
toencoders and backpropagation. A comparative study between the performance of a
Appl. Sci. 2022, 12, 9021. https://doi.org/10.3390/app12189021 https://www.mdpi.com/journal/applsci
