Article
On Comparing Cross-Validated Forecasting Models with a
Novel Fuzzy-TOPSIS Metric: A COVID-19 Case Study
Dalton Garcia Borges de Souza
1,2,3,
* , Erivelton Antonio dos Santos
4,5
and Francisco Tarcísio Alves Júnior
6,7
and Mariá Cristina Vasconcelos Nascimento
2,3
Citation: de Souza,
D.G.B.; dos Santos, E.A.; Alves Júnior,
F.T.; Nascimento, M.C.V. On
Comparing Cross-Validated
Forecasting Models with a Novel
Fuzzy-TOPSIS Metric: A COVID-19
Case Study. Sustainability 2021, 13,
13599. https://doi.org/10.3390/
su132413599
Academic Editor: Edmundas
Kazimieras Zavadskas
Received: 4 November 2021
Accepted: 7 December 2021
Published: 9 December 2021
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
1
Institute of Science and Technology, Fluminense Federal University, Rio das Ostras 28890-000, Brazil
2
Division of Computer Science, Aeronautics Institute of Technology, São José dos Campos 12228-900, Brazil;
mariah@ita.br
3
Institute of Science and Technology, Federal University of Sao Paulo, São José dos Campos 12247-014, Brazil
4
Institute of Industrial Engineering and Management, Federal University of Itajubá, Itajubá 37500-903, Brazil;
d2020101864@unifei.edu.br
5
Department of Administration Course, José do Rosário Vellano University, Alfenas 37132-440, Brazil
6
Collegiate of Industrial Engineering, University of Amapa State, Macapá 68900-070, Brazil;
francisco.junior@ueap.edu.br
7
Post-Graduate Program in Intellectual Property and Technology Transfer for Innovation, Federal University
of Macapá, Macapá 68903-419, Brazil
* Correspondence: daltonborges@id.uff.br
Abstract:
Time series cross-validation is a technique to select forecasting models. Despite the
sophistication of cross-validation over single test/training splits, traditional and independent metrics,
such as Mean Absolute Error (MAE) and Root Mean Square Error (RMSE), are commonly used
to assess the model’s accuracy. However, what if decision-makers have different models fitting
expectations to each moment of a time series? What if the precision of the forecasted values is also
important? This is the case of predicting COVID-19 in Amapá, a Brazilian state in the Amazon
rainforest. Due to the lack of hospital capacities, a model that promptly and precisely responds
to notable ups and downs in the number of cases may be more desired than average models that
only have good performances in more frequent and calm circumstances. In line with this, this
paper proposes a hybridization of the Technique for Order of Preference by Similarity to Ideal
Solution (TOPSIS) and fuzzy sets to create a similarity metric, the closeness coefficient (CC), that
enables relative comparisons of forecasting models under heterogeneous fitting expectations and
also considers volatility in the predictions. We present a case study using three parametric and
three machine learning models commonly used to forecast COVID-19 numbers. The results indicate
that the introduced fuzzy similarity metric is a more informative performance assessment metric,
especially when using time series cross-validation.
Keywords:
fuzzy sets; TOPSIS; multicriteria decision making; decision support systems; forecasting
1. Introduction
By 27 October 2021, almost two years after the initial occurrence of SARS-COV-2, the
World Health Organization (WHO) announced a total of 219.4 million cases worldwide
and 5 million accumulated deaths due to coronavirus disease [
1
]. Indeed, by 22 November
2021, some countries in Europe have announced a partial or complete lockdown aimed at
overcoming the infections spread across Europe, notwithstanding sustainability economy
problems [
2
]. After 80.2 thousand confirmed cases in the world in almost two months,
Brazilian authorities stated the SARS-COV-2’s primary infection on 25 February 2020 [
3
].
After a lag of two months, Brazil saw its initial pandemic numbers soar. At the end of
October 2021, Brazil had the third-largest number of confirmed cases globally (21.75 million)
and the second-highest number of deaths (606 thousand). Moreover, the number of daily
Sustainability 2021, 13, 13599. https://doi.org/10.3390/su132413599 https://www.mdpi.com/journal/sustainability
