Article
Automating Visual Blockage Classification of Culverts with
Deep Learning
Umair Iqbal
1,
* , Johan Barthelemy
1
, Wanqing Li
2
and Pascal Perez
1
Citation: Iqbal, U.; Barthelemy, J.;
Li, W.; Perez, P. Automating Visual
Blockage Classification of Culverts
with Deep Learning. Appl. Sci. 2021,
11, 7561. https://doi.org/10.3390/
app11167561
Academic Editors: Nikos D. Lagaros
and Oscar Reinoso García
Received: 2 July 2021
Accepted: 16 August 2021
Published: 18 August 2021
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
1
SMART Infrastructure Facility, University of Wollongong, Wollongong 2500, Australia;
johan@uow.edu.au (J.B.); pascal@uow.edu.au (P.P.)
2
School of Computing and Information Technology, University of Wollongong, Wollongong 2500, Australia;
wanqing@uow.edu.au
* Correspondence: ui010@uowmail.edu.au
Abstract:
Blockage of culverts by transported debris materials is reported as the salient contributor
in originating urban flash floods. Conventional hydraulic modeling approaches had no success
in addressing the problem primarily because of the unavailability of peak floods hydraulic data
and the highly non-linear behavior of debris at the culvert. This article explores a new dimension
to investigate the issue by proposing the use of intelligent video analytics (IVA) algorithms for
extracting blockage related information. The presented research aims to automate the process
of manual visual blockage classification of culverts from a maintenance perspective by remotely
applying deep learning models. The potential of using existing convolutional neural network (CNN)
algorithms (i.e., DarkNet53, DenseNet121, InceptionResNetV2, InceptionV3, MobileNet, ResNet50,
VGG16, EfficientNetB3, NASNet) is investigated over a dataset from three different sources (i.e.,
images of culvert openings and blockage (ICOB), visual hydrology-lab dataset (VHD), synthetic
images of culverts (SIC)) to predict the blockage in a given image. Models were evaluated based on
their performance on the test dataset (i.e., accuracy, loss, precision, recall, F1 score, Jaccard Index,
region of convergence (ROC) curve), floating point operations per second (FLOPs) and response
times to process a single test instance. Furthermore, the performance of deep learning models
was benchmarked against conventional machine learning algorithms (i.e., SVM, RF, xgboost). In
addition, the idea of classifying deep visual features extracted by CNN models (i.e., ResNet50,
MobileNet) using conventional machine learning approaches was also implemented in this article.
From the results, NASNet was reported most efficient in classifying the blockage images with the
5-fold accuracy of 85%; however, MobileNet was recommended for the hardware implementation
because of its improved response time with 5-fold accuracy comparable to NASNet (i.e., 78%).
Comparable performance to standard CNN models was achieved for the case where deep visual
features were classified using conventional machine learning approaches. False negative (FN)
instances, false positive (FP) instances and CNN layers activation suggested that background noise
and oversimplified labelling criteria were two contributing factors in the degraded performance of
existing CNN algorithms. A framework for partial automation of the visual blockage classification
process was proposed, given that none of the existing models was able to achieve high enough
accuracy to completely automate the manual process. In addition, a detection-classification pipeline
with higher blockage classification accuracy (i.e., 94%) has been proposed as a potential future
direction for practical implementation.
Keywords:
convolutional neural networks; visual blockage of culverts; intelligent video analytic;
image classification
1. Introduction
Cross-drainage structures (e.g., culverts, bridges) are prone to blockage by debris
and are reported as one of the leading causes of flash floods in urban areas [
1
–
7
]. The
Appl. Sci. 2021, 11, 7561. https://doi.org/10.3390/app11167561 https://www.mdpi.com/journal/applsci
