Citation: Chen, C.; Chandra, S.; Han,
Y.; Seo, H. Deep Learning-Based
Thermal Image Analysis for
Pavement Defect Detection and
Classification Considering Complex
Pavement Conditions. Remote Sens.
2022, 14, 106. https://doi.org/
10.3390/rs14010106
Academic Editors: Yangquan Chen,
Subhas Mukhopadhyay,
Nunzio Cennamo, M. Jamal Deen,
Junseop Lee, Simone Morais and
Fabio Tosti
Received: 25 November 2021
Accepted: 24 December 2021
Published: 27 December 2021
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2021 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Deep Learning-Based Thermal Image Analysis for Pavement
Defect Detection and Classification Considering Complex
Pavement Conditions
Cheng Chen
1
, Sindhu Chandra
2
, Yufan Han
3
and Hyungjoon Seo
2,
*
1
Department of Civil Engineering, Xi’an Jiaotong Liverpool University, Suzhou 215123, China;
Cheng.Chen19@student.xjtlu.edu.cn
2
Department of Civil Engineering and Industrial Design, University of Liverpool, Liverpool L69 3BX, UK;
S.Chandra@liverpool.ac.uk
3
Department of Computer Science, Xi’an Jiaotong Liverpool University, Suzhou 215123, China;
Yufan.Han20@student.xjtlu.edu.cn
* Correspondence: hyungjoon.seo@liverpool.ac.uk; Tel.: +44-151-795-7312
Abstract:
Automatic damage detection using deep learning warrants an extensive data source that
captures complex pavement conditions. This paper proposes a thermal-RGB fusion image-based
pavement damage detection model, wherein the fused RGB-thermal image is formed through multi-
source sensor information to achieve fast and accurate defect detection including complex pavement
conditions. The proposed method uses pre-trained EfficientNet B4 as the backbone architecture and
generates an argument dataset (containing non-uniform illumination, camera noise, and scales of
thermal images too) to achieve high pavement damage detection accuracy. This paper tests separately
the performance of different input data (RGB, thermal, MSX, and fused image) to test the influence of
input data and network on the detection results. The results proved that the fused image’s damage
detection accuracy can be as high as 98.34% and by using the dataset after augmentation, the detection
model deems to be more stable to achieve 98.35% precision, 98.34% recall, and 98.34% F1-score.
Keywords:
pavement defect detection; machine learning; thermal analysis; multichannel image fusion
1. Introduction
The growth of urban traffic and the consequent increase in traffic volume over the years
have made the timely maintenance of pavements extremely important. Repetitive traffic
loads [
1
], rapid temperature changes [
2
], and reflection from base layers [
3
] are deemed to
contribute directly to pavement damages. Also, water ingress into initial pavement cracks
can deepen the damage resulting in distresses like potholes, even pavement structural
failures [
4
]. Thus, timely maintenance can not only ensure safe operation but also increase
and the service life of pavements. The current pavement crack detection is manual with
subjective human interpretation and reparation mainly involve filling of the crack with
sealant. Although, automated pavement detection systems have been studied for many
years, previous researches were primarily focused on crack extraction. However, for the
actual complex road conditions, the existing methods have limited error detection rates
to identify all kinds of cases [
5
]. Multi-sensor fusion processing idea for complex road
conditions was considered where acceleration sensors [
6
], infrared sensors [
7
], multi-vision
cameras [
8
], and 3D laser scanning [
9
] can provide additional identification information to
the optical images of the pavement.
Automated pavement detection has undergone several significant technological changes,
and digital image-based methods have been widely used for pavement crack detection and
segmentation. The difference in grayscale values of crack pixels and background of digital
images makes segmentation as well as detection logical [
10
]. Other factors such as lighting
Remote Sens. 2022, 14, 106. https://doi.org/10.3390/rs14010106 https://www.mdpi.com/journal/remotesensing
