Citation: Boone, J.; Goodin, C.;
Dabbiru, L.; Hudson, C.; Cagle, L.;
Carruth, D. Training Artificial
Intelligence Algorithms with
Automatically Labelled UAV Data
from Physics-Based Simulation
Software. Appl. Sci. 2023, 13, 131.
https://doi.org/10.3390/
app13010131
Academic Editors: M. Jamal Deen,
Subhas Mukhopadhyay,
Yangquan Chen, Simone Morais,
Nunzio Cennamo and Junseop Lee
Received: 28 November 2022
Revised: 13 December 2022
Accepted: 16 December 2022
Published: 22 December 2022
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Training Artificial Intelligence Algorithms with Automatically
Labelled UAV Data from Physics-Based Simulation Software
Jonathan Boone
1
, Christopher Goodin
2
, Lalitha Dabbiru
2
, Christopher Hudson
2
, Lucas Cagle
2,
*
and Daniel Carruth
2,
*
1
Information Technology Laboratory, United States Army Engineer Research and Development Center,
3909 Halls Ferry Road, Vicksburg, MS 39180, USA
2
Center for Advanced Vehicular Systems, Mississippi State University, Box 5405,
Mississippi State, MS 39762, USA
* Correspondence: ldc290@msstate.edu (L.C.); dwc2@cavs.msstate.edu (D.C.)
Abstract:
Machine-learning (ML) requires human-labeled “truth” data to train and test. Acquiring
and labeling this data can often be the most time-consuming and expensive part of developing trained
models of convolutional neural networks (CNN). In this work, we show that an automated workflow
using automatically labeled synthetic data can be used to drastically reduce the time and effort
required to train a machine learning algorithm for detecting buildings in aerial imagery acquired
with low-flying unmanned aerial vehicles. The MSU Autonomous Vehicle Simulator (MAVS) was
used in this work, and the process for integrating MAVS into an automated workflow is presented in
this work, along with results for building detection with real and simulated images.
Keywords:
artificial intelligence; machine-learning; smart trained models; convolutional neural
networks; simulator; synthetic image data; human-labeled
1. Introduction
Semantic segmentation using neural networks has enabled rapid advances in machine
vision and scene understanding in recent years [
1
]. Acquiring adequate labeled training
data is often the most significant challenge when using machine learning. Recent work
has shown that using physics-based simulation can increase the amount and diversity
of labeled training data while simultaneously reducing the cost and time required to
collect and semantically label raw data [
2
]. The problem with current approaches that use
simulation is that it is difficult to automatically create digital assets—synthetic terrains
with desired characteristics—without a human in-the-loop. In this work, we implement a
technique where the digital assets are created automatically and regenerated for different
images, thereby introducing both automation and randomization into the training process.
To address and reduce the substantial labeled data requirements of modern machine
learning techniques, active learning has become an area of intense research. Active learning
in its traditional formulation allows a machine learning algorithm to query an oracle to
label unsupervised data during training [
3
]. Current research in active learning is directed
towards selecting the most useful sample to label automatically. However, recent work
has reversed the selection problem: instead of selecting the best sample to label from a
pre-generated set, the problem is instead that of selecting a sample to generate. This can
be in the form of moving a drone for novel viewing perspectives [
4
] or even selecting
simulation parameters used to generate synthetic data batches [5].
More recent work has shown that automated machine-learning can be achieved with-
out human intervention in the model training phase [
6
]. However, these results were
achieved by using publicly available datasets of cats and dogs (CIFAR-10) and everyday
objects (ImageNet). As these labeled datasets still required a tremendous of time and
manpower to acquire and label, the process is not truly automated. In contrast, this work
Appl. Sci. 2023, 13, 131. https://doi.org/10.3390/app13010131 https://www.mdpi.com/journal/applsci
