Citation: Ren, X.; Sun, M.; Zhang, X.;
Liu, L.; Wang, X.; Zhou, H. An AR
Geo-Registration Algorithm for UAV
TIR Video Streams Based on
Dual-Antenna RTK-GPS. Remote Sens.
2022, 14, 2205. https://doi.org/
10.3390/rs14092205
Academic Editors: Kamil Krasuski
and Damian Wierzbicki
Received: 30 March 2022
Accepted: 29 April 2022
Published: 5 May 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
An AR Geo-Registration Algorithm for UAV TIR Video Streams
Based on Dual-Antenna RTK-GPS
Xiang Ren
1
, Min Sun
1,
*, Xianfeng Zhang
1
, Lei Liu
2
, Xiuyuan Wang
1
and Hang Zhou
1
1
Institute of Remote Sensing and GIS, Peking University, Beijing 100871, China; xelmirage@pku.edu.cn (X.R.);
xfzhang@pku.edu.cn (X.Z.); adhamwang@pku.edu.cn (X.W.); hang.zhou@pku.edu.cn (H.Z.)
2
School of Space Information, Space Engineering University, Beijing 101416, China; htgc_dr.liu@nuaa.edu.cn
* Correspondence: sunmin@pku.edu.cn; Tel.: +86-010-6276-4484
Abstract:
In emergency response and disaster rescue, unmanned aerial vehicles (UAVs) onboard
thermal infrared (TIR) sensors are an essential means of acquiring ground information in the nighttime
working environment. To enable field personnel to make better decisions based on TIR video streams
returned from a UAV, the geographic information enhancement of TIR video streams is required. At
present, it is difficult for low-cost UAVs to carry high-precision attitude sensors and thus obtain high-
precision camera attitude information to meet the enhanced processing requirements of UAV TIR
video streams. To this end, this paper proposes an improved Kalman filter algorithm to improve the
geographic registration (geo-registration) accuracy by fusing the positioning and heading data from
the dual-antenna real-time kinematic global positioning system (RTK-GPS) with onboard internal
measurement unit (IMU) data. This method can yield high-precision position and attitude data in
real time based on low-cost UAV hardware, based on which high-precision geo-registration results
can be obtained. The computational complexity can be reduced compared with video stream feature
tracking algorithms. Furthermore, the problem of unstable matching due to the low resolution and
texture level of TIR video streams can be avoided. The experimental results prove that the proposed
method can reduce the registration error by 66.15%, and significantly improve the geo-registration
accuracy of UAV TIR video streams. Thus, it can strongly support the popularization and practicality
of the application of augmented reality (AR) technology to low-cost UAV platforms.
Keywords: TIR video; UAV; augmented reality; RTK; geo-registration
1. Introduction
Low-cost unmanned aerial vehicle (UAV) platforms have a wide range of applications
in emergency rescue and other activities. The use of low-cost UAV platforms to obtain
necessary information for emergency rescue and other operations is a trending topic
in related research. Most previous research has been based on RGB images and video
streams [
1
–
4
]. However, most disaster rescue operations must be performed continuously.
Thermal infrared (TIR) sensors are ideal for the acquisition of information at night by
collecting thermal radiation from ground objects without additional lighting measures, and
for the imaging of ground targets with good stealth; moreover, these sensors can easily be
carried by low-cost UAVs.
However, compared with RGB video cameras, TIR cameras have some limitations. For
example, they have low resolution, and the appearance of targets may differ significantly
from their appearance in RGB video streams. Some distinctive features in the visible
band may also become difficult to distinguish in nighttime TIR images. These limitations
increase the difficulty of the use of TIR video streams by ground personnel to a certain
extent, making it challenging to obtain the state information of the scene solely by relying
on UAV TIR video streams. The overlay of existing geo-information onto real-time UAV
TIR video streams can provide rich information to describe the on-site situation, which
Remote Sens. 2022, 14, 2205. https://doi.org/10.3390/rs14092205 https://www.mdpi.com/journal/remotesensing
