Citation: Wan, Y.; Zhao, Q.; Guo, C.;
Xu, C.; Fang, L. Multi-Sensor Fusion
Self-Supervised Deep Odometry and
Depth Estimation. Remote Sens. 2022,
14, 1228. https://doi.org/10.3390/
rs14051228
Academic Editors: Yangquan Chen,
Subhas Mukhopadhyay, Nunzio
Cennamo, M. Jamal Deen, Junseop
Lee and Simone Morais
Received: 16 December 2021
Accepted: 14 February 2022
Published: 2 March 2022
Publisher’s Note: MDPI stays neutral
with regard to jurisdictional claims in
published maps and institutional affil-
iations.
Copyright: © 2022 by the authors.
Licensee MDPI, Basel, Switzerland.
This article is an open access article
distributed under the terms and
conditions of the Creative Commons
Attribution (CC BY) license (https://
creativecommons.org/licenses/by/
4.0/).
Article
Multi-Sensor Fusion Self-Supervised Deep Odometry and
Depth Estimation
Yingcai Wan
1
, Qiankun Zhao
1
, Cheng Guo
1
, Chenlong Xu
2
and Lijing Fang
1,
*
1
Faculty of Robot Science and Engineering, Northeastern University, Shenyang 110819, China;
1710333@std.neu.edu.cn (Y.W.); 2110701@std.neu.edu.cn (Q.Z.); 1901941@std.neu.edu.cn (C.G.)
2
College of Intelligent Systems Science and Engineering, Harbin Engineering University, Harbin 150001, China;
xurobot@hrbeu.edu.cn
* Correspondence: ljfang@mail.neu.edu.cn; Tel.: +86-138-4019-2905
Abstract:
This paper presents a new deep visual-inertial odometry and depth estimation framework
for improving the accuracy of depth estimation and ego-motion from image sequences and inertial
measurement unit (IMU) raw data. The proposed framework predicts ego-motion and depth with
absolute scale in a self-supervised manner. We first capture dense features and solve the pose by
deep visual odometry (DVO), and then combine the pose estimation pipeline with deep inertial
odometry (DIO) by the extended Kalman filter (EKF) method to produce the sparse depth and pose
with absolute scale. We then join deep visual-inertial odometry (DeepVIO) with depth estimation by
using sparse depth and the pose from DeepVIO pipeline to align the scale of the depth prediction
with the triangulated point cloud and reduce image reconstruction error. Specifically, we use the
strengths of learning-based visual-inertial odometry (VIO) and depth estimation to build an end-to-
end self-supervised learning architecture. We evaluated the new framework on the KITTI datasets and
compared it to the previous techniques. We show that our approach improves results for ego-motion
estimation and achieves comparable results for depth estimation, especially in the detail area.
Keywords: self-supervised; autonomous driving; depth estimation; visual-inertial odometry
1. Introduction
Dense depth estimation from an RGB image is the fundamental issue for 3D scene
reconstruction that is useful for computer vision applications, such as automatic driving [
1
],
simultaneous localization and mapping (SLAM) [
2
], and 3D scene understanding [
3
].
With rapid development of in depth estimation (from monocular), many supervised and
unsupervised learning methods have been proposed. Instead of traditional supervised
methods depending on expensively collected ground truth, unsupervised learning from
stereo images or monocular videos is a more universal solution [
4
,
5
]. However, due to the
lack of perfect ground truth and geometric constraints, unsupervised depth estimation
methods that suffer from inherent scale ambiguity and poor performance, perform well
in some scenarios, such as occlusion, non-textured regions, dynamic motion objects, and
indoor environment.
To overcome the lack of geometric constraints in unsupervised depth estimation train-
ing, recent works have used sparse LiDAR data [
6
–
8
] to guide depth estimation in the
process of image feature extraction and improve the quality of supervised depth map gen-
eration. These methods lead to the dependence on sparse LiDAR data, which are relatively
expensive. A recent trend in depth estimation methods involves traditional SLAM [
9
],
which could provide an accurate sparse point cloud, learning to predict monocular depth
and odometry in a self-supervised manner [10,11].
To integrate visual odometry (VO) or the SLAM system into depth estimation, the
authors of [
10
,
12
,
13
] presented a neural network to correct classical VO estimators in a self-
supervised manner and enhance geometric constraints. Self-supervised depth estimation,
Remote Sens. 2022, 14, 1228. https://doi.org/10.3390/rs14051228 https://www.mdpi.com/journal/remotesensing
