LEARNING TO EXPLORE USING ACTIVE NEURAL SLAM Paper Reading

Paper information

Title : LEARNING TO EXPLORE USING ACTIVE NEURAL SLAM
Author : Devendra Singh Chaplot, Dhiraj Gandhi
Project address : https://devendrachaplot.github.io/projects/Neural-SLAM
Code address : https://github.com/devendrachaplot/Neural- SLAM
Source : LCLR
Time : 2022

Abstract

This work proposes a modular and hierarchical approach to learning policies for exploring 3D environments, called "Active Neural SLAM".

Our approach combines the strengths of classical and learning-based methods by using an analytical path planner with a learned SLAM module together with global and local policies.

The use of learning provides flexibility in input patterns (in the SLAM module), exploits the structural regularities of the world (in the global policy), and provides robustness against errors in state estimation (in the local policy).

The proposed model is also easily transferable to the PointGoal task and is the winning entry of the CVPR 2019 Habitat PointGoal Navigation Challenge.

Introduction

While using learning for exploration is well motivated, translating the exploration problem into an end-to-end learning problem has its own drawbacks. Learning mappings, state estimation, and path planning purely from data in an end-to-end manner can be very difficult. Thus, past end-to-end learning work for exploration by Chen et al. (2019) relies on imitation learning and the use of millions of frames of experience, but still performs worse than classical methods that do not require any training at all.

In this paper, we examine alternatives to exploration using learning that preserve the advantages that learning has to offer but without the drawbacks of full-fledged end-to-end learning. Our key conceptual insights are that learning is used in order to exploit structural regularities of indoor environments, robustness to state estimation errors, and flexibility to input patterns. These occur on different time scales and thus can be factored out.

Our proposed exploration architecture consists of a learned neural SLAM module, a global policy, and a local policy, which are connected by a map and an analytical path planner.

The learning neural SLAM module generates free-space maps and estimates agent poses from input RGB images and motion sensors.
The global policy uses agent poses to occupy this free-space map, and uses learning to exploit structural regularities in the layout of real-world environments to generate long-term goals.
These long-term goals are used to generate short-term goals for local policy (using a geometric path planner).
The local policy uses learning to directly map the raw RGB image to what the agent should do.

Using learning in the SLAM module provides flexibility in terms of input modality, the learned global policy can exploit regularities in the real-world environment layout, and the learned local policy can use visual feedback to exhibit more robust behavior.

Related Work

Navigation Approaches

Classical navigation methods divide the problem into two parts: mapping and path planning.

Exploration in Navigation

While many works focus on passive map building, path planning, and goal-driven policy learning, a small number of works address the problem of active SLAM, i.e. how to actively control cameras to build maps.

Hierachical and Modular Policies (hierarchical and policy learning)

Hierarchical reinforcement learning (Dayan and Hinton, 1993; Sutton et al., 1999; Barto and Mahadevan, 2003) is an active research area aimed at automatically discovering hierarchical structures to accelerate learning. However, this has proven challenging, so most works resort to using manually defined hierarchies. For example, in terms of navigation, Bansal et al. (2019) and Kaufmann et al. (2019) design modular policies for navigation, connecting learned policies with low-level feedback controllers. Hierarchical and modular strategies have also been used for embedded question answering (Das et al., 2018a; Gordon et al., 2018; Das et al., 2018b).

Task Setup

Actuation and Noise Model (drive and noise model)

We denote the agent's pose by (x, y, o), assuming the agent moves from $p_0=(0,0,0)$ to start. Now, suppose the agent takes an action. Each action is implemented as a control command for the robot.

Let the corresponding control command be $u{\_}_a=(x_a,y_a,o_a)$ . Let the pose of the agent after the action be$p_1=(x^{\ast },y^{\ast },o^{\ast })$. Driving noise (${ϵ}_{act}$) is the actual agent pose after the action ( $p_1$) with the expected agent pose ( $p_0+\Delta u$ ) Independent function:
$${ϵ}_{act}=p_1-(p_0+u)\_=(x^{\ast }-x_a,y^{\ast }-y_a,o^{\ast }-o_a)$$
Mobile robots typically have sensors that estimate the pose of the robot while it is moving. Let the sensor estimate the pose of the agent after the action is$p_1^{\prime }=(x^{\prime },y^{\prime },o^{\text{'}})$. Sensor noise (${ϵ}_{sen}$) is estimated by the sensor pose ( $p_1^{\prime }$) and the actual agent pose ( $p_1$) is given by the difference between
$${ϵ}_{sen}=p_1^{\prime }-p_1=(x^{\prime }-x^{\ast },y^{\prime }-y^{\ast },o^{\prime }-o^{\ast })$$

We use three default navigation actions: forward: move forward 25 cm, turn right: rotate 10 degrees clockwise in place, turn left: rotate 10 degrees counterclockwise in place. The implementation of the control command is:
$$u_{Forward}=(0.25,0,0)$$$$u_{Right}:(0,0,-10\ast p/180),u_{Left}:(0,0,10\ast p/180)$$

Methods

"Active Neural SLAM". It consists of three components: neural SLAM module, global policy and local policy, as shown in Figure 1. The neural SLAM module predicts environment maps and agent poses based on current observations and previous predictions. The global policy uses the predicted map and agent poses to generate long-term goals. Use path planning to convert long-term goals into short-term ones. Local policies take navigational actions based on current observations to achieve short-term goals.

Map Representation

Active Neural SLAM model internal maintenance space map, $m_t$and agent $x_t$attitude. spatial map $m_t$is a 2 × M × M matrix, where M × M represents the map size, and each element in the spatial map corresponds to a cell of size 25cm2 (5cm × 5cm) in the physical world. Each element in the first channel represents the probability that an obstacle exists at the corresponding location, and each element in the second channel represents the probability that the location is being explored. A cell is considered explored when it is known to be free space or an obstacle. The spatial map is initialized with all zeros at the beginning of the step, $m_0=[0{]}^{2\times M\times M}$ 。

Pose $x_t\in {\mathbb{R}}^3$ represents the x and y coordinates of the agent and the orientation of the agent at time t. The agent always starts facing east from the center of the map at the beginning of the step,$x_0=(M/2,M/2,0.0)$

Neural SLAM Module

Neural SLAM module ( $f_{SLAM}$) Receive the current RGB observation value $s_t$, the agent pose xt − 1 acquired by the current and last sensor : t ′ x^{\prime}_{t−1:t}$x_{t-1:t}^{\prime }$, the last agent pose and map estimation ${\hat{x}}_{t-1}$、$m_{t-1}$, and output the updated map $m_t$and the current agent pose estimate ${\hat{x}}_t$（见图 2）：$m_t,{\hat{x}}_t=f_{SLAM}(s_t,x_{t-1:t}^{\prime },{\hat{x}}_{t-1},m_{t-1}|{\theta }_S)$ , where$i_S$Represents a trainable parameter neural SLAM module.

It consists of two learning components: a mapper and a pose estimator. Mapper( $f_{Map}$) outputs an egocentric top-down two-dimensional spatial map, $p_t^{ego}\in [0,1{]}^{2\times V\times V}$ (where V is the field of view), predicting obstacles and exploration areas in the current observation. pose estimator ($f_{PE}$) based on past pose estimates ( ${\hat{x}}_{t-1}$) and the last two egocentric map predictions ( $p_{t-1:t}^{ego}$) to predict the agent pose ( ${\hat{x}}_t$). It essentially compares the current egocentric map prediction with the last egocentric map prediction transformed to the current frame to predict the pose change between the two maps. Based on the pose estimate given by the pose estimator, the egocentric map from the mapper is transformed into a geocentric map, which is then compared with the previous spatial map ( $m_{t-1}$) aggregated to get the current map ( $m_t$）。

Global Policy

Global Policy以$h_t\in [0,1{]}^{4\times M\times M}$ is the input, where the first two channels of ht are the spatial map$m_t$, the third channel represents the position of the current agent estimated by the SLAM module, and the fourth channel represents the visited position, that is,
$\forall i,j\in 1,2,...,m$ :

in$h_t$We perform two transformations before passing to the global policy model. The first transformation pair comes from $h_t$A window of size 4 × G × G around the agent is subsampled. The second transformation performs a max pooling operation to get $h_t$Obtain an output of size 4×G×G. Both transformations are stacked to form a tensor of size 8 × G × G that is passed as input to the global policy model. The global strategy uses a convolutional neural network to predict the long-term target $g_t^l：g_t^l=\pi G(h_t|{\theta }_G)$ , where$i_G$is a parameter of the global strategy.

Planner

Planner will long-term goals ( $g_t^l$), spatial obstacle map ( $m_t$) and agnet pose estimation ( ${\hat{x}}_t$) as input to calculate the short-term target $g_t^s$ ，即 $g_t^s=f_{Plan}(g_t^l,m_t,{\hat{x}}_t$). It uses $m_t$The fast-marching method (Sethian, 1996) computes the distance from the current agent position to the long-term goal ( $g_t^l$) of the shortest path. Undeveloped areas are considered planning free spaces. We compute the short-term goal coordinates (the farthest point within ds (= 0.25m) from the agent) on the planned path.

Local Policy

The local strategy takes the current RGB observations ( $s_t$) and short-term goals ( $g_t^s$) as input, and output the navigation operation $a_t={Pi}_L(s_t,g_t^s|{\theta }_L)$ , where$i_L$is a parameter of the local policy. The short-term goal coordinates are converted to relative distances and angles from the agent's position before being passed to the local policy. The local policy is a recurrent neural network consisting of a pretrained ResNet18 (He et al., 2016) as the visual encoder.

Experiment

expand

We propose a modular navigation model that leverages the strengths of classical and learning-based navigation methods. We show that the proposed model outperforms previous methods on both the Exploration and PointGoal tasks, and shows strong generalization across domains, objectives, and tasks. In the future, the proposed model can be extended to complex semantic tasks such as semantic object navigation and Ebmbodied question answering by using the Semantic Neural SLAM module , which creates multi-channel maps that capture the semantic properties of objects in the environment. The model can also be combined with previous localization efforts, relocating in previously created maps for efficient navigation in subsequent episodes.