NVIDIA Jetson Xavier NX development board deploys VINS-fusion-GPU

foreword

The CPU performance of the NVIDIA Jetson Xavier NX card is low, but the GPU performance is good, so the GPU version of VINS-fusion can be deployed to speed up the operation. This post records some pitfalls and processes of deployment.

reference

reference documents

1. Install opencv GPU version

The installation of Opencv requires special care . It is not recommended to delete the previous library frequently, because the versions used by multiple projects may be different, so you can install the commonly used version under /usr/local, and install the rest of the versions under your own home .
My NX card has already installed the opencv3.3 version in the /usr/local directory. The 3.4 version of opencv installed this time will not be installed in the system directory, fearing that different versions of files will overwrite conflicts.

(1) Installation dependencies

sudo apt-get install -y cmake libavcodec-dev libavformat-dev libavutil-dev \
    libglew-dev libgtk2.0-dev libgtk-3-dev libjpeg-dev libpng-dev libpostproc-dev \
    libswscale-dev libtbb-dev libtiff5-dev libv4l-dev libxvidcore-dev \
    libx264-dev qt5-default zlib1g-dev libgl1 libglvnd-dev pkg-config \
    libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev mesa-utils  

(2) Download opencv source code

wget -O opencv.zip https://github.com/opencv/opencv/archive/3.4.1.zip # check version
unzip opencv.zip

(3) Compile the GPU version of opencv

The path I installed is: /home/nvidia/opencv/opencv-3.4.1, that is to say, the source code file path decompressed in the previous step is in/home/nvidia/opencv/opencv-3.4.1

cd opencv-3.4.1/ && mkdir build && cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE \
        -D CMAKE_INSTALL_PREFIX=/usr/local \ #设置安装路径，不过由于我已经有其他opencv版本了，该版本并不会安装，因此这里路径无所谓了，只要后面不运行make install
        -D WITH_CUDA=ON \
        -D CUDA_ARCH_BIN=7.2 \ #对于NX和AGX,选7.2，Nano，TX2选6.2 
        -D CUDA_ARCH_PTX="" \
        -D ENABLE_FAST_MATH=ON \
        -D CUDA_FAST_MATH=ON \
        -D WITH_CUBLAS=ON \
        -D WITH_LIBV4L=ON \
        -D WITH_GSTREAMER=ON \
        -D WITH_GSTREAMER_0_10=OFF \
        -D WITH_QT=ON \
        -D WITH_OPENGL=ON \
        -D CUDA_NVCC_FLAGS="--expt-relaxed-constexpr" \
        -D WITH_TBB=ON \
         ../
make
# 建议不要install，避免不同版本的opencv文件覆盖了，以后要使用该cv就设置路径到build文件夹中
# sudo make install

Q1： error Please include the appropriate gl headers before including cuda_gl_interop.h

A1：

sudo vim /usr/local/cuda/include/cuda_gl_interop.h
# 修改cuda_gl_interop.h的62行到68行，如下：
//#if defined(__arm__) || defined(__aarch64__)
//#ifndef GL_VERSION
//#error Please include the appropriate gl headers before including cuda_gl_interop.h
//#endif
//#else
 
 #include <GL/gl.h>

//#endif

2. Install the corresponding CV-Bridge

We need to compile a CV-Bridge from our own source code, and specify opencv as the CV of the GPU version just installed

(1) Download the source code

Click to enter the official website , choose your own ROS version to download:

(2) Modify to your own opencv

gedit vision_opencv/cv_bridge/CMakeLists.txt
# 在find_package(OpenCV 3 REQUIRED前面添加自己的cv路径：
set(OpenCV_DIR "/home/nvidia/opencv/opencv-3.4.1/build")

(3) Compile the source code

cd vision_opencv/cv_bridge
mkdir build
cd build
cmake ..
make
sudo make install

3. Compile and install Vin-fusion GPU

(1) Download the source code

cd ~/catkin_ws/src 
git clone https://github.com/pjrambo/VINS-Fusion-gpu

(2) Modify CMakeLists.txt, specifying the GPU version of CV and CV-Bridge
is actually to compile the ROS package later, as long as a package imports opencv, specify the path of opencv and cv bridge as the path you installed earlier

gedit VINS-Fusion-gpu/camera_models/CMakeLists.txt
# 在最前面添加
set(OpenCV_DIR "/home/nvidia/opencv/opencv-3.4.1/build")

gedit VINS-Fusion-gpu/loop_fusion/CMakeLists.txt
# 在最前面添加
set(OpenCV_DIR "/home/nvidia/opencv/opencv-3.4.1/build")
set(cv_bridge_DIR /usr/local/share/cv_bridge/cmake)  

gedit VINS-Fusion-gpu/vins_estimator/CMakeLists.txt
# 在最前面添加
set(OpenCV_DIR "/home/nvidia/opencv/opencv-3.4.1/build")
set(cv_bridge_DIR /usr/local/share/cv_bridge/cmake)  

(3) Compile the source code

cd ~/catkin_ws
# 由于其他库会依赖camera_models，所以用catkin build编译得先单独编译该模块
catkin build camera_models
catkin build

Q2：

double free or corruption (out)
已放弃 (核心已转储)

A2：

经调试，发现是cv::FileStorage fsSettings读取yaml配置文件中的矩阵时就会报错（读取其他的数据类型如字符串和数值不会报错），可能原因是混用了不同版本的cv，暂时解决方法是直接在代码中输入yaml配置文件中的矩阵：
# 手动输入矩阵数据
cv::Mat cv_T = (cv::Mat_<double>(3, 3)<< 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16);