Robot_ depth camera sensor survey

title: Robot_Depth camera sensor survey
date: 2020-7-23 21:00:00
tags: Robot, Robot, ROS

---

Most of the opinions in this article come from papers, magazines, and blogs collected online.
It is the data compiled during the individual's learning stage. If you make any mistakes, please let us correct them.

There are three main types of depth cameras:

• COOL
  • i-ToF
  • d-ToF
• Binocular
  • RGB binocular
• Structured light
  • Monocular IR+Projection IR dot matrix
  • Binocular IR+projection IR dot matrix
  • Monocular IR+projection phase shift fringe

The following is to compare the indicators of the three mainstream cameras.

Cool

For more detailed information, please refer to: https://blog.csdn.net/dianmao0917/article/details/17389637
Computer Vision Life: https://blog.csdn.net/electech6/article/details/78349107

The full name of ToF is Time-of-Flight, divided into
i-ToF (indirect Time-of-Flight) phase ranging and
d-Tof (direct Time-of-Flight) time ranging

working principle

ToF technology measurement camera refers to the back and forth flight time of the actively projected beam after being reflected by the target surface and received by the camera. Based on the speed of light, the distance from the target to the camera can be obtained.
The ToF sensor gives a modulation signal to the light source drive chip. The modulation signal controls the laser to emit high-frequency pulse modulation (CW sinusoidal/PL pulse) near-infrared light. After encountering the diffuse reflection of the object, the receiving end uses the phase difference or the phase difference between the emitted light and the received light. Time difference to calculate depth information.

According to different modulation methods, there are generally two types: pulse modulation (Pulsed Modulation) and continuous wave modulation (Continuous Wave Modulation). For more in-depth reference to computer vision life: https://blog.csdn.net/electech6/article/details/78349107 information.

Among them, i-ToF is based on the analog signal obtained by CIS (Camera Image Sensor) for ranging.
Among them, d-ToF is based on the digital signal obtained by SPAD Array (Single Photon Avalanche Diode Array) for ranging.

Hardware index

Power consumption

The ToF scheme has high power consumption, full exposure, and high-frequency pulses.
The power consumption of i-ToF is high, the power consumption of
d-ToF is medium

measurement accuracy

Linear relationship with distance
in centimeter level

Resolution

Generally cannot reach VGA (640x480)

Frame rate

High 100+ fps

Mass production calibration

i-ToF trouble
d-ToF medium

Technical bottleneck

1. Flying pixels (Flying pixels)
   Each pixel has a certain physical size. When measuring the edge of an object, a single pixel will receive the light reflected from the foreground and the background at the same time; the energy generated by the two is superimposed together to make the sensor obtain The original data contains multiple distance information, and the wrong depth measurement value will be obtained when the phase is calculated.
   As a result, there are often a large number of erroneous depth measurement values ​​at the edge of the object. After the 3D point cloud is generated, it visually appears as an invalid point flying in the air.
   That is, the 3D information of the edge of the object cannot be effectively obtained.
   Solved by edge detection algorithm.
2. Multi-path interference (Multi-Path Interference, MPI)
   has complex diffuse reflections and even specular reflections in real scenes, which
   in principle will increase the measured value and seriously affect the effect of 3D reconstruction.
   Unable to resolve.
3. Intensity Realted Error (Intensity Realted Error)
   The areas of different reflectivity on the same plane reflect different depths.
4. Trade-off between range and precision
5. High frequency drive
   In order to ensure the measurement accuracy, CW-iToF adopts the method of increasing the modulation frequency, and PL-iToF adopts the drive method of narrow pulse and high peak power. Taken together, iToF's main requirements for drive chips are high modulation frequency and high peak power.
6. On-chip integration The
   mainstream i-ToF sensor pixels are generally around QVGA (320x240)

Binocular

working principle

Triangular geometric parallax to obtain the distance information from the target to the camera.
Specifically, the same object is observed from two cameras, and the position of the observed object in the images captured by the two cameras will have a certain position difference.
The closer the distance, the greater the parallax. When the relative positional relationship between the two cameras is known, the distance from the subject to the camera can be calculated by the principle of similar triangles.

Hardware index

Power consumption

Low to high, most of the power consumption is relatively low

measurement accuracy

Close distance has high accuracy, but the error will increase with the increase of distance and squared.
The distance can reach the millimeter level.

Resolution

RGB binocular up to 2k resolution

Frame rate

High 60fps

Mass production calibration

easy

Technical bottleneck

1. The amount of calculation is large.
   Solved by adding algorithm chip ASIC.
2. Depends on the texture of the subject and ambient lighting.
   For example, the white wall cannot match the corresponding pixel.

Structured light

working principle

The essence of the principle is the same as the dual purpose, but also uses the triangulation method.

Monocular IR+projection IR dot matrix
Binocular IR+projection IR dot matrix
Monocular IR+projection phase shift fringe

Hardware index

Power consumption

In, need to project the pattern, only illuminate a local area

measurement accuracy

Close distance has high accuracy, but the error will increase with the increase of distance and squared.
The distance can reach the millimeter level.

Resolution

Can reach 1080P

Frame rate

30fps lower

Mass production calibration

Medium difficulty

Technical bottleneck

1. High computational load.
   Solved by adding algorithm chip ASIC.
2. The data missing at the deep mutation
   can be solved by ARM-based soft core algorithm.

Market mainstream solution

Microsoft、Intel、Leap Motion、Orbbec、图漾、Occipital Structure、Stereolabs 、DUO。

Intel Realsense R200

Type: The binocular IR

infrared projector projects infrared speckles. The two infrared cameras on the left and right collect two infrared images. The processor calculates the disparity map according to the speckle feature points in the two infrared images, and finally obtains the depth map.

Distance: indoor 0.5m-3.5m, the farthest outdoor is 10m.
Application scenario: indoor. Outdoors are greatly affected by ambient light (the infrared transmitter has limited power)

The SDK is highly compatible and supports C++, C#, JavaScript, Processing, Unity, and Cinder frameworks. It should be noted that the R200 SDK only supports face tracking, not gesture tracking and skeleton tracking.

Intel Realsense SR300

Type: Binocular structured light (officially called coded light)

Distance: 0.2-1.5 meters.
Application scenarios: face and hand tracking. Laptops, Pads, virtual reality devices.

Description:
https://www.intelrealsense.com/coded-light/
forum: https: //community.intel.com/t5/tag/Intel%C2%AE%20RealSense/tg-p/tag-id/513
procurement :
Https://store.intelrealsense.com/buy-intel-realsense-depth-module-sr300.html

Intel Realsense D435

Type: structured light (active stereo IR)

Distance: 0.1-10 meters
Introduction:
https://www.intelrealsense.com/zh-hans/depth-camera-d435/

Ledong TOF IDC3224R_LD01

3D TOF high resolution depth camera

Range: 0.2-4 meters
Resolution: 320x240
FOV: 92°X 74°X 57°

Application scenarios: robot precise positioning and mapping, navigation and obstacle avoidance, face/body/object recognition, AR/VR, etc.
Introduction: https://www.ldrobot.com/product/43

Microsoft Kinect V1 V2

V1 is based on structured light (officially called coded light)
V2 is based on TOF

V1 has been discontinued.
V2 supports six people's skeleton tracking, basic gesture operations and face tracking, supports Cinder and Open Frameworks, and has a built-in Unity 3D plug-in

LeTV LeTMC-520 (actually made by Obi Zhongguang)

Trinocular
IR projection

Orbbec Astra

Obi Zhongguang has: Orbbec Astra, Orbbec Astra Mini, Orbbec Persee,
such as the first generation Orbbec Astra and Pro
type: IR structured light. The
depth maps are all VGA (640×480) resolution @30FPS. The difference lies in the resolution of the color camera. Astra provides VGA@30FPS, while Astra Pro provides 720p@ 30 FPS.

The SDK is relatively simple, supports basic gesture tracking, and can be used for human-computer interaction for gesture recognition, but does not support skeleton extraction. The farthest measurement range can reach 8m. Therefore, Orbecc Astra is more suitable for long-distance indoor applications. However, the device only supports the C++-based OpenNI framework.
Official website: https://orbbec3d.com/

Orbbec Astra Mobile 3D camera

In order to meet the 3D vision requirements of small handheld terminals, Orbbec successfully developed the Astra Mobile 3D camera to meet the various application scenarios of mobile/tablet computer manufacturers.

Can't find the specific purchase link for the plan

https://orbbec3d.com/mobile/

Tuyang RGBD series FS830-HD

Type: Active infrared binocular depth camera (with RGB)
https://item.taobao.com/item.htm?spm=a1z10.1-cs.w4004-18920235119.6.5f2214a9IyeoVQ&id=591956273890
USB2.0 consumer grade camera, HD depth Resolution, 720P RGB resolution, RGB alignment, suitable for robot obstacle avoidance, face brushing, living body detection and other applications.
There is also a FS830-Mobile application that is consumer

RGB-D alignment: Yes
Power consumption: IDLE 1.5／WORK 3.5／TRIGGER 3.0
Output interface: USB
Working range: 240~3500mm
Depth resolution: 1280x960
Depth frame rate: 15
RGB resolution: 2592x1944
Baseline distance: 25
data unit mm: 1.0
Synchronous acquisition: Yes
With RGB: Yes

Support Windows, Linux, Android, ROS platforms. And multiple devices work at the same time without interference. It is suitable for long-distance application scenarios that do not require high frame rate.
Introduction: https://www.percipio.xyz/dev_detail/?model_id=288
Official website: https://www.percipio.xyz/
Purchasing link: https://shop564213940.taobao.com/

Occipital Structure

https://structure.io/

More

https://zhuanlan.zhihu.com/p/32375622
Microsoft、Intel、Leap Motion、Orbbec、图漾、Occipital Structure、Stereolabs 、DUO