1.1 Overview of Depth Camera
Depth camera is a new technology that has emerged in recent years. Compared with traditional cameras, depth camera adds a depth measurement function, so that it is more convenient and accurate to perceive the surrounding environment and changes. There are many application scenarios for depth cameras, and there are many shadows of depth cameras in our daily life. What can it be used for? 3D modeling, unmanned driving, robot navigation, mobile phone face unlocking, somatosensory games, etc. all use depth cameras to realize their functions. The biggest innovation in the iPhone X is the use of a front-facing depth camera. Now, the application of depth cameras has been unlocked by smart terminals, and has gradually extended to mobile payment, liveness detection and other scenarios. The technical means behind it: 3D structured light, has gradually entered the public eye.
The iPhone X uses a structured light solution. Here is the workflow of FaceID to explain how it obtains the depth value:
1. When the face is close to the camera, the proximity sensor is first activated, and if the approach is close, a signal is sent to notify the flood illuminator
2. The flood illuminator will emit unstructured infrared light and project it on the surface of the object, and then the infrared camera will detect whether it is a human face after receiving the light
3. If it is a human face, the dot matrix projector will project more than 30,000 structured light patterns invisible to the naked eye onto the object
4. The infrared lens receives the reflected dot matrix pattern, and calculates the deformation of the pattern to obtain the distance of different positions on the face
1.2 Common Technologies of Depth Cameras
The depth camera based on binocular stereo vision is sensitive to the intensity of ambient light and relies on the characteristics of the image itself. Therefore, it is difficult to extract effective and robust features under conditions such as insufficient illumination and lack of texture, which leads to increased matching errors or even Match failed. The depth camera based on the structured light method is proposed to solve the complexity and robustness problems of the binocular matching algorithm mentioned above. The structured light method does not depend on the color and texture of the object itself, and uses the method of actively projecting known patterns to achieve fast and robust matching feature points, which can achieve high precision and greatly expand the scope of application.
Structured light uses phase information for three-dimensional reconstruction, which mainly includes dephasing and unfolding phase, and uses the unfolded phase to calculate depth information. Phase measurement is an optical three-dimensional measurement technology based on the principle of triangulation, which is essentially a fringe projection measurement technology. It can be seen from its measurement principle that when using this method to measure, it is necessary to obtain the deformed fringe image caused by the change of the mirror profile, obtain the phase distribution of the deformed fringe, find out the relationship between the phase distribution and the mirror surface shape, and then restore the mirror surface 3D shape. After the measurement system based on phase measurement projects periodic cosine fringes on the surface of the measured object, the deformed fringe image collected by the CCD camera (which can be regarded as a two-dimensional space carrier signal whose phase and amplitude are modulated by the object surface contour) can be expressed as :
Among them, I(x, y) is the light intensity of the image collected by the CCD camera; a(x, y) is the background light intensity of the object plane; b(x, y) is the modulation degree; φ(x, y) is the fringe phase . It can be seen that we need to solve the phase distribution of the fringe image to solve the normal vector of the object surface, and then solve the shape of the object surface. Therefore, the phase analysis of fringe images for the purpose of obtaining reliable phase information is also a key link in the mirror three-dimensional measurement technology based on phase measurement deflection. So far, researchers have proposed a variety of methods for solving the phase of fringe images, which can be divided into two categories: spatial domain phase analysis technology and time domain phase analysis technology according to different processing domains.
1.3 Phase solution of depth camera
The whole system of 3D structured light includes structured light projection equipment, camera, image acquisition and processing system. The process is that the projection device emits light onto the measured object, the camera captures the three-dimensional light pattern formed on the measured object, and the captured image is processed by the acquisition and processing system to obtain the surface data of the measured object. In this system, when the relative position of the camera and the projection device is constant, the degree of distortion of the light projected on the measured object depends on the depth of the surface of the object, so a light image with depth can be obtained in the captured image.
The 3D structured light mode includes point, line, and surface modes, which refer to the type of projected light. For example, here the phase difference between point A and point C is needed to calculate the height, so how are these two points reflected in the distorted fringe image captured by the camera?
This is the fringe image taken before and after being modulated. The red point in the picture is equivalent to point A and point C in the above picture. The pixel coordinates of the two points are the same, but the phase is different. The phase difference under the same coordinates is The height of point D that can be obtained indirectly. That is to say, the speckle or fringe structured light patterns in any two places in the space are different. As long as such structured light is applied in the space, the entire space will be marked. Put an object into this space, and just look at the speckle pattern on the object to know where the object is. Of course, the speckle pattern of the entire space must be recorded before this, so the light source calibration must be done first, and the distance between the current object and the camera can be accurately calculated by comparing the spot distribution of the calibration plane.
1.4 Summary
At this stage, it is mainly used for unlocking and identity verification on high-end smart models. The accuracy and speed of unlocking/recognition are better than fingerprint recognition. In terms of virtual interaction, it has also been used in somatosensory games, gesture control and other purposes. The technical conditions for the implementation of 3D structured light in application scenarios are already in place, and more derivative application scenarios are being developed and tried. Generally speaking, the network structure composed of multiple vertical bidirectional lines is the most commonly used. This mode can realize three-dimensional contour measurement without scanning, and the speed is fast.