0. Background
Wide-angle camera modules are becoming more and more popular in mobile phones. They generally appear in the rear camera modules of mobile phones, and are even equipped in the front cameras of some mobile phones. Its FOV is generally not less than 110°. Compared with conventional camera modules, it can obtain a wider field of view, and has its unique advantages in shooting scenes and scene performance. The picture below shows the shooting effect of a common wide-angle lens compared to ordinary lenses. Comparison chart.
But on the other hand, the input image obtained by the wide-angle camera has a large distortion, and the intuitive feeling is that the image is curved at the edge of the image. In view of this phenomenon, the camera can be calibrated to obtain the internal reference, and then the input image can be distorted to obtain the corrected image, which can eliminate the distortion, as shown in the figure below.
At present, this method is used in most mobile phones, and the imaging effect is very good when shooting landscapes. In terms of technical implementation, it can be converted into a grid interpolation operation. Through SIMD, multi-threading and other operations, a high running frame rate can be obtained on the CPU, or the GPU can be used for interpolation rendering; real-time preview and video recording are not a problem. However, when shooting a scene with people and the portrait is at the edge of the image, the "stretch" operation of the distortion correction will make the portrait area in the result image wider and the proportion will be out of balance, resulting in a bad visual effect, as shown in the figure below.
This leads to the topic of today's discussion, how to improve the existing distortion correction algorithm that causes portrait stretching, that is, how to implement a wide-angle distortion correction algorithm with portrait protection effect.
1. Technical solution
At present, there are roughly three mainstream technical solutions, which are listed one by one.
1.1 Based on spherical transformation
The core of this method lies in the spherical transformation. For the spherical transformation, please refer to the link . One of its characteristics is that compared with the input image, the surrounding images of the output image are not proportionally shrunk, and the image in the central area will be enlarged to a certain extent. Apply this operation to the distortion-corrected image, and the proportion of the portrait at the edge of the result image can be restored to a certain extent due to the shrinkage operation, as shown in the figure below.
In actual engineering, it can be combined with the face detection operation to determine whether there is a portrait when taking a photo, and judge whether further processing is required based on its position from the center of the image and the size of the portrait (the farther away from the center of the image, the more serious the deformation of the portrait with a larger area ). If not needed, only perform distortion correction on the input image; if necessary, perform the above-mentioned spherical transformation after distortion correction. The processing flow during preview is similar, except that a transition process needs to be added to the two deformation operations to avoid image jitter caused by sudden switching.
The advantage of this method is that the amount of calculation is small, the implementation is simple, and the technical implementation is still an interpolation operation; the disadvantage is that there is no difference in the operation of the whole image, and the background will also be affected and cause it to bend again. Generally speaking, this is a "two guarantees and one" strategy, which cannot have both fish and bear's paws, but it is indeed implemented in some low-end mobile phones.
1.2 Based on optimization strategy
The representative paper of this method is "Distortion-Free Wide-Angle Portraits on Camera Phones", which is a paper published by Google researchers in 2019. For the specific effect, see https://36kr.com/p/1723902918657 . The specific implementation process is relatively complicated, but the effect is indeed significantly improved compared with the first method. Here is a brief introduction to the algorithm process.
step1. Use deep learning for face detection and portrait matting to obtain the foreground area that needs to be protected. In the paper, it is mainly the head area; the rest of the area is regarded as the background area
step2. Obtain the distortion correction result map according to the distortion correction of the input image, as the result reference of the above-mentioned background area; perform spherical transformation on the distortion correction result map to obtain the spherical transformation result map, as the result reference of the above-mentioned foreground area;
step3. The foreground and background need to be smoothly transitioned to avoid abrupt transformations at the boundaries of the result graph; supplemented by some other constraints, the above problem is converted into an optimization problem of transformed grids for solution
step4. After solving the transformed grid, do interpolation to obtain the final result map
The figure below succinctly illustrates the core idea of the algorithm (from the PPT provided by the author of the paper)
The advantage of the algorithm is that it takes into account both the processing of portraits (mainly faces) and the background, and the improvement effect is significant; the disadvantage is that the calculation process is relatively complicated, and it has high requirements for platform computing power, so it cannot be applied in the preview.
1.3 Based on deep learning
Check out a new paper published this year, see https://arxiv.org/pdf/2104.12464.pdf . The author implements the above optimization-based solving operation through deep learning. It is said that some disadvantages of the method in 1.2 can be improved. For example, the slight impact on the background around the portrait area can be reduced or even eliminated. I am not familiar with this aspect personally, so I won’t talk about it. Interested students can study it by themselves.
2. Programming implementation
At present, the calibration algorithm of the wide-angle module has been completed, and a small batch of samples is used for calibration to obtain a reference internal parameter data and apply it to the entire batch of modules. At the same time, based on the method in 1.2, the portrait protection distortion correction algorithm is realized, which basically reproduces the effect in the paper. After the algorithm is optimized, it can complete the processing within 1.2s under the input of a 13M size image in a mid-to-high-end Android phone, which is not inferior to the effect in mainstream phones. Here are a few effect comparison pictures.
