Background introduction
The random vibration of a car has the following characteristics:
1. The vibration has no fixed period, and the amplitude, speed and acceleration at a certain moment t cannot be predicted
2. The test is carried out under the same conditions, and the results of each record cannot be completely consistent
3. The high-frequency vibration of the car produces intra-frame blur to the camera imaging, and the low-frequency vibration produces inter-frame blur
Due to the inherent characteristics and technical characteristics of electronic image stabilization technology, electronic image stabilization technology is difficult to adapt to large-scale jitter. Electronic image stabilization algorithms are generally used as secondary stabilization after the gyro-stabilized platform or random jitter of the carrier with relatively small jitter. For vehicle-mounted Camera scenes or fixed camera scenes are more suitable.
Experiments show that the grayscale projection algorithm is effective in eliminating or reducing the random jitter of the carrier (small translational movement or rotation), and the jitter phenomenon of the image sequence after electronic image stabilization is reduced.
Electronic image stabilization algorithm-grayscale projection algorithm
Electronic image stabilization algorithms mainly include block matching method (BMP), representative point method (RPMA), bit plane method (BPM), optical flow method and grayscale projection method (GPA). According to the characteristics of the vehicle image series, a grayscale projection algorithm is used . The gray-scale projection algorithm uses the gray-scale projection curves of two frames of images to perform cross-correlation calculations, which can more accurately obtain the motion vector of the image, and the calculation speed is fast.
1. Grayscale projection mapping
Each frame of two-dimensional image is decomposed into two independent one-dimensional vectors in the horizontal and vertical directions. The grayscale projection formula of the k-th frame image:
G k ( y ) = ∑ x G k ( x , y ) G k ( x ) = ∑ y G k ( x , y ) G_{k}(y)=\sum_{x}G_{k}(x,y)\\ G_{k}(x)=\sum_{y}G_{k}(x,y) Gk(y)=x∑Gk(x,y)Gk(x)=y∑Gk(x,y)
G k ( x , y ) G_{k}(x,y) Gk(x,y ) isxxx行yyPixel value of column y , G k ( y ) G_k(y)Gk( and )是yyCumulative sum of pixels in column y .
2. Curve projection filtering
Each frame of image contains unique edge information. When the offset between adjacent frames is large, the trough value of the cross-correlation calculation curve will be affected by the edge information of the image, resulting in a decrease in calculation accuracy. Cosine filtering can reduce the impact of edge information and retain the 1-grayscale projection information in the middle part of the image. This method is one of the best choices to improve estimation accuracy .
The projection filtering effect is good and the calculation amount is small. However, the cosine operation is complicated for FPGA. Since the occupation of logic resources is difficult to estimate, in order to ensure accuracy, use the unaffected intermediate image part as much as possible when doing image processing .
3. Projection cross-correlation calculation
After the projection filtering process, the smoothed mapped gray waveforms of the current frame and the reference frame are cross-correlated, and the projection curves of the current frame and the reference frame are matched, and finally the row and column cross-correlation curves are obtained. Finding the extremum point of the curve can determine the motion offset vector in the horizontal direction and vertical direction of the image.
There are many ways to calculate cross-correlation. The following formula is the method of calculating the difference that is often used in the paper:
C o r r o w ( u ) = ∑ x = 1 H ( G k ( x + u ) − G k − 1 ( x + m ) ) 2 , 1 ≤ u ≤ 2 m + 1 C o r c o l ( v ) = ∑ y = 1 W ( G k ( y + v ) − G k − 1 ( y + n ) ) 2 , 1 ≤ v ≤ 2 n + 1 Cor_{row}(u)=\sum_{x=1}^{H}(G_{k}(x+u)-G_{k-1}(x+m))^{2} ,1\leq u\leq2m+1 \\ Cor_{col}(v)=\sum_{y=1}^{W}(G_{k}(y+v)-G_{k-1}(y+n))^{2} ,1\leq v\leq2n+1 \\ Corrow(u)=x=1∑H(Gk(x+u)−Gk−1(x+m))2,1≤u≤2 m+1Corcol(v)=y=1∑W(Gk(y+v)−Gk−1(y+n))2,1≤v≤2n _+1
Among them, G k G_{k}GkRepresents the current frame, G k − 1 G_{k-1}Gk−1represents the reference frame, H represents the image height, W represents the image width, C orrow ( u ) Cor_{row}(u)Corrow( u ) represents the row cross-correlation value at position u. m and n represent the search width in the vertical and horizontal directions. The larger the search range, the better the image stabilization performance of the system and the greater the calculation load.
Let’s complain: 1. Many papers use the difference between the row grayscale projection curves of the current frame and the reference frame to calculate cross-correlation. The lower the cross-correlation result, the more consistent the two contrast line segments are, that is, looking for the valley of the cross-correlation curve ;
2. There are many errors in the cross-correlation calculation formulas that many papers want to express here. More importantly, they do not understand the meaning of the parameters in the formulas, resulting in formulas full of errors.
The figure below is a cross-correlation curve calculated using matlab's xcorr function. Find the sum of products of the grayscale projection curve, find the peak value of the cross-correlation curve, and then analyze the motion offset of the peak value.
4. Motion compensation algorithm
Due to the accumulated error generated by adjacent frames, frame-by-frame compensation may cause image stabilization to fail. Once the image goes out of the fixed frame area, the fixed frame compensation algorithm will cause image mismatch. According to the advantages and disadvantages of the fixed frame and adjacent frame compensation algorithms, conditions are set to replace the fixed frame to compensate for the shortcomings between the two.
Usually, the difference between the current frame and the grayscale image of the reference frame is used as a fixed threshold to judge whether to switch the reference frame, which is not suitable for hardware transplantation. But setting thresholds with counters is a good solution. Count the input video sequence, and when the counter exceeds a certain number, use the next frame as the new reference frame.
Subsequent optimization
However, the traditional grayscale projection method requires that the grayscale changes of the image should be rich and have a certain contrast. In addition, the traditional projection method is to project the entire image in rows and columns. If there are local moving objects in the image, the accuracy of the projection algorithm will be reduced.
1. Median filtering improves projection accuracy
Filter the collected images to eliminate noise and improve the accuracy of image projection, using the 3x3 median filtering method
2. Histogram equalization enhancement processing
Processing images requires a certain contrast. Use grayscale histogram equalization to achieve contrast enhancement, clear images, and easy detection of inter-frame motion in image sequences.
3. Local motion in the image leads to misjudgment as global motion.
Case 1: (The image does not shake, and the movement of objects in the picture is misjudged as image shaking)
Case 2: (The image shakes, and the movement of objects in the picture has a certain impact on the grayscale projection algorithm)
Case 1 is basically the same as Case 2. At the initial stage when the object enters the image, the gray scale change area is small, which does not affect the gray scale projection algorithm.
Grayscale projection algorithm simulation effect demonstration
Original algorithm effect (left is the original video, right is the image stabilization result):
car_GP
Algorithm optimization effect (the left is the original video, and the right is the image stabilization result): the calculation of row and column displacement is more precise, and the calculation takes a long time
car_GP_imp
Test code code path:
https://github.com/AomanHao/ISP_Video_Stabilization
references:
"Research on Electronic Image Stabilization Technology Based on FPGA"
"A Vehicle Electronic Image Stabilization Method Based on Grayscale Projection Algorithm"