Multi-focus Image Fusion Based on Laplacian Pyramid

1. Introduction to multi-focus image fusion

At a specific focal length setting of the optical lens, only objects within the depth-of-fifield (DOF) range will have a clear appearance in the photo, while other objects will be blurred. Multi-focus image fusion is to take pictures with different focal length settings Multiple images of the same scene are fused to obtain a full-scene focused image.

At present, multi-focus image fusion methods can be divided into: transform domain methods, spatial domain methods and deep learning methods.

The methods involved in the above figure are summarized at https://github.com/xingchenzhang/MFIF .

2. Laplacian Pyramid Image Fusion

Paper: A Multi-focus Image Fusion Method Based on Laplacian Pyramid
Code: https://github.com/sjawhar/focus-stacking/tree/87f87a5a436dd07aa53ebfa837d776632ea9f57b

2.1 Gaussian Pyramid

The bottom of the pyramid is a high-resolution representation of the image to be processed, while the top is a low-resolution approximation, with higher levels resulting in smaller and lower-resolution images.

The generation process of the Gaussian pyramid is:

1. For a given image, first do a Gaussian smoothing process, that is, use the following convolution kernel to perform convolution operations on the image;
   

2. Then sample the image, remove the even-numbered rows and even-numbered columns in the image, and then get a down-sampled picture;

3. Cycle 1) and 2) on this picture to get a Gaussian pyramid.

2.2 Laplacian Pyramid

The Laplacian pyramid is a residual image obtained by subtracting the upsampled image from the previous layer (low resolution) image from the source image at each layer of the Gaussian pyramid.

The upsampling process is as follows:

1. Expand the low-resolution image to twice the original size in each direction, and fill the new rows and columns with 0;
2. Use the same previous convolution kernel (multiplied by 4) to convolve with the enlarged image to obtain the approximate value of the "new pixel";
3. Compared with the original image of the same resolution, the enlarged image will lose information, and the single-layer Laplacian image can be obtained by subtracting the two;
   

2.3 Laplacian image fusion

2.3.1 Fusion pipeline

The Laplacian pyramid contains the edge details of each layer of the image (the area with more detailed information represents the focus area), so the Laplacian pyramids corresponding to the two images are fused through a certain fusion strategy, and finally from the Laplacian By performing image reconstruction with the Si pyramid, a focused image of the whole scene can be obtained.

The most important thing here is to design the fusion strategy. On the one hand, it is to consider the fusion granularity: pixel-level and region-level, and on the other hand, it is necessary to focus on the pixel or region. To put it simply, it is necessary to design a function to judge the corresponding positions (pixels/areas) of the layer-by-layer Laplacian images of the input two images, and which image at this position is more focused (clearer).

In this paper, the fusion strategy of the top layer (blue) of the Laplacian pyramid is inconsistent with the fusion strategy of the remaining layers (green).

2.3.2 Top-level fusion strategy

This paper adopts the region-level fusion strategy, and uses variance and entropy for region focus discrimination for the top-level focus discrimination;

• Variance: The formula of variance is not specifically introduced, it is used here, that is to say, when the current local block is in the detail area of ​​the Laplace image, the variance is larger, and when it is in the smooth area, the variance is smaller;

• Entropy: The entropy here adopts the algorithm of unary entropy of the image. First, count the frequency of each gray level of 0-255 in all pixels in the entire image entropy, and then calculate the entropy of the center point according to the following figure. In the Laplacian image, the edge area accounts for less and the smooth area accounts for more, due to $-$ The negative correlation curve of the l o$g$$function$$,\; so\; the\; entropy\; of\; pixels\; in\; an\; edge\; region\; (small\; frequency\; value)\; will\; be\; larger\; than\; the\; entropy\; of\; pixels\; in\; a\; smooth\; region\; (large\; frequency\; value).$

So, for fusion at the top (lowest resolution) level of the Laplacian pyramid :

• When the variance and entropy of image A in the local block where a certain pixel is located are greater than those of image B, the value of the current pixel at the top layer of fused Laplacian takes the value of the corresponding pixel at the top layer of Laplacian of image A.
• When the variance and entropy of image B in the local block where a certain pixel is located are greater than those of image A, the value of the current pixel at the top layer of fused Laplacian takes the value of the corresponding pixel at the top layer of Laplacian of image B.
• If it does not belong to the above two cases, the value of the current pixel on the top layer of fused Laplacian is the average value of the corresponding pixels on the top layer of Laplacian in image A and image B.

2.3.3 Fusion strategy of other layers

For the remaining layers, the region fusion strategy is also adopted, as follows, that is, each pixel is convolved with a convolution kernel similar to a Gaussian kernel, and the convolution result is used as the entropy of the current pixel. Since the Laplacian image itself can reflect the focus information, there are many details on the edge of the focus area, and the absolute value of the Laplacian pixels is also large. Here, the convolution operation is used to aggregate the pixel values ​​​​of the neighborhood. Finally, the value of a certain pixel in the fused Laplacian image is equal to the value of the corresponding pixel in the layer with the largest entropy value.

2.3.4 Reconstruction of fused images

Finally, by fusing Laplacian, a fused clear image containing full scene focus can be reconstructed.

reference

1. Multi-focus Image Fusion: A Benchmark
2. Deep Learning-based Multi-focus Image Fusion: A Survey and A Comparative Study