论文阅读——Retinal Vasculature Segmentation Using Local Saliency Maps and Generative Adversarial Networks For Image Super Resolution
from IBM Research Australia
ICIAR2018
Abstract
This paper proposes a super-resolution reconstruction based on a generative confrontation network, and generates a high-resolution image with scale=16 based on a low-resolution fundus image. This can assist in sophisticated automated image analysis, especially for the diagnosis of fuzzy or small images. The local saliency map marks the importance of each pixel, and the saliency loss is calculated according to the weight of each point as a part of the GAN loss function. The experimental results show that the generated high-resolution image is visually closer to the original image, and the accuracy of using the super-resolution image for retinal vessel segmentation is close to the original image.
Section I Introduction
Conventional retinal fundus images require higher resolution to detect and segment early pathological rents, but they are not sufficient to detect schooled, inconspicuous microaneurysms, bleeding or small blood vessel branches. With the help of the image super-resolution algorithm, high-precision super-resolution images can be obtained from a single video disc image, which is very suitable for remote ophthalmology, because the transmission of LR low-resolution images in the process, and super-resolution images help Detecting new blood vessels and segmenting small blood vessel branches are not satisfied in the original LR image. Ophthalmologists can use it to analyze abnormalities in suspicious areas.
Medical image super-resolution algorithms often rely on examples or self-similarity (self similarity) methods, but they all rely heavily on external data; then the parameterized generative model can learn and map from the original LR image to the high-resolution MR Brain image and ventricle image. But the computing power is very demanding, because it is necessary to search for the candidate patch with the highest matching degree in the training set as a reference for HR images; other methods include random forest, CNN, etc. Recently, more focus has been on GAN to solve the analysis problem of super-resolution medical images.
The GAN architecture based on ResNet as a generator is the most advanced GAN framework, but the effect is not good for the retina Xu Xiang with a scale factor of more than 4, because the loss function does not explicitly include enough structural information. In order to solve this problem, This paper proposes a super-resolution algorithm based on GAN, and makes the following innovations to the loss function:
(1) Calculate the local saliency map according to the curvature map, and emphasize the local structure information
(2) Use entropy filtering to emphasize the compact area.
The combination of the two can better maintain the content information and sensory information in the original LR image, and then the method proposed in this paper was tested on the retinal super-resolution image.
Section II saliency map calculation
The existing saliency calculation methods all highlight the global saliency area. However, the weight of each pixel value needs to use the local saliency map. Inspired by paper [11], this paper will abstract (abstraction) and element distribution ( Element distribution) and uniqueness (uniqueness) are combined to calculate the local saliency map.
Abstraction:
fx, fy are the first derivative, fxx, fyy are the second derivative
Element Distribution:
used to capture the compactness and continuity of the structure, first calculate the pixel entropy Pixel(s):
Ns specifies the size of the neighborhood (7x7), and Pi is the probability density calculated based on the histogram.
The obtained Ient will be assigned a lower value for the non-compact area. After normalization to [0,1]->1-Ient, the highlighted compact area can be obtained.
**Uniqueness:**
Df calculates the sum of weight differences, and in order to avoid introducing blur, it calculates the absolute value of the difference between the i point and the i-th adjacent point si, and it is also normalized to [0,1] the final saliency map The calculation formula is:
Fig1 shows in turn: original image->gray image->1-Ient->curvature image Icurv->Isal
Section III GAN
ISR is responsible for predicting high-resolution images based on low-resolution images.
During training, Ilr is the low-resolution version corresponding to the high-resolution image Ihr, which is obtained by Gaussian filtering on the HR image.
The generation network G is used to generate super-resolution images. It can be seen from Fig2 that the residual module is introduced in G, and each block contains 2 layers of convolution-BN-ReLU; the discriminant network D is responsible for distinguishing the SR image from the real image, and the size of the convolution kernel in D ranges from 2- >64->512, Leaky ReLU is used for activation, and the final probability map is obtained through two fully connected layers and sigmoid.
Part A Loss Function
Content Loss:
Lsr includes content loss and counter loss. Alpha is a factor that weighs the proportion of the two. The MSE in content loss will cause the lack of high-frequency information in the SR image, which is to calculate the L2 distance between the SR image and the original image.
Therefore, in the MSEloss of this article, additional weighting is used to retain detailed information that is important for perception.
Loss Saliency Loss
What is measured is the difference between the saliency map and the HR image, so that important structural information is included in the loss function.
Adversarial Loss:
Generative loss calculated based on all training samples
Section IV Experiments and Results
Dataset: Acquired 5000 retinal images from various sources, which were enlarged by 10 times after rotation and affine transformation; in addition, the dark area was removed and resized to 1024x1024.
The optimizer Adam used by GAN, the G generation network is based on ResNet. When the scale factor r=2, the training time is 14hrs, r=4 26hrs, r=8 40hrs.
Part A super-resolution results
participate in the comparison: SRGAN-leding, SRGAN-sal (this article), SR-CNN, SSR, etc. Evaluation indicators are: PSNR, SSIM, S3, RMSE. The first three indicators are higher and higher, and the last indicator is as low as possible.
Table I shows the comparison results. It can be seen that the SRGAN-sal in this article has achieved the best results, and Fig3 is the visualization result, where (a) is the original HR image, (b) is the super-resolution image obtained by the text method , It can be seen that the fine blood vessel branches are clearly visible, and the effect is better than (c) and (d).
Part B Retinal Vessel Segmentation Results
The data sets used are: DRIVE, STARE, CHASE_DB1, which contain 40, 20, and 28 fundus images, respectively. The segmentation framework uses UNet, and the evaluation indicators include Acc and Sen.
It can be seen from Fig.4 that the segmentation results of SRGAN-sal and HR images are the closest, and the yellow arrow marks the part with imprecise segmentation. It can be seen that almost all images cannot perform fine segmentation of fine capillary structures, and SSR and SR-RF are not very accurate for segmentation of some major blood vessels.
Section V Conclusion
This paper proposes a super-resolution algorithm for fundus images based on GAN, which effectively improves the perceptual correlation of pixels with the help of local saliency maps, so that saliency loss can be calculated according to the weight of the pixels, and this loss is added to the training of GAN to effectively improve Improve the quality of the SR image. The experimental results also show that the combined effect of saliency loss and CNN loss is better than that of the existing GAN ISR algorithm. The final super-resolution image obtained can be further used for different analysis, such as when using SR image for retinal blood vessel analysis. The original image has achieved similar segmentation results, and the method in this article can also be applied to other medical images.