Image Restoration and Reconstruction
1. Introduction to the problem
Image is a very common information carrier, but the image may be affected by noise due to various reasons during the process of image acquisition, transmission, and storage.
How to remove the influence of noise and restore the original information of the image is an important research problem in computer vision.
The following artificially adds a certain amount of noise, and then denoises through different denoising algorithms to observe and compare the implementation effects of different algorithms.
It is recommended to run the code in the notebook, and it may cause errors to run directly in the exe
2. Define related functions
1. Guide package
from matplotlib import pyplot as plt # 展示图片
import numpy as np # 数值处理
import cv2 # opencv库
from sklearn.linear_model import LinearRegression, Ridge, Lasso # 回归分析
import random
from skimage.measure import compare_ssim as ssim
from scipy import spatial
from PIL import Image
2. Define the related function
read_image function; read the image
plot_image function: display the image
save_image function: save the image
normalization function: normalize the image data
get_noise_mask function: obtain the noise image
compute_error function: evaluate the difference between the restored image and the original image Error; the smaller the better
calc_ssim function: Calculate Cosine similarity; the larger the better
calc_csim function: Calculate SSIM similarity; the bigger the better
def read_image(img_path):
"""
读取图片，图片是以 np.array 类型存储
:param img_path: 图片的路径以及名称
:return: img np.array 类型存储
"""
# 读取图片
img = cv2.imread(img_path)
# 彩色图像使用 OpenCV 加载时是 BGR 模式，后面使用matplotlib显示图像时是采用RGB模式，所以这里需要将 BGR模式转换为 RGB 模式【cv2.cvtColor】
if len(img.shape) == 3:
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
return img
def plot_image(image, image_title, is_axis=False):
"""
展示图像
:param image: 展示的图像，一般是 np.array 类型
:param image_title: 展示图像的名称
:param is_axis: 是否需要关闭坐标轴，默认展示坐标轴
:return:
"""
# 展示图片
plt.imshow(image)
# 关闭坐标轴,默认关闭
if not is_axis:
plt.axis('off')
# 展示受损图片的名称
plt.title(image_title)
# 展示图片
plt.show()
def save_image(filename, image):
"""
将np.ndarray 图像矩阵保存为一张 png 或 jpg 等格式的图片
:param filename: 图片保存路径及图片名称和格式
:param image: 图像矩阵，一般为np.array
:return:
"""
# np.copy() 函数创建一个副本。
# 对副本数据进行修改，不会影响到原始数据，它们物理内存不在同一位置。
img = np.copy(image)
# 从给定数组的形状中删除一维的条目
img = img.squeeze()
# 将图片数据存储类型改为 np.uint8
if img.dtype == np.double:
# 若img数据存储类型是 np.double ,则转化为 np.uint8 形式
img = img * np.iinfo(np.uint8).max
# 转换图片数组数据类型
img = img.astype(np.uint8)
# 将 RGB 方式转换为 BGR 方式
img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
# 生成图片
cv2.imwrite(filename, img)
def normalization(image):
"""
将数据线性归一化
:param image: 图片矩阵，一般是np.array 类型
:return: 将归一化后的数据，在（0,1）之间
"""
# 获取图片数据类型对象的最大值和最小值
info = np.iinfo(image.dtype)
# 图像数组数据放缩在 0-1 之间
return image.astype(np.double) / info.max
def get_noise_mask(noise_img):
"""
获取噪声图像，一般为 np.array
:param noise_img: 带有噪声的图片
:return: 噪声图像矩阵
"""
# 将图片数据矩阵只包含 0和1,如果不能等于 0 则就是 1。
return np.array(noise_img != 0, dtype='double')
def compute_error(res_img, img):
"""
计算恢复图像 res_img 与原始图像 img 的 2-范数
:param res_img:恢复图像
:param img:原始图像
:return: 恢复图像 res_img 与原始图像 img 的2-范数
"""
# 初始化
error = 0.0
# 将图像矩阵转换成为np.narray
res_img = np.array(res_img)
img = np.array(img)
# 如果2个图像的形状不一致，则打印出错误结果，返回值为 None
if res_img.shape != img.shape:
print("shape error res_img.shape and img.shape %s != %s" % (res_img.shape, img.shape))
return None
# 计算图像矩阵之间的评估误差
error = np.sqrt(np.sum(np.power(res_img - img, 2)))
return round(error,3)
def calc_ssim(img, img_noise):
"""
计算图片的结构相似度
:param img: 原始图片， 数据类型为 ndarray, shape 为[长, 宽, 3]
:param img_noise: 噪声图片或恢复后的图片，
数据类型为 ndarray, shape 为[长, 宽, 3]
:return:
"""
return ssim(img, img_noise,
multichannel=True,
data_range=img_noise.max() - img_noise.min())
def calc_csim(img, img_noise):
"""
计算图片的 cos 相似度
:param img: 原始图片， 数据类型为 ndarray, shape 为[长, 宽, 3]
:param img_noise: 噪声图片或恢复后的图片，
数据类型为 ndarray, shape 为[长, 宽, 3]
:return:
"""
img = img.reshape(-1)
img_noise = img_noise.reshape(-1)
return 1 - spatial.distance.cosine(img, img_noise)
3. Add noise
Idea: first generate a noise mask mask, and multiply it with the matrix data of the original image to get the damaged image.
Since the dimension of the original image data is (h, w, channel), it can be converted to (channel, h, w) first. h and w respectively represent the height and width of the image, and channel refers to the channel dimension. You can use the np.trsnspose function.
After conversion, the matrix [channel, h, w] of img1 can be understood as a matrix with 3 [h, w], corresponding to the RGB3 channels in turn.
def noise_mask_image(img, noise_ratio=[0.8,0.4,0.6]):
"""
根据题目要求生成受损图片
:param img: cv2 读取图片,而且通道数顺序为 RGB
:param noise_ratio: 噪声比率，类型是 List,，内容:[r 上的噪声比率,g 上的噪声比率,b 上的噪声比率]
默认值分别是 [0.8,0.4,0.6]
:return: noise_img 受损图片, 图像矩阵值 0-1 之间，数据类型为 np.array,
数据类型对象 (dtype): np.double, 图像形状:(height,width,channel),通道(channel) 顺序为RGB
"""
# 受损图片初始化
noise_img = None
# -------------实现受损图像答题区域-----------------
#mask为噪声遮罩；noise_img为受损图片
img1=np.transpose(img.copy(),(2,0,1))
mask=np.transpose(img,(2,0,1)) #对img的矩阵作变换，将channel通道放在第一个维度
for i in range(mask.shape[0]):
for j in range(mask.shape[1]):
rate=int(noise_ratio[i]*mask.shape[2])
mask[i][j][:rate]=0
mask[i][j][rate:]=1
np.random.shuffle(mask[i][j])
noise_img=img1*mask
noise_img=np.transpose(noise_img,(1,2,0))
return noise_img
The generated noise-added image:
3. Filter processing method
1. Mean filtering
(1) If all pixels are processed together, the effect is not good:
def restore_image(noise_img, size=4):
"""
使用 你最擅长的算法模型 进行图像恢复。
:param noise_img: 一个受损的图像
:param size: 输入区域半径，长宽是以 size*size 方形区域获取区域, 默认是 4
:return: res_img 恢复后的图片，图像矩阵值 0-1 之间，数据类型为 np.array,
数据类型对象 (dtype): np.double, 图像形状:(height,width,channel), 通道(channel) 顺序为RGB
"""
# 恢复图片初始化，首先 copy 受损图片，然后预测噪声点的坐标后作为返回值。
res_img = np.copy(noise_img)
# 获取噪声图像
noise_mask = get_noise_mask(noise_img)
# -------------实现图像恢复代码答题区域----------------------------
noise_img = np.transpose(noise_img, (2, 0, 1))
size = 3 # 核的尺寸
num = int((size - 1) / 2) # 输入图像需要填充的尺寸
list = []
list1 = []
for x in range(3):
img = cv2.copyMakeBorder(noise_img[x], num, num, num, num, cv2.BORDER_REPLICATE)
h, w = img.shape[0:2] # 获取输入图像的长宽和高
for i in range(num, h - num): # 遍历出原图像
for j in range(num, w - num):
list.extend(
[img[i, j], img[i - 1, j - 1], img[i - 1, j], img[i - 1, j + 1], img[i, j - 1], img[i, j + 1],
img[i + 1, j], img[i + 1, j - 1], img[i + 1, j + 1]])
# list.sort() #中值滤波
list[4] = sum(list) / 9 # 均值滤波
list1.append(list[4])
list = []
res_img = np.array(list1).reshape(noise_img.shape)
res_img = np.transpose(res_img, (1, 2, 0))
# ---------------------------------------------------------------
return res_img
final effect:
(2) Process only valid pixels:
def restore_image(noise_img, size=4):
"""
使用 你最擅长的算法模型 进行图像恢复。
:param noise_img: 一个受损的图像
:param size: 输入区域半径，长宽是以 size*size 方形区域获取区域, 默认是 4
:return: res_img 恢复后的图片，图像矩阵值 0-1 之间，数据类型为 np.array,
数据类型对象 (dtype): np.double, 图像形状:(height,width,channel), 通道(channel) 顺序为RGB
"""
# 恢复图片初始化，首先 copy 受损图片，然后预测噪声点的坐标后作为返回值。
res_img = np.copy(noise_img)
# 获取噪声图像
noise_mask = get_noise_mask(noise_img)
# -------------实现图像恢复代码答题区域----------------------------
#只针对有效像素点进行均值处理
init_img = np.transpose(res_img, (2, 0, 1))
init_mask= np.transpose(noise_mask, (2, 0, 1))
res_img = np.copy(init_img)
gao, kuang = res_img.shape[1], res_img.shape[2]
size = 2 # 核的尺寸；可以自定义大小，size越大，恢复效果越模糊
valid_pixel = 0 #定义表示有效像素点变量【valid_pixel】
for channel in range(3):
for i in range(gao):
for j in range(kuang):
valid_pixel=init_mask[channel][i:i + size, j:j + size].sum() #获取每个size区域内有效像素（不为0的像素）的个数
if valid_pixel != 0:
res_img[channel][i][j] = (init_img[channel][i:i + size, j:j + size] * init_mask[channel][i:i + size,j:j + size]).sum() / valid_pixel
valid_pixel = 0
else:
size+=1 #如果size区域内有效像素点个数为0，则扩大处理区域
for channel in range(3):
for i in range(gao):
for j in range(kuang):
valid_pixel = init_mask[channel][i:i + size, j:j + size].sum()
if valid_pixel != 0:
res_img[channel][i][j] = (init_img[channel][i:i + size, j:j + size] * init_mask[
channel][
i:i + size,
j:j + size]).sum() / valid_pixel
res_img=np.transpose(res_img,(1,2,0))
# ---------------------------------------------------------------
return res_img
The final effect:
2. Median filtering
The idea is the same as the above average value, except that the median value of all effective pixel values in the filter window is taken.
Since I want to use the np.median method, all the invalid pixel values are converted into a list and then deleted, and this method is used in the conversion into a matrix.
def restore_image(noise_img, size=4):
"""
使用 你最擅长的算法模型 进行图像恢复。
:param noise_img: 一个受损的图像
:param size: 输入区域半径，长宽是以 size*size 方形区域获取区域, 默认是 4
:return: res_img 恢复后的图片，图像矩阵值 0-1 之间，数据类型为 np.array,
数据类型对象 (dtype): np.double, 图像形状:(height,width,channel), 通道(channel) 顺序为RGB
"""
# 恢复图片初始化，首先 copy 受损图片，然后预测噪声点的坐标后作为返回值。
res_img = np.copy(noise_img)
# 获取噪声图像
noise_mask = get_noise_mask(noise_img)
# -------------实现图像恢复代码答题区域----------------------------
#只针对有效像素点进行中值滤波
init_img = np.transpose(res_img, (2, 0, 1))
init_mask= np.transpose(noise_mask, (2, 0, 1))
res_img = np.copy(init_img)
gao, kuang = res_img.shape[1], res_img.shape[2] #获取图像的高和宽
size = 2 # 核的尺寸
valid_pixel = 0 #定义表示有效像素点变量【valid_pixel】
for channel in range(3):
for i in range(gao):
for j in range(kuang):
valid_pixel = init_mask[channel][i:i + size, j:j + size].sum() # 计算有效像素点个数
if valid_pixel != 0:
r1 = (init_img[channel][i:i + size, j:j + size] * init_mask[channel][i:i + size,
j:j + size]).flatten() #将矩阵展平成一维数组
r2 = np.matrix.tolist(r1) # r1转换为列表
r3 = list(set(r2)) # 去掉r2中的'0'元素，但是转换成集合后还是会留下一个'0'
if '0' in r3:
r3.remove(0) # 删除唯一的0元素
r4 = np.median(np.array(r3)) # r3再转换为矩阵并求中值
res_img[channel][i][j] = r4
else:
size+=1 #如果size区域内有效像素点个数为0，则扩大处理区域
for channel in range(3):
for i in range(gao):
for j in range(kuang):
valid_pixel = init_mask[channel][i:i + size, j:j + size].sum()
if valid_pixel != 0:
r1 = (init_img[channel][i:i + size, j:j + size] * init_mask[channel][i:i + size,
j:j + size]).flatten()
r2 = np.matrix.tolist(r1)
r3 = list(set(r2))
if '0' in r3:
r3.remove(0)
r4 = np.median(np.array(r3))
res_img[channel][i][j] = r4
res_img=np.transpose(res_img,(1,2,0))
# ---------------------------------------------------------------
return res_img
final effect:
Four. Summary
In the end, I found that the effect of using the median filter here is not as good as that of the mean (it may also be a problem with the code I wrote myself). One of the current doubts is that the larger the error, the better the effect of the picture looks?
If you have a chance later, try other filtering methods. If you find it useful, please pay attention!