遗留问题:2019-9-6
1.retinanet_bbox()中__build_anchors出来的anchor和模型的预测值regression = model.outputs[0]之间,基本是box和偏移量的关系,为什么regression不直预测目标的真实位置呢
2.不同尺寸的照片,目标的大小就不一样,网络是如何做到对不同尺寸的适应呢?
keras实现自定义层
前言:keras提供众多常见的已编写好的层对象,例如常见的卷积层,池化层等,我们可以直接通过代码调用,keras中的层大致上分为两种类型:
第一种:带有训练参数,比如dense, conv2d层等,我们在训练的过程中需要训练层的权重和偏置项;
第二种: 不带训练参数, 比如dropout, flatten层等, 我们不需要训练它的权重,只需要对输入进行加工处理再输出就行了.
在实际应用中,我们经常需要自己构建一些层对象,以满足某些自定义网络的特殊需求,也无非就是上面两种,一种是带有参数的,一种是不带参数的,不管是哪一种,幸运的是,keras对自定义层都提供了良好的支持.
下面我们着重来看下自定义层
1.dense层解析
2.基类layer中的定义
2. 三个核心方法的解析
2.1 build方法(我看到retinanet中的自定义层都没有写这个build,是不是可以省略啊)
2.2 call方法regression
2.3 compute_output_shape方法
要定制自己的层,需要实现三个方法:
build(input_shape),这是定义权重的方法,可循练得权重应该在这里被加入列表self.trainable_weights中,其他的属性还包括列表self.non_trainable_weights和self.updates(需要更新的形如(tensor,new_tensor)的tuple的列表),这个方法必须设置self.built=True,可以通过super([layer],self).build()实现
call(x): 这是定义层功能的地方,除非你希望你写的层支持masking(这个是啥东东?),否则你只需要关心call的第一个参数:输入张量.
compute_output_shape(input_shape):如果你的层修改了输入数据的shape,你应该在这里指定shape变化的方法,这个函数使得keras可以做自动shape推断,(若我对输入数据的shape没有做改变,是不是这个函数就不用写了?)
我们看一下这个自定义层在retinanet中的一个应用
box回归模型为:
def retinanet_bbox(
model = None,
nms = True,
class_specific_filter = True,
name = 'retinanet-bbox',
anchor_params = None,
**kwargs
):
""" Construct a RetinaNet model on top of a backbone and adds convenience functions to output boxes directly.
This model uses the minimum retinanet model and appends a few layers to compute boxes within the graph.
These layers include applying the regression values to the anchors and performing NMS.
Args
model : RetinaNet model to append bbox layers to. If None, it will create a RetinaNet model using **kwargs.
nms : Whether to use non-maximum suppression for the filtering step.
class_specific_filter : Whether to use class specific filtering or filter for the best scoring class only.
name : Name of the model.
anchor_params : Struct containing anchor parameters. If None, default values are used.
*kwargs : Additional kwargs to pass to the minimal retinanet model.
Returns
A keras.models.Model which takes an image as input and outputs the detections on the image.
The order is defined as follows:
```
[
boxes, scores, labels, other[0], other[1], ...
]
```
"""
# if no anchor parameters are passed, use default values
if anchor_params is None:
anchor_params = AnchorParameters.default
# create RetinaNet model
if model is None:
model = retinanet(num_anchors=anchor_params.num_anchors(), **kwargs)
else:
assert_training_model(model)
# compute the anchors
features = [model.get_layer(p_name).output for p_name in ['P3', 'P4', 'P5', 'P6', 'P7']]
anchors = __build_anchors(anchor_params, features)#根据FPN和预置的anchor参数,由自定义层,生成anchor,这里的维度是[batchsize,num_anchors,4]
# we expect the anchors, regression and classification values as first output
regression = model.outputs[0]#骨干网络的预测输出, 这里是预测的box,问题,这个骨干网络是?
classification = model.outputs[1]#骨干网络的预测输出,这里预测的是分类信息
# "other" can be any additional output from custom submodels, by default this will be []
other = model.outputs[2:]
# apply predicted regression to anchors
boxes = layers.RegressBoxes(name='boxes')([anchors, regression]) #自定义层,将网络的预测输出box与根据fpn,用自定义层得出的anchors作为参数输入这一层,
boxes = layers.ClipBoxes(name='clipped_boxes')([model.inputs[0], boxes]) #自定义层
# filter detections (apply NMS / score threshold / select top-k)
detections = layers.FilterDetections(
nms = nms,
class_specific_filter = class_specific_filter,
name = 'filtered_detections'
)([boxes, classification] + other)
# construct the model
return keras.models.Model(inputs=model.inputs, outputs=detections, name=name)
其中用到了自定义层RegressBoxes, ClipBoxes,FilterDetections,下面我们来看下自定义层RegressBoxes
class RegressBoxes(keras.layers.Layer):
""" Keras layer for applying regression values to boxes.
"""
def __init__(self, mean=None, std=None, *args, **kwargs):
""" Initializer for the RegressBoxes layer.
Args
mean: The mean value of the regression values which was used for normalization.
std: The standard value of the regression values which was used for normalization.
"""
if mean is None:
mean = np.array([0, 0, 0, 0])
if std is None:
std = np.array([0.2, 0.2, 0.2, 0.2])
if isinstance(mean, (list, tuple)):
mean = np.array(mean)
elif not isinstance(mean, np.ndarray):
raise ValueError('Expected mean to be a np.ndarray, list or tuple. Received: {}'.format(type(mean)))
if isinstance(std, (list, tuple)):
std = np.array(std)
elif not isinstance(std, np.ndarray):
raise ValueError('Expected std to be a np.ndarray, list or tuple. Received: {}'.format(type(std)))
self.mean = mean
self.std = std
super(RegressBoxes, self).__init__(*args, **kwargs)
#call是这个自定义层的功能实现的地方,它接受的参数是inputs,返回output,这是搭建model的关键所在,在这个函数里面实现这个自定义层的运算
def call(self, inputs, **kwargs):
anchors, regression = inputs
return backend.bbox_transform_inv(anchors, regression, mean=self.mean, std=self.std)
def compute_output_shape(self, input_shape):
return input_shape[0]
def get_config(self):
config = super(RegressBoxes, self).get_config()
config.update({
'mean': self.mean.tolist(),
'std' : self.std.tolist(),
})
return config可以看出,除了必须方法call和compute_output_shape()之外,这个自定义层还包含了其他方法.我们来看下这个自定义层的功能实现方法call(),可以看出,在前面的retinanet.py-->retinanet_bbox()函数调用这个自定义层时,入参为[anchors, regression],程序运行时,这两个参数被传进了自定义层的call()方法,所以在call方法中,会解析这个输入
anchors, regression = inputs后面再调用函数中backend.bbox_transform_inv()对入参[anchors, regression]进行进一步处理,基本上就是更新anchors,更新原则是,假设box为anchors中的一个元素,则更新之后,box_new=box+0.2* regression*box_width(或高),就是根据回归结果,把box的位置更新一下,
def bbox_transform_inv(boxes, deltas, mean=None, std=None):
""" Applies deltas (usually regression results) to boxes (usually anchors).
Before applying the deltas to the boxes, the normalization that was previously applied (in the generator) has to be removed.
The mean and std are the mean and std as applied in the generator. They are unnormalized in this function and then applied to the boxes.
Args
boxes : np.array of shape (B, N, 4), where B is the batch size, N the number of boxes and 4 values for (x1, y1, x2, y2).
deltas: np.array of same shape as boxes. These deltas (d_x1, d_y1, d_x2, d_y2) are a factor of the width/height.
mean : The mean value used when computing deltas (defaults to [0, 0, 0, 0]).
std : The standard deviation used when computing deltas (defaults to [0.2, 0.2, 0.2, 0.2]).
Returns
A np.array of the same shape as boxes, but with deltas applied to each box.
The mean and std are used during training to normalize the regression values (networks love normalization).
"""
if mean is None:
mean = [0, 0, 0, 0]
if std is None:
std = [0.2, 0.2, 0.2, 0.2]
#mean [0 0 0 0].std [0.2 0.2 0.2 0.2], 问题,说这个均值和偏差是在generator中确定出来的,怎么没见到?
width = boxes[:, :, 2] - boxes[:, :, 0] #anchors中都是一个个的box,算出这些box的宽,
height = boxes[:, :, 3] - boxes[:, :, 1]#算出这些box的高
x1 = boxes[:, :, 0] + (deltas[:, :, 0] * std[0] + mean[0]) * width #这里是在原来的anchor的基础上,对box进行位置精调,
y1 = boxes[:, :, 1] + (deltas[:, :, 1] * std[1] + mean[1]) * height#x1_new=x1+0.2*delta*width
x2 = boxes[:, :, 2] + (deltas[:, :, 2] * std[2] + mean[2]) * width
y2 = boxes[:, :, 3] + (deltas[:, :, 3] * std[3] + mean[3]) * height
pred_boxes = keras.backend.stack([x1, y1, x2, y2], axis=2)
return pred_boxes从上述代码,基本可以看出,网络的回归值regression其实是回归的每个坐标(x1,y1,x2,y2)的偏移量.
下面再来看另一个自定义层layers.ClipBoxes,其实是对网络的预测的box做进一步处理,比如一张图片的尺寸(416,510),但网络预测了一个box=[-10,-9,500,600],可以看出这个预测box超出了图片的边界,当然与实际情况不符(实际中,所有的检测目标都应该在画面内啊),所以将这个box调整下,调整到画面内,于是这个box就被修正为(0,0,416,510),也就是下面的tf.clip_by_value函数所起的作用.
class ClipBoxes(keras.layers.Layer):
""" Keras layer to clip box values to lie inside a given shape.
"""
def call(self, inputs, **kwargs):
image, boxes = inputs #这里的输入image为原始图像,boxes为网络预测的box的位置,
shape = keras.backend.cast(keras.backend.shape(image), keras.backend.floatx())
if keras.backend.image_data_format() == 'channels_first':
height = shape[2]
width = shape[3]
else:
height = shape[1]
width = shape[2]
x1 = backend.clip_by_value(boxes[:, :, 0], 0, width)#clip_by_value输入一个张量A，把A中的每一个元素的值都压缩在min和max之间。小于min的让它等于min，大于max的元素的值等于max。
y1 = backend.clip_by_value(boxes[:, :, 1], 0, height)#网络预测的box的位置有可能会超出图像的边界,所以这里把预测的box的值都界定到图像内
x2 = backend.clip_by_value(boxes[:, :, 2], 0, width)
y2 = backend.clip_by_value(boxes[:, :, 3], 0, height)
return keras.backend.stack([x1, y1, x2, y2], axis=2)
def compute_output_shape(self, input_shape):
return input_shape[1]备注:从上面的两个自定义层功能就可以看出,就是对网络的预测box做进一步处理,没有训练参数的存在哦.
当我们导入的模型含有自定义层layer或者自定义函数loss时,需要在load_model()中使用,函数原型为
def load_model(filepath, custom_objects=None, compile=True):
"""Loads a model saved via `save_model`.
# Arguments
filepath: one of the following:
- string, path to the saved model, or
- h5py.File or h5py.Group object from which to load the model
custom_objects: Optional dictionary mapping names
(strings) to custom classes or functions to be
considered during deserialization.
compile: Boolean, whether to compile the model
after loading.
# Returns
A Keras model instance. If an optimizer was found
as part of the saved model, the model is already
compiled. Otherwise, the model is uncompiled and
a warning will be displayed. When `compile` is set
to False, the compilation is omitted without any
warning.
# Raises
ImportError: if h5py is not available.
ValueError: In case of an invalid savefile.
"""函数功能:加载在训练过程中以save_model形式保存下来的模型,一般为h5模型
参数:file_path, h5模型路径
custom_objects: 以字典形式来指定目标层layer或目标函数loss.
compile: 布尔值, 确定在加载模型后是否编译,默认=true
例如,我的一个模型含有自定义层"SincConv1D",和自定义函数"my_loss",需要使用下面的代码导入
from keras.models import load_model
model = load_model('model.h5', custom_objects={'my_loss':my_loss,'SincConv1D': SincConv1D})如果不加custom_objects指定目标层layer,则会出现以下报错
ValueError: Unknown layer: SincConv1D