Resnet笔记

                   Resnet残差网络

Introduction
(1)Address the degradation problem for deep conv。
(2)Shortcut connection: add neither extra params nor computational complexity

(3)The dimensions of x and F are equal。
(4)F has two or three layers, while more layers are possible. If F only has one layers , is similar to a liner layer y=wx+x, has no advantages.
(5)用GAP代替fc层：减少参数量，对整个网络在结构上正则化，减少过拟合，可能会造成收敛速度慢
(6)Conv S=2 instead of pooling。 when Feature map size is halves, the number of filters is doubled, because of perserve the time complexity per layers. F和x尺寸不相等时，zero padding and 1x1conv。 Adopt bn after each conv and before activation

(7)Bottleneck Architectures

(8)梯度消失
不同layers的学习速率差异很大，参数更新时，靠近output的layer学习较好，靠近input的layer学习较差。如果出现梯度消失，那么靠近input的layer参数可能就几乎不再更新。
对于resnet为什么能够防止梯度消失问题，个人理解是：由于存在shortcut结构相当于（1+x），即使你梯度消失x=0了，但是我还有1，来进行参数的更新。