神经网络学习（十二）卷积神经网络与BP算法

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/hoho1151191150/article/details/79705332

系列博客是博主学习神经网络中相关的笔记和一些个人理解，仅为作者记录笔记之用，不免有很多细节不对之处。

DNN基本计算流程

1、 首先计算输出层的 $\delta^L$:

$${\delta ^L} = \frac{{\partial C}}{{\partial {a^L}}} \odot \sigma '\left( {{z^L}} \right)\tag{BP1}$$ 注：若输出函数采用 softmax 函数 $a_j^L = {{{e^{z_j^L}}}}/{{\sum\nolimits_{\rm{k}} {{e^{z_k^L}}} }}$，则对应的交叉熵函数选择 $C = - \sum\nolimits_{\rm{k}} {{y_k}\ln \left( {a_k^L} \right)}$

若输出函数采用 sigmoid 函数 $a_j^L = {1}/{{( 1 + {e^{-z_j^L}}) }}$，则对应的交叉熵函数选择 $C = - \sum\nolimits_{\rm{k}} {\left[ {{y_k}\ln \left( {a_k^L} \right) + \left( {1 - {y_k}} \right)\ln \left( {1 - a_k^L} \right)} \right]}$

2、 利用链式法则，由 $\delta^{l+1}$ 可一步一步推出 $\delta^{l}$ ，其表达式为

$${\delta ^l} = {\left( {{W^{l + 1}}} \right)^T}{\delta ^{l + 1}} \odot \sigma '\left( {{z^l}} \right)\tag{BP2}$$ 注：解决梯度弥散问题：ReLU激活函数， $\sigma \left( z \right) = \max \left( {0,z} \right)$，其导数为 $\sigma \left( z \right) = \max \left( {0,z} \right) /z$

3、 由 $\delta^l$ 计算出 $W$ 和 $b$ 的梯度表达式

$$\frac{{\partial C}}{{\partial W}} = {\delta ^l}{\left( {{a^{l - 1}}} \right)^T}\tag{BP3}$$ $$\frac{{\partial C}}{{\partial b}} = {\delta ^l}\tag{BP4}$$

4、 模型更新:

$$W \leftarrow W - \eta \frac{{\partial C}}{{\partial W}}$$ $$b \leftarrow b - \eta \frac{{\partial C}}{{\partial b}}$$ 注：M小批量随机梯度下降法+L2规则化缓解过拟合 $$W \leftarrow \left( {1 - \frac{{\eta \lambda }}{n}} \right)W - \frac{\eta }{m}\sum\nolimits_x {\frac{{\partial {C_x}}}{{\partial W}}}, b \leftarrow b - \frac{\eta }{m}\sum\nolimits_x {\frac{{\partial {C_x}}}{{\partial b}}}$$

卷积神经网络的反向传播算法

1、全连接到池化层的反向传播

池化层是没有激活函数的，但是我们可以令池化层的激活函数$\sigma(z)=z$, 那么它的导数为 1。现假设 $\delta^{l+1}$ 为全连接层的误差敏感项， $\delta^l$ 是池化层的误差敏感项，由公式（BP2）有

$${\delta ^l} = {\left( {{W^{l + 1}}} \right)^T}{\delta ^{l + 1}}$$

2、池化层到卷积层的反向传播

池化层对输入数据进行了一定的压缩处理（这里假设每个小区域不重复，即输入矩阵 $N\times N$, 池化矩阵大小 $k\times k$，则输出矩阵为$(N/k) \times (N/k)$），池化的两种常见方式如下图所示

Mean-pooling将每个小区域内的平均值作为池化结果；Max-pooling将每个小区域内的最大值作为池化结果

在误差反向传播过程中要对数据做逆向降采样。现假设 $\delta^{l+1}$ 为池化层的误差敏感项，$\delta^{l}$ 是卷积层的误差敏感项，那么有

$${\delta ^l} = {\rm{upsample}}\left( {{\delta ^{l + 1}}} \right) \odot \sigma '\left( {{z^l}} \right)$$ 对于mean-pooling， upsample 函数将池化层的结果平均分配至原有小区域内；对于max-pooling, upsample 函数将池化层的结果放置于原最大值的位置，其余位置为0 (在池化的正向计算过程中需记录每个小区域内最大值的位置)。下面是一个upsample的例子：

3、卷积层到上一层的反向传播

卷积层向上一层的误差反传方式与BP神经网络是不一样的，但是其思路仍旧是一样的。咱们先从正向传播开始，逐步推导出其反向传播算法（为简单起见，卷积核的移动步幅stride设置为1）。

假设 $a^l$ 是第 $l$ 层的输出，$z^{l+1}$ 是第 $l+1$ 层的输入，则卷积层的输入为，有

$$\left( {\begin{array}{*{20}{c}} {a_{11}^l}&{a_{12}^l}&{a_{13}^l}\\ {a_{21}^l}&{a_{22}^l}&{a_{23}^l}\\ {a_{31}^l}&{a_{32}^l}&{a_{33}^l} \end{array}} \right) * \left( {\begin{array}{*{20}{c}} {w_{11}^{l + 1}}&{w_{12}^{l + 1}}\\ {w_{21}^{l + 1}}&{w_{22}^{l + 1}} \end{array}} \right) = \left( {\begin{array}{*{20}{c}} {z_{11}^{l + 1}}&{z_{12}^{l + 1}}\\ {z_{21}^{l + 1}}&{z_{22}^{l + 1}} \end{array}} \right)$$ 利用卷积的定义（ $W$先进行左右和上下翻转，再相乘求和），很容易得出： $$\begin{array}{l} z_{11}^{l + 1} = a_{11}^lw_{22}^{l + 1} + a_{12}^lw_{21}^{l + 1} + a_{21}^lw_{12}^{l + 1} + a_{22}^lw_{11}^{l + 1}\\ z_{12}^{l + 1} = a_{12}^lw_{22}^{l + 1} + a_{13}^lw_{21}^{l + 1} + a_{22}^lw_{12}^{l + 1} + a_{23}^lw_{11}^{l + 1}\\ z_{21}^{l + 1} = a_{21}^lw_{22}^{l + 1} + a_{22}^lw_{21}^{l + 1} + a_{31}^lw_{12}^{l + 1} + a_{32}^lw_{11}^{l + 1}\\ z_{22}^{l + 1} = a_{22}^lw_{22}^{l + 1} + a_{23}^lw_{21}^{l + 1} + a_{32}^lw_{12}^{l + 1} + a_{33}^lw_{11}^{l + 1} \end{array}$$ 上式可由MATLAB中卷积函数的’valid’模式计算得出： ${z^{l + 1}} = {\rm{conv2}}\left( {{a^l},{W^{l + 1}},'valid'} \right)$。为了方便推导，我们希望借助MATLAB的符号计算功能，但是 conv （conv2,convn）函数并不适用于符号计算。咱们自己先写一个可以进行符号计算的卷积函数 symconv

function z = symconv(a,k)
%SYMCONV valid模式下的syms符号卷积 
syms zero real
k = rot90(k,2);
[hw,ww] = size(k);
[ha,wa] = size(a);
h = ha - hw + 1;
w = wa - ww + 1;
for in = 1:h
    for im = 1:w
        z(in,im) = zero;
        for jn = 1:hw
            for jm = 1:ww
                z(in,im) = z(in,im)+a(in+jn-1,im+jm-1)*k(jn,jm);
            end
        end
    end
end
z = z - zero;
end

下面是计算 $z$ 的脚本：

syms a11 a12 a13 a21 a22 a23 a31 a32 a33 real % a in L layer
syms w11 w12 w21 w22 real
syms z11 z12 z21 z22 real   % z in L+1 layer
syms d11 d12 d21 d22 real   % delta in L+1 layer
a = [a11 a12 a13; a21 a22 a23; a31 a32 a33];
d = [d11 d12; d21 d22];
w = [w11 w12; w21 w22];
z = symconv(a,w);

这样，我们就可以随意对 $a_{ij}$ 或者 $w_{ij}$ 进行求导了（本节大部分推导利用了这些代码），如下

diff(z,a11); %对a11求导
diff(z,w11); %对w11求导

接着卷积层的误差敏感项 $\delta^l$，由链式法则

$$\delta _{ij}^l = \frac{{\partial C}}{{\partial z_{ij}^l}} = \left( {\sum\nolimits_{mn} {\frac{{\partial C}}{{\partial z_{mn}^{l + 1}}}\frac{{\partial z_{mn}^{l + 1}}}{{\partial a_{ij}^l}}} } \right)\frac{{\partial a_{ij}^l}}{{\partial z_{ij}^l}}$$ 对于这个例子，可以把上式改写为 $$\delta _{ij}^l = \frac{{\partial C}}{{\partial z_{ij}^l}} = \left( {{\delta ^{l + 1}} \odot \frac{{\partial {z^{l + 1}}}}{{\partial a_{ij}^l}}} \right) * \left[ {\begin{array}{*{20}{c}} 1&1\\ 1&1 \end{array}} \right]\sigma '\left( {z_{ij}^l} \right)$$ 那么 $$\delta _{11}^l = \left( {\begin{array}{*{20}{c}} {\delta _{11}^{l + 1}w_{22}^{l + 1}}&0\\ 0&0 \end{array}} \right) * \left[ {\begin{array}{*{20}{c}} 1&1\\ 1&1 \end{array}} \right]\sigma '\left( {z_{11}^l} \right) = \left( {\begin{array}{*{20}{c}} 0&0\\ 0&{\delta _{11}^{l + 1}} \end{array}} \right) * \left( {\begin{array}{*{20}{c}} {w_{22}^{l + 1}}&0\\ 0&0 \end{array}} \right)\sigma '\left( {z_{11}^l} \right)$$ $$\delta _{12}^l= \left( {\begin{array}{*{20}{c}} {\delta _{11}^{l + 1}w_{21}^{l + 1}}&{\delta _{12}^{l + 1}w_{22}^{l + 1}}\\ 0&0 \end{array}} \right) * \left[ {\begin{array}{*{20}{c}} 1&1\\ 1&1 \end{array}} \right]\sigma '\left( {z_{12}^l} \right) = \left( {\begin{array}{*{20}{c}} 0&0\\ {\delta _{11}^{l + 1}}&{\delta _{12}^{l + 1}} \end{array}} \right) * \left( {\begin{array}{*{20}{c}} {w_{22}^{l + 1}}&{w_{21}^{l + 1}}\\ 0&0 \end{array}} \right)\sigma '\left( {z_{12}^l} \right)$$ $$\delta _{13}^l = \left( {\begin{array}{*{20}{c}} 0&{\delta _{12}^{l + 1}w_{21}^{l + 1}}\\ 0&0 \end{array}} \right) * \left[ {\begin{array}{*{20}{c}} 1&1\\ 1&1 \end{array}} \right]\sigma '\left( {z_{13}^l} \right) = \left( {\begin{array}{*{20}{c}} 0&0\\ {\delta _{12}^{l + 1}}&0 \end{array}} \right) * \left( {\begin{array}{*{20}{c}} 0&{w_{21}^{l + 1}}\\ 0&0 \end{array}} \right)\sigma '\left( {z_{13}^l} \right)$$ $$\delta _{22}^l = \left( {\begin{array}{*{20}{c}} {\delta _{11}^{l + 1}w_{11}^{l + 1}}&{\delta _{12}^{l + 1}w_{12}^{l + 1}}\\ {\delta _{21}^{l + 1}w_{21}^{l + 1}}&{\delta _{22}^{l + 1}w_{22}^{l + 1}} \end{array}} \right) * \left[ {\begin{array}{*{20}{c}} 1&1\\ 1&1 \end{array}} \right]\sigma '\left( {z_{13}^l} \right) = \left( {\begin{array}{*{20}{c}} {\delta _{11}^{l + 1}}&{\delta _{12}^{l + 1}}\\ {\delta _{12}^{l + 1}}&{\delta _{22}^{l + 1}} \end{array}} \right) * \left( {\begin{array}{*{20}{c}} {w_{22}^{l + 1}}&{w_{21}^{l + 1}}\\ {w_{12}^{l + 1}}&{w_{11}^{l + 1}} \end{array}} \right)\sigma '\left( {z_{22}^l} \right)$$ 其他几个不再列举了，仔细观察，可以发现 $\delta_{11}^l$中的 $% MathType!MTEF!2!1!+- % feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn % hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqr1ngB % PrgifHhDYfgatCvAUfeBSn0BKvguHDwzZbqegSSZmxoasaacH8srps % 0lbbf9q8WrFfeuY-Hhbbf9v8qqaqFr0xc9pk0xbba9q8WqFfea0-yr % 0RYxir-Jbba9q8aq0-yq-He9q8qqQ8frFve9Fve9Ff0dmeaabaqaci % GacaGaaeqabaWaaeaaeaqbaOqaamaabmaabaqbaeqabiGaaaqaaiaa % dEhadaqhaaWcbaGaaGOmaiaaikdaaeaacaWGSbGaey4kaSIaaGymaa % aaaOqaaiaaicdaaeaacaaIWaaabaGaaGimaaaaaiaawIcacaGLPaaa % aaa!49D6! \left( {\begin{array}{*{20}{c}} {w_{22}^{l + 1}}&0\\ 0&0 \end{array}} \right)$可以写为 $% MathType!MTEF!2!1!+- % feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn % hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqr1ngB % PrgifHhDYfgatCvAUfeBSn0BKvguHDwzZbqegSSZmxoasaacH8srps % 0lbbf9q8WrFfeuY-Hhbbf9v8qqaqFr0xc9pk0xbba9q8WqFfea0-yr % 0RYxir-Jbba9q8aq0-yq-He9q8qqQ8frFve9Fve9Ff0dmeaabaqaci % GacaGaaeqabaWaaeaaeaqbaOqaamaabmaabaqbaeqabiGaaaqaaiaa % dEhadaqhaaWcbaGaaGOmaiaaikdaaeaacaWGSbGaey4kaSIaaGymaa % aaaOqaaiaadEhadaqhaaWcbaGaaGOmaiaaigdaaeaacaWGSbGaey4k % aSIaaGymaaaaaOqaaiaadEhadaqhaaWcbaGaaGymaiaaikdaaeaaca % WGSbGaey4kaSIaaGymaaaaaOqaaiaadEhadaqhaaWcbaGaaGymaiaa % igdaaeaacaWGSbGaey4kaSIaaGymaaaaaaaakiaawIcacaGLPaaaaa % a!574F! \left( {\begin{array}{*{20}{c}} {w_{22}^{l + 1}}&{w_{21}^{l + 1}}\\ {w_{12}^{l + 1}}&{w_{11}^{l + 1}} \end{array}} \right)$, $\delta_{12}^l, \delta_{13}^l$也可以这么处理，那么 $$% MathType!MTEF!2!1!+- % feaagKart1ev2aaatCvAUfeBSjuyZL2yd9gzLbvyNv2CaerbuLwBLn % hiov2DGi1BTfMBaeXatLxBI9gBaerbd9wDYLwzYbItLDharqqr1ngB % PrgifHhDYfgatCvAUfeBSn0BKvguHDwzZbqegSSZmxoasaacH8srps % 0lbbf9q8WrFfeuY-Hhbbf9v8qqaqFr0xc9pk0xbba9q8WqFfea0-yr % 0RYxir-Jbba9q8aq0-yq-He9q8qqQ8frFve9Fve9Ff0dmeaabaqaci % GacaGaaeqabaWaaeaaeaqbaOqaaiabes7aKnaaCaaaleqabaGaamiB % aaaakiabg2da9maabmaabaqbaeqabqabaaaaaeaacaaIWaaabaGaaG % imaaqaaiaaicdaaeaacaaIWaaabaGaaGimaaqaaiabes7aKnaaDaaa % leaacaaIXaGaaGymaaqaaiaadYgacqGHRaWkcaaIXaaaaaGcbaGaeq % iTdq2aa0baaSqaaiaaigdacaaIYaaabaGaamiBaiabgUcaRiaaigda % aaaakeaacaaIWaaabaGaaGimaaqaaiabes7aKnaaDaaaleaacaaIXa % GaaGOmaaqaaiaadYgacqGHRaWkcaaIXaaaaaGcbaGaeqiTdq2aa0ba % aSqaaiaaikdacaaIYaaabaGaamiBaiabgUcaRiaaigdaaaaakeaaca % aIWaaabaGaaGimaaqaaiaaicdaaeaacaaIWaaabaGaaGimaaaaaiaa % wIcacaGLPaaacqGHxiIkdaqadaqaauaabeqaciaaaeaacaWG3bWaa0 % baaSqaaiaaikdacaaIYaaabaGaamiBaiabgUcaRiaaigdaaaaakeaa % caWG3bWaa0baaSqaaiaaikdacaaIXaaabaGaamiBaiabgUcaRiaaig % daaaaakeaacaWG3bWaa0baaSqaaiaaigdacaaIYaaabaGaamiBaiab % gUcaRiaaigdaaaaakeaacaWG3bWaa0baaSqaaiaaigdacaaIXaaaba % GaamiBaiabgUcaRiaaigdaaaaaaaGccaGLOaGaayzkaaGaeq4WdmNa % ai4jamaabmaabaGaamOEamaaCaaaleqabaGaamiBaaaaaOGaayjkai % aawMcaaiabg2da9iaabchacaqGHbGaaeizaiaabsgacaqGPbGaaeOB % aiaabEgadaqadaqaaiabes7aKnaaCaaaleqabaGaamiBaiabgUcaRi % aaigdaaaaakiaawIcacaGLPaaacaGGQaGaaeOCaiaab+gacaqG0bGa % aeyoaiaabcdadaqadaqaaiaadEfadaahaaWcbeqaaiaadYgacqGHRa % WkcaaIXaaaaOGaaiilaiaaikdaaiaawIcacaGLPaaaaaa!9D0A! {\delta ^l} = \left( {\begin{array}{*{20}{c}} 0&0&0&0\\ 0&{\delta _{11}^{l + 1}}&{\delta _{12}^{l + 1}}&0\\ 0&{\delta _{12}^{l + 1}}&{\delta _{22}^{l + 1}}&0\\ 0&0&0&0 \end{array}} \right) * \left( {\begin{array}{*{20}{c}} {w_{22}^{l + 1}}&{w_{21}^{l + 1}}\\ {w_{12}^{l + 1}}&{w_{11}^{l + 1}} \end{array}} \right)\sigma ‘\left( {{z^l}} \right) = {\rm{padding}}\left( {{\delta ^{l + 1}}} \right)*{\rm{rot90}}\left( {{W^{l + 1}},2} \right)$$ 在MATLAB中，上式可以用卷积函数的’full’模式计算: ${\delta ^l} = {\rm{conv2}}\left( {{\delta ^{l + 1}},{\rm{rot90}}\left( {W,2} \right),’full’} \right)$

4、卷积层$W$和$b$的梯度

下面看看 $W$ 的梯度表达式

$$\frac{{\partial C}}{{\partial W_{ij}^l}} = \sum\nolimits_{mn} {\frac{{\partial C}}{{\partial z_{mn}^l}}\frac{{\partial z_{mn}^l}}{{\partial W_{ij}^l}}}$$
那么有： $$\frac{{\partial C}}{{\partial W_{11}^l}} = \left( {\begin{array}{*{20}{c}} {a_{33}^{l - 1}}&{a_{32}^{l - 1}}\\ {a_{23}^{l - 1}}&{a_{22}^{l - 1}} \end{array}} \right) * \left( {\begin{array}{*{20}{c}} {\delta _{11}^l}&{\delta _{12}^l}\\ {\delta _{21}^l}&{\delta _{22}^l} \end{array}} \right),\frac{{\partial C}}{{\partial W_{12}^l}} = \left( {\begin{array}{*{20}{c}} {a_{32}^{l - 1}}&{a_{31}^{l - 1}}\\ {a_{22}^{l - 1}}&{a_{21}^{l - 1}} \end{array}} \right) * \left( {\begin{array}{*{20}{c}} {\delta _{11}^l}&{\delta _{12}^l}\\ {\delta _{21}^l}&{\delta _{22}^l} \end{array}} \right)$$ $$\frac{{\partial C}}{{\partial W_{21}^l}} = \left( {\begin{array}{*{20}{c}} {a_{23}^{l - 1}}&{a_{22}^{l - 1}}\\ {a_{13}^{l - 1}}&{a_{12}^{l - 1}} \end{array}} \right) * \left( {\begin{array}{*{20}{c}} {\delta _{11}^l}&{\delta _{12}^l}\\ {\delta _{21}^l}&{\delta _{22}^l} \end{array}} \right),\frac{{\partial C}}{{\partial W_{22}^l}} = \left( {\begin{array}{*{20}{c}} {a_{22}^{l - 1}}&{a_{21}^{l - 1}}\\ {a_{12}^{l - 1}}&{a_{11}^{l - 1}} \end{array}} \right) * \left( {\begin{array}{*{20}{c}} {\delta _{11}^l}&{\delta _{12}^l}\\ {\delta _{21}^l}&{\delta _{22}^l} \end{array}} \right)$$ 如果，我们把上面四个式子的 $a_{ij}^{l-1}$,排列起来并去掉重复的元素，得到的刚好是 ${\rm{rot90}}\left( {{a^{l - 1}},2} \right)$,那么有 $$\frac{{\partial C}}{{\partial {W^l}}} = {\delta ^l} * {\rm{rot90}}\left( {{a^{l - 1}},2} \right)$$ 在MATLAB中的实现为 $$\frac{{\partial C}}{{\partial {W^l}}} = conv2\left( {{\delta ^l}{\rm{,rot90}}\left( {{a^{l - 1}},2} \right),'valid'} \right)$$

对于偏执 $b$ 的梯度表达式，由于 $\delta$ 是张量，而 $b$ 只是一个标量（如果 $\delta$ 是三维张量, 则 $b$ 是一个向量），不能像DNN那样，$b$ 直接与 $\delta$ 相同。通常的做法（UFLDL Tutorial）是将$\delta$ 的各个子矩阵的项分别求和，得到一个误差向量，即 $b$ 的梯度为：

$$\frac{{\partial C}}{{\partial {b^l}}} = \sum\nolimits_{mn} {\delta _{mn}^l}$$ 但是，为什么是和，而不是平均值呢? 一般的解释是，因为 $b$ 影响了所有的输出神经元，所以把所有的误差加起来。但是我认为平均值更合理些（个人愚见）: $$\frac{{\partial C}}{{\partial {b^l}}} = {\rm{mean}}\left( {\sum\nolimits_{mn} {\delta _{mn}^l} } \right)$$ 在常规BP网络， $\frac{{\partial C}}{{\partial {{\bf{b}}^l}}} = {\left( {\begin{array}{*{20}{c}} {\frac{{\partial C}}{{\partial b_1^l}}}&{\frac{{\partial C}}{{\partial b_2^l}}}& \ldots &{\frac{{\partial C}}{{\partial b_n^l}}} \end{array}} \right)^T} = {\left( {\begin{array}{*{20}{c}} {\delta _1^l}&{\delta _2^l}& \ldots &{\delta _n^l} \end{array}} \right)^T}$，假如，我们也共享偏置，那么应该有 $\frac{{\partial C}}{{\partial {{\bf{b}}^l}}} = {\left( {\begin{array}{*{20}{c}} {\frac{{\partial C}}{{\partial b_{}^l}}}&{\frac{{\partial C}}{{\partial b_{}^l}}}& \ldots &{\frac{{\partial C}}{{\partial b_{}^l}}} \end{array}} \right)^T} = {\left( {\begin{array}{*{20}{c}} {\delta _1^l}&{\delta _2^l}& \ldots &{\delta _n^l} \end{array}} \right)^T}$,显然 $\frac{{\partial C}}{{\partial {b^l}}} = {\rm{mean}}\left( {{\delta ^l}} \right)$会更合理。

5、卷积神经网络反向传播算法总结

下面，我们以最基本的批量随机梯度下降法对卷积神经网络的的反向传播算法做一个简单的总结：

需要确定参数有：

• 小批量数据的大小$m$
• CNN模型的层数 $L$ 和所有隐藏层的类型
• 对于卷积层，要定义卷积核的大小 $k$，卷积核子矩阵的维度 $d$，填充大小 $p$，步幅 $s$
• 对于池化层，要定义池化区域大小 $h$ 和池化标准(max 或者 mean)
• 对于全连接层，要定义全连接层的激活函数和各层的神经元个数
• 对于输出层，要定义输出函数和代价函数，多分类任务一般采用 softmax 函数和交叉熵代价函数
• 超参数：学习速率 $\eta$, 最大迭代次数 max_iter, 和停止条件$\epsilon$
• ……

计算步骤
1. 初始化每个隐含层的 $W,b$ 的值为随机数
2. 正向传播
2.1).将输入数据 $x$ 赋值于输入神经元 $a^1, a^1 = x$
2.2).从第二层开始，根据下面3种情况进行前向传播计算:

• 如果当前是全连接层：则有 $a^{l} = \sigma(z^{l}) = \sigma(W^la^{l-1} + b^{l})$
• 如果当前是卷积层：则有 $a^{l} = \sigma(z^{l}) = \sigma(W^l*a^{l-1} + b^{l})$
• 如果当前是池化层：则有 $a^{l}= \texttt{pool}(a^{l-1})$

2.3).对于输出层第 $L$ 层，计算输出 $a^{L}= \texttt{softmax}(z^{l}) = \texttt{softmax}(W^la^{l-1} + b^{l})$

3. 反向传播
3.1).通过损失函数计算输出层的 $\delta^L$
3.2).从倒数第二层开始，根据下面3种情况逐层进行反向传播计算：

• 如果当前是全连接层：则有 $\delta^{l} = (W^{l+1})^T\delta^{l+1}\odot \sigma^{'}(z^{l})$
• 如果上层是卷积层：则有 $\delta^{l} = \delta^{l+1}*\texttt{rot180}(W^{l+1}) \odot \sigma^{'}(z^{l})$
• 如果上层是池化层：则有 $\delta^{l} = \texttt{upsample}(\delta^{l+1})$。

4. 根据以下两种情况进行模型更新：
4.1).如果当前是全连接层： $$W^l = W^l -\frac{\eta}{m} \sum \left[ \delta^{l}(a^{ l-1})^T\right]$$ $$b^l = b^l -\frac{\eta}{m} \sum \left( \delta^{l} \right)$$ 4.2).如果当前是卷积层，对于每一个卷积核有： $$W^l = W^l - \frac{\eta}{m} \sum \left[ \delta^{l}*\texttt{rot90}(a^{ l-1},2)\right]$$ $$b^l = b^l - \frac{\eta}{m} \sum \left[ \texttt{mean}(\delta^{l})\right]$$