The second section, the four basic formulas of back propagation in neural networks - BackPropagation

Reference article

Neural Network Basics

Neural Networks and Deep Learning.       Michael A. Nielsen

 An article to understand the backpropagation method in neural networks : it is very detailed, and the update process of the weights in the backpropagation method is demonstrated with an example, but the update of the bias is not involved

 

Suppose a three-layer neural network structure diagram is as follows:

For a single training sample x its quadratic cost function can be written as:

          C = 1/2|| y - a ^L || ²  = 1/2∑ _j (y _j  - a _j ^L ) ²

          a _j ^L = σ (z _j ^L )

          z _j ^l  = ∑ _k ω _jk ^l a _k ^l-1  + b _j ^l

The cost function C is a function of a _j ^L , a _j ^L is a function of z _j ^L , z _j ^L is a function of ω _jk ^L , and a function of a _k ^L-1 at the same time ...

Prove four fundamental equations (BP1-BP4), all of which are corollaries of the chain rule of multivariate calculus

            δ _j ^L  = (∂C / ∂a _j ^L ) σ '(z _j ^L ) (BP1)

             δ_j^l = ∑_k ω_kj^l+1δ_k^l+1σ'(z_j^l)                                                    (BP2)

　　　   ∂C/∂ω _jk ^l  = δ _j ^l a _k ^l-1                                                                            (BP3)

              ∂C/∂b_j^l = δ_j^l                                                                                      (BP4)

           

1. Let us start with equation (BP1), which gives the expression for the output error ^{δL .}

            δ_j^L = ∂C/∂z_j^L

    Applying the chain rule, we can reformulate the partial derivatives above in terms of the partial derivatives of the output activations:

              δ_j^L = ∑_k (∂C/∂a_k^L)(∂a_k^L/∂z_j^L)

    Here the summation is run on all neurons k in the output layer, of course, the output activation value a _k ^{L of the kth neuron depends only on the weighted input z of the jth} neuron when k=j _jL . ^_ So when k≠j

    , ∂a _k ^L / ∂z _j ^L =0. The result simplifies to:

               δ_j^L = (∂C/∂a_j^L)(∂a_j^L/∂z_j^L)

    Since a _j ^L =σ(z _j ^L ), the second term on the right can be written as σ'(z _j ^L ), and the equation becomes

                δ _j ^L  = (∂C / ∂a _j ^L ) σ '(z _j ^L )

 

2.证明BP2，它给出了下一层误差δ^l+1的形式表示误差δ^l。为此我们要以δ_k^l+1=∂C/∂z_k^l+1的形式重写 δ_j^l = ∂C/∂z_j^l

              δ_j^l = ∂C/∂z_j^l

                   =∑_k (∂C/∂z_k^l+1)(∂z_k^l+1/∂z_j^l)

                  =∑_k (∂z_k^l+1/∂z_j^l)δ_k^l+1

     这里最后一行我们交换了右边的两项，并用δ_k^l+1的定义带入。为此我们对最后一行的第一项求值，

     注意：

             z_k^l+1 = ∑_jω_kj^l+1a_j^l + b_k^l+1 =  ∑_jω_kj^l+1σ(z_j^l) + b_k^l+1 

     做微分得到

            ∂z_k^l+1 /∂z_j^l = ω_kj^l+1σ'(z_j^l)

     带入上式:

            δ_j^l = ∑_k ω_kj^l+1δ_k^l+1σ'(z_j^l)

 

3.证明BP3。计算输出层∂C/∂ω_jk^L：

            ∂C/∂ω_jkL = ∑_m (∂C/∂a_m^L)(∂a_m^L/∂ω_jk^L )

     这里求和是在输出层的所有神经元k上运行的，当然，第k^th个神经元的输出激活值a_m^L只依赖于当m=j时第j^th个神经元的输入权重ω_jk^L。所以当k≠j

　  时，∂a_m^L/∂ω_jk^L=0。结果简化为：

　　    ∂C/∂ω_jk^L = (∂C/∂a_j^L)(∂a_j^L/∂z_j^L)*(∂z_j^L/∂ω_jk^L)

                         = δ_j^La_k^L-1

       计算输入层上一层(L-1):

           ∂C/∂ω_jk^L-1= (∑_m(∂C/∂a_m^L)(∂a_m^L/∂z_m^L)(∂z_m^L/∂a_j^L-1))(/∂a_j^L-1/∂z_j^L-1)(∂z_j^L-1/∂ω_jk^L-1)

                           = (∑_mδ_m^Lω_mj^L)σ'(z_j^L-1)a_k^L-2

                           = δ_j^L-1a_k^L-2

      对于处输入层的任何一层(l)：

            ∂C/∂ω_jk^l = (∂C/∂z_j^l )(∂z_j^l/∂ω_jk^l ) = δ_j^la_k^l-1

 

4.证明BP4。计算输出层∂C/∂b_j^L：

            ∂C/∂b_j^L = ∑_m (∂C/∂a_m^L)(∂a_m^L/∂b_j^L )

     这里求和是在输出层的所有神经元k上运行的，当然，第k^th个神经元的输出激活值a_m^L只依赖于当m=j时第j^th个神经元的输入权重b_j^L。所以当k≠j

　  时，∂a_m^L/∂b_j^L=0。结果简化为：

　　    ∂C/∂b_j^L = (∂C/∂a_j^L)(∂a_j^L/∂z_j^L)*(∂z_j^L/∂b_j^L)

                         = δ_j^L

       计算输入层上一层(L-1):

           ∂C/∂b_j^L-1= (∑_m(∂C/∂a_m^L)(∂a_m^L/∂z_m^L)(∂z_m^L/∂a_j^L-1))(/∂a_j^L-1/∂z_j^L-1)(∂z_j^L-1/∂b_j^L-1)

                           = (∑_mδ_m^Lω_mj^L)σ'(z_j^L-1)

                           = δ_j^L-1

      对于处输入层的任何一层(l)：

            ∂C/∂b_j^l = (∂C/∂z_j^l )(∂z_j^l/∂b_j^l) = δ_j^l