Deep learning I - III Shallow Neural Network - Derivatives of activation functions激励函数导数推导

Derivatives of activation functions 激励函数的导数

Derivatives of Sigmod

\begin{matrix} (1) & s i g m o d = g (z) = \frac{1}{1 + e^{- z}} \end{matrix}

$sigmod = g(z) = \frac{1}{1+e^{-z}}\tag{1}$

\begin{matrix} (2) & \begin{aligned} g^{'} (z) & = \frac{d}{d z} g (z) \\ = \frac{d (1 + e^{- z})^{- 1}}{d (1 + e^{- z})} * \frac{d (1 + e^{- z})}{d (- z)} * \frac{d (- z)}{d z} \\ = - (1 + e^{- z})^{- 2} * e^{- z} * - 1 \\ = \frac{1}{(1 + e^{- z})} * \frac{e^{- z}}{(1 + e^{- z})} \\ = \frac{1}{(1 + e^{- z})} * \frac{1 + e^{- z} - 1}{(1 + e^{- z})} \\ = \frac{1}{(1 + e^{- z})} * (1 - \frac{1}{(1 + e^{- z})}) \\ = g (z) * (1 - g (z)) \end{aligned} \end{matrix}

$\begin{split} g'(z) &= \frac{{\rm d}}{{\rm d}z}g(z) \\&= \frac{{\rm d}(1 + {e^{-z}})^{-1}}{{\rm d}(1 + {e^{-z}})} * \frac{{\rm d}(1 + {e^{-z}})}{{\rm d}{(-z)}} * \frac{{\rm d}{(-z)}}{{\rm d}z} \\&= -(1 + {e^{-z}})^{-2} * e^{-z} * -1 \\&= \frac{1}{(1 + e^{-z})} * \frac{e^{-z}}{(1 + e^{-z})} \\&= \frac{1}{(1 + e^{-z})} * \frac{1 + e^{-z} -1}{(1 + e^{-z})} \\&= \frac{1}{(1 + e^{-z})} *( 1- \frac{1}{(1 + e^{-z})}) \\&= g(z) * (1-g(z))\end{split}\tag{2}$
这里写图片描述

Derivatives of tanh

\begin{matrix} (3) & s i n h (z) = \frac{e^{z} - e^{- z}}{2} \end{matrix}

$sinh(z) = \frac{e^z - e^{-z}}{2}\tag{3}$

\begin{matrix} (4) & c o s h (z) = \frac{e^{z} + e^{- z}}{2} \end{matrix}

$cosh(z) = \frac{e^z + e^{-z}}{2}\tag{4}$

\begin{matrix} (5) & \frac{d}{d z} s i n h (z) = \frac{e^{z} + e^{- z}}{2} = c o s h (z) \end{matrix}

$\frac{{\rm d}}{{\rm d}z}sinh(z) = \frac{e^z + e^{-z}}{2} = cosh(z)\tag{5}$

\begin{matrix} (6) & \frac{d}{d z} c o s h (z) = \frac{e^{z} - e^{- z}}{2} = s i n h (z) \end{matrix}

$\frac{{\rm d}}{{\rm d}z}cosh(z) = \frac{e^z - e^{-z}}{2} = sinh(z)\tag{6}$

\begin{matrix} (7) & t a n h (z) = \frac{s i n h (z)}{c o s h (z)} = g (z) = \frac{e^{z} - e^{- z}}{e^{z} + e^{- z}} \end{matrix}

$tanh(z) = \frac{sinh(z)}{cosh(z)} = g(z) = \frac{e^{z} - e^{-z}}{e^{z} + e^{-z}}\tag{7}$

\begin{matrix} (8) & \begin{aligned} g^{'} (z) & = \frac{d}{d z} (s i n h (z) c o s h (z)^{- 1}) \\ = \frac{d}{d z} s i n h (z) * c o s h (z)^{- 1} + s i n h (z) * \frac{d}{d c o s h (z)} c o s h (z)^{- 1} * \frac{d}{d z} c o s h (z) \\ = 1 + s i n h (z) * - \frac{1}{c o s h (z)^{2}} * s i n h (z) \\ = 1 - t a n h (z)^{2} \end{aligned} \end{matrix}

$\begin{split}g'(z) &= \frac{{\rm d}}{{\rm d}z}(sinh(z)cosh(z)^{-1}) \\&= \frac{{\rm d}}{{\rm d}z}sinh(z) * cosh(z)^{-1} + sinh(z)*\frac{{\rm d}}{{\rm d}cosh(z)}cosh(z)^{-1} * \frac{{\rm d}}{{\rm d}z}cosh(z) \\&= 1 + sinh(z)*-\frac{1}{cosh(z)^2}*sinh(z) \\&= 1 - tanh(z)^2 \end{split}\tag{8}$
这里写图片描述

Derivatives of ReLU

\begin{matrix} (9) & R e L U = g (z) = m a x (0, z) \end{matrix}

$ReLU = g(z) = max(0, z)\tag{9}$

\begin{matrix} (10) & g^{'} (z) = {\begin{cases} 0 & i f z < 0 \\ 1 & i f z \geq 0 \end{cases} \end{matrix}

$g'(z) = \begin{cases} 0 & if\ z < 0\\ 1 & if\ z \geq 0 \end{cases}\tag{10}$

\begin{matrix} (11) & L e a k y R e L U = g (z) = m a x (0.01 z, z) \end{matrix}

$Leaky\ ReLU = g(z) = max(0.01z, z)\tag{11}$

\begin{matrix} (12) & g^{'} (z) = {\begin{cases} 0.01 & i f z < 0 \\ 1 & i f z \geq 0 \end{cases} \end{matrix}

$g'(z) = \begin{cases} 0.01 & if\ z < 0\\ 1 & if\ z \geq 0 \end{cases}\tag{12}$
这里写图片描述