0. 前言
激活函数的主要作用是提供网络的非线性建模能力。其主要特性是：可微性、单调性、输出值有范围。本节总结常见的一些激活函数，以及对其可视化。
1. Sign(符号)函数
S i g n ( x ) = { − 1 , x < 0 0 , x = 0 1 , x > 0 Sign(x)=\left\{\begin{aligned} -1, & \text \ x<0 \\0, & \text \ x=0 \\1, & \text \ x>0\end{aligned}\right. Sign(x)=⎩⎪⎨⎪⎧−1,0,1, x<0 x=0 x>0
2. Sigmoid函数
s i g m o i d ( x ) = sigmoid(x)= sigmoid(x)= 1 1 + e − x {1}\over{1+e^{-x}} 1+e−x1
s i g m o i d ′ ( x ) = sigmoid '(x)= sigmoid′(x)= s i g m o i d ( x ) ⋅ ( 1 − s i g m o i d ( x ) ) sigmoid(x) \cdot (1-sigmoid(x)) sigmoid(x)⋅(1−sigmoid(x))
3. tanh(双曲正切)函数
tanh ( x ) = sinh x cosh x = e x − e − x e x + e − x \tanh (x)=\frac{\sinh x}{\cosh x}=\frac{e^{x}-e^{-x}}{e^{x}+e^{-x}} tanh(x)=coshxsinhx=ex+e−xex−e−x
tanh ′ ( x ) = 1 − ( t a n h ( x ) ) 2 \tanh' (x)=1-(tanh(x))^2 tanh′(x)=1−(tanh(x))2
4. ReLU函数
f ( x ) = { 0 , x ≤ 0 x , x > 0 f(x)=\left\{\begin{array}{ll}0, & x \leq 0 \\ x, & x>0\end{array}\right. f(x)={ 0,x,x≤0x>0
5. Leaky ReLU函数
RReLU： f ( x ) = { α x , x ≤ 0 x , x > 0 f(x)=\left\{\begin{array}{ll}\alpha x, & x \leq 0 \\ x, & x>0\end{array}\right. f(x)={ αx,x,x≤0x>0
当 α = 0.01 \alpha=0.01 α=0.01 时，为Leaky ReLU
f ( x ) = { 0.01 x , x ≤ 0 x , x > 0 f(x)=\left\{\begin{array}{ll}0.01x, & x \leq 0 \\ x, & x>0\end{array}\right. f(x)={ 0.01x,x,x≤0x>0
