激活函数（Activation Functions）

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/YhL_Leo/article/details/56488640

本系列文章由 @yhl_leo 出品，转载请注明出处。
文章链接： http://blog.csdn.net/yhl_leo/article/details/56488640

---

简单整理了一下目前深度学习中提出的激活函数，按照激活函数是否可微的性质分为3类：

论文及资料已上传至GitHub: yhlleo/Activations.

• 平滑非线性函数（Smooth nonlinearities）:

  • tanh : Efficient BackProb, Neural Networks 1998
    $$f(x) = \frac{e^x-e^{-x}}{e^x+e^{-x}}$$
  • sigmoid: Efficient BackProb, Neural Networks 1998
    $$f(x) = \frac{1}{1+e^{-x}}$$
  • softplus: Incorporating Second-Order Functional Knowledge for Better Option Pricing, NIPS 2001
    $$f(x) = ln(1+e^x)$$
  • softsign:
    $$f(x) = \frac{x}{|x| + 1}$$
  • ELU: Fast and Accuracy Deep Network Learning by Exponential Linear Units, ICLR 2016
    $$f(x)= \begin{cases} \begin{array}{lcl} x & & ; x>0 \\ \alpha(e^x - 1) & & ; x \leq 0， \alpha = 1.0 \\ \end{array} \end{cases}$$

• 连续但并不是处处可微（Continuous but not everywhere differentiable）

  • ReLU: Deep Sparse Rectifier Neural Networks, AISTATS 2011
    $$f(x) = max(0, x)$$
  • ReLU6: tf.nn.relu6
    $$f(x) = min(max(0, x), 6)$$
  • SReLU: Shift Rectified Linear Unit
    $$f(x) = max(-1, x)$$
  • Leaky ReLU: Rectifier Nonlinearities Improve Neural Network Acoustic Models, ICML 2013
    $$f(x) = max(\alpha x, x), \alpha =0.01$$
  • PReLU: Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification, arXiv 2015
    $$f(x) = max(ax, x), a\in [0,1), \alpha \text{ is learned}$$
  • RReLU: Empirical Evaluation of Rectified Activations in Convolution Network, arXiv 2015
    $$f(x) = max(\alpha x, x), \alpha \text{~} U(l, u), l<u \ and \ l, u \in[0, 1)$$
  • CReLU: Understanding and Improving Convolutional Neural Networks via Concatenated Rectified Linear Units, arXiv 2016
    $$f(x) = concat(\text{relu}(x), \text{relu}(-x))$$

• 离散的（Discrete）

  • NReLU: Rectified Linear Units Improve Restricted Boltzmann Machines, ICML 2010
    $$f(x) = max(0, x+\mathcal{N}(0, \sigma(x)))$$
  • Noisy Activation Functions: Noisy Activation Functions, ICML 2016

---

简单绘制部分激活函数曲线：

---

References:

• http://cs224d.stanford.edu/lecture_notes/LectureNotes3.pdf
• https://www.tensorflow.org/api_guides/python/nn
• http://www.cnblogs.com/rgvb178/p/6055213.html