Deep learning II - I Practical aspects of deep learning - Regularizing your neural network 神经网络范数正则化

Regularizing your neural network 神经网络正则化

---

Logistic regression regularization

先用简单的逻辑回归正则化作为例子，因为神经网络的参数$W$是2维的。

1. 无正则
   

   $$J(w,b) = \frac{1}{m} \sum_{i=1}^{m} {\cal L}(\hat{y}^{(i)} - y^{(i)})$$

2. $L_2$ 正则
   

   $$J(w,b) = \frac{1}{m} \sum_{i=1}^{m} {\cal L}(\hat{y}^{(i)} - y^{(i)}) + \frac{\lambda}{2m}||w||^2_2$$
   $$||w||^2_2 = \sum_{j=1}^{n_x}w_j^2 = w^Tw$$

3. $L_1$ 正则
   $$J(w,b) = \frac{1}{m} \sum_{i=1}^{m} {\cal L}(\hat{y}^{(i)} - y^{(i)}) + \frac{\lambda}{m}||w||_1$$

$$||w||_1 = \sum_{j=1}^{n_x}|w|_j$$

Neural network regularization

1. Frobenius正则(类似$L_2$正则)
   $$J(w^{[1]},b^{[1]}, \cdots , w^{[l]},b^{[l]}) = \frac{1}{m} \sum_{i =1}^m {\cal L}(\hat{y}^{(i)}, y^{(i)}) + \frac{1}{2m} \sum_{l=1}^L ||w^{[l]}||_F^2$$
   $$||w^{[l]}||_F^2 = \sum_{i = 1}^{n^{[l]}} \sum_{j = 1}^{n^{[l-1]}}(w^{[l]}_{ij})^2$$

相较于无正则化的反向传播，正则化的反向传播在更新$W$时，会对其进行权重衰减（weight decay），并下降。

$${\rm d}w^{[l]} = (from\ backpropagation) + \frac{\lambda}{m}w^{[l]}$$
$$\begin{split}w^{[l]}: &=w^{[l]} - \alpha {\rm d}w^{[l]} \\&= w^{[l]} - \alpha \frac{\lambda}{m}w^{[l]} - \alpha (from\ backpropagation) \\&= (1- \alpha \frac{\lambda}{m})w^{[l]} - \alpha (from\ backpropagation)\end{split}$$