SVM（四）：支持向量回归

4 支持向量回归

4.1 问题定义

给定训练样本 $D= \{(\boldsymbol x_1,y_1), (\boldsymbol x_2,y_2), ..., (\boldsymbol x_m,y_m)\},y_i \in \Bbb R$，希望学得形如 $f(\boldsymbol x) = \boldsymbol w^T \boldsymbol x + b$ 的模型，使 $f(\boldsymbol x)与y$尽可能接近，其中 $\boldsymbol w$和 $\boldsymbol b$是待确定参数。

支持向量回归（SVR，Support Vector Regression）假设能够容忍 $f(\boldsymbol x)$与 $y$之间最多 $\epsilon$的偏差，即仅当 $f(\boldsymbol x)$与 $y$之间的差别绝对值大于 $\epsilon$时才计算损失。这相当于以
$f(\boldsymbol x) = \boldsymbol w^T \boldsymbol x + b$为中心，构建了一个宽带 $2\epsilon$的间隔带，若训练样本落入此间隔带，则被认为预测正确。

SVR优化目标如下：
$\min\;\; \frac{1}{2}||\boldsymbol w||^2 +C\sum\limits_{i=1}^{m} {l}_{\epsilon} (y_i(\boldsymbol w^T \boldsymbol x_i + b)-1)$

其中 $C>0$是正则化常数， ${l}_{\epsilon}$是 $\epsilon$-不敏感损失（ $\epsilon$-insensitive loss）损失函数：
$l_{\epsilon}(z) = \begin{cases} 0 & {if } |z| \leq \epsilon \\ |z|-\epsilon & {otherwise } \end{cases}$

4.2 对偶问题

引入松弛变量 $\xi_i^{\lor}>0, \xi_i^{\land}>0$（两边松弛变量可能不同），优化目标变为：
$\begin{aligned} \min & \; \frac{1}{2}||w||_2^2 + C\sum\limits_{i=1}^{m}(\xi_i^{\lor}+ \xi_i^{\land}) \\ s.t. \;\; &y_i - (\boldsymbol w^T \boldsymbol x_i + b) \leq \epsilon + \xi_i^{\land}, \\& \; \boldsymbol w^T \boldsymbol x_i + b - y_i \; \leq \epsilon + \xi_i^{\lor}, \\&\xi_i^{\lor} \geq 0, \;\; \xi_i^{\land} \geq 0 \;(i = 1,2,..., m) \end{aligned}$

引入拉格朗日乘子 $\mu_i^{\lor} \geq 0, \mu_i^{\land} \geq 0, \alpha_i^{\lor} \geq 0, \alpha_i^{\land} \geq 0$：

$L(\boldsymbol w,b,\boldsymbol \alpha^{\lor}, \boldsymbol \alpha^{\land}, \boldsymbol \xi_i^{\lor}, \boldsymbol \xi_i^{\land}, \boldsymbol \mu^{\lor}, \boldsymbol \mu^{\land}) = \frac{1}{2}||w||_2^2 + C\sum\limits_{i=1}^{m}(\xi_i^{\lor}+ \xi_i^{\land}) - \sum\limits_{i=1}^{m}\mu_i^{\lor}\xi_i^{\lor} - \sum\limits_{i=1}^{m}\mu_i^{\land}\xi_i^{\land}+ \sum\limits_{i=1}^{m}\alpha_i^{\lor}(f(\boldsymbol x_i)-y_i -\epsilon - \xi_i^{\lor}) + \sum\limits_{i=1}^{m}\alpha_i^{\land}(y_i - f(\boldsymbol x_i) -\epsilon - \xi_i^{\land})$其中，
$f(\boldsymbol x_i) = \boldsymbol w^T \boldsymbol x_i + b$

优化目标 $\min_{\boldsymbol w,b,\xi_i^{\lor}, \xi_i^{\land}}\; \max_{\mu_i^{\lor}, \mu_i^{\land}, \alpha_i^{\lor}, \alpha_i^{\land}}\; L(\boldsymbol w,b,\boldsymbol \alpha^{\lor}, \boldsymbol \alpha^{\land}, \boldsymbol \xi^{\lor}, \boldsymbol \xi^{\land}, \boldsymbol \mu^{\lor}, \boldsymbol \mu^{\land})$

满足KTT条件，对偶问题为： $\max_{\mu_i^{\lor}, \mu_i^{\land}, \alpha_i^{\lor}, \alpha_i^{\land}}\; \min_{\boldsymbol w,b,\xi_i^{\lor}, \xi_i^{\land}}\; L(\boldsymbol w,b,\boldsymbol \alpha^{\lor}, \boldsymbol \alpha^{\land}, \boldsymbol \xi^{\lor}, \boldsymbol \xi^{\land}, \boldsymbol \mu^{\lor}, \boldsymbol \mu^{\land})$
首先通过对 $\boldsymbol w,b,\xi_i^{\lor}, \xi_i^{\land}$求偏导，计算极小值：
$\frac{\partial L}{\partial \boldsymbol w} = 0 \;\Rightarrow \boldsymbol w = \sum\limits_{i=1}^{m}(\alpha_i^{\land} - \alpha_i^{\lor})\boldsymbol x_i$ $\frac{\partial L}{\partial b} = 0 \;\Rightarrow \sum\limits_{i=1}^{m}(\alpha_i^{\land} - \alpha_i^{\lor}) = 0$ $\frac{\partial L}{\partial \xi_i^{\lor}} = 0 \;\Rightarrow C = \alpha^{\lor} + \mu^{\lor}$ $\frac{\partial L}{\partial \xi_i^{\land}} = 0 \;\Rightarrow C= \alpha^{\land}+ \mu^{\land}$

代回至
$L(\boldsymbol w,b,\boldsymbol \alpha^{\lor}, \boldsymbol \alpha^{\land}, \boldsymbol \xi^{\lor}, \boldsymbol \xi^{\land}, \boldsymbol \mu^{\lor}, \boldsymbol \mu^{\land})$

$\begin{aligned} \min_{\boldsymbol w,b,\xi_i^{\lor}, \xi_i^{\land}}\; L(\boldsymbol w,b,\boldsymbol \alpha^{\lor}, \boldsymbol \alpha^{\land}, \boldsymbol \xi^{\lor}, \boldsymbol \xi^{\land}, \boldsymbol \mu^{\lor}, \boldsymbol \mu^{\land}) & = \sum\limits_{i=1}^{m}y_i(\alpha_i^{\land}- \alpha_i^{\lor}) - \epsilon(\alpha_i^{\land} + \alpha_i^{\lor})- \frac{1}{2}\sum_{i=1}^{m}\sum_{j=1}^{m}(\alpha_i^{\land} - \alpha_i^{\lor})(\alpha_j^{\land} - \alpha_j^{\lor}) \boldsymbol x_i^T \boldsymbol x_j \\ s.t. \; &\sum\limits_{i=1}^{m}(\alpha_i^{\land} - \alpha_i^{\lor}) = 0 \\ & 0 \leq \alpha_i^{\lor},\alpha_i^{\land} \leq C \; (i =1,2,...m) \end {aligned}$

原问题最终转换为如下形式的对偶问题：
$\begin{aligned} \max_{\boldsymbol \alpha^{\land},\boldsymbol \alpha^{\lor}}\;\; & \sum\limits_{i=1}^{m}y_i(\alpha_i^{\land}- \alpha_i^{\lor}) - \epsilon(\alpha_i^{\land} + \alpha_i^{\lor})- \frac{1}{2}\sum_{i=1}^{m}\sum_{j=1}^{m}(\alpha_i^{\land} - \alpha_i^{\lor})(\alpha_j^{\land} - \alpha_j^{\lor}) \boldsymbol x_i^T \boldsymbol x_j \\ s.t. \; &\sum\limits_{i=1}^{m}(\alpha_i^{\land} - \alpha_i^{\lor}) = 0 \\ & 0 \leq \alpha_i^{\lor},\alpha_i^{\land} \leq C \; (i =1,2,...m) \end {aligned}$

此时，优化函数仅有 $\boldsymbol \alpha^{\land},\boldsymbol \alpha^{\lor}$做为参数，可采用SMO（Sequential Minimal Optimization）求解，进而得出 $\boldsymbol w,b$。