多元线性回归推导 — Multi-Variable Linear Regression Derivation

多元线性回归通式定义如下：

⎧ ⎩ ⎨ ⎪ ⎪ y ̂ i = f (x) = θ 0 + \sum j = 1 n θ j x j i y i = y ̂ i + ϵ i

$\begin{cases} \hat y_i = f(x) = \theta_0 + \sum\limits_{j=1}^n \theta_j x_i^j \\ y_i = \hat y_i + \epsilon_i \end{cases}$
其中

θ0 $\theta_0$ 为偏置项，

xji $x_i^j$ 为第

i $i$ 条数据的第

j $j$ 项特征值，

θj $\theta_j$ 为第

j $j$ 项特征的权重，

ŷ i $\hat y_i$ 为第

i $i$ 条数据

xi $x_i$ 的预测值，

yi $y_i$ 为第

i $i$ 条数据

xi $x_i$ 对应的实际值，

ϵi $\epsilon_i$ 为预测值

ŷ i $\hat y_i$ 与实际值

yi $y_i$ 之间的残差。

令 $x_i^0 = 1$ ， $\hat y_i = \theta_0 + \sum\limits_{j=1}^n \theta_j x_i^j$ 可被改写成 $\hat y_i = \theta^Tx_i$ 。

假设每条数据 $x_i$ 相互独立且同分布，残差项 $\epsilon_i$ 符合标准正态分布。残差项的概率密度函数如下：

p (ϵ i) = 1 2 π ‾ ‾ ‾ \sqrt σ exp (- ϵ 2 i 2 σ 2) = 1 2 π ‾ ‾ ‾ \sqrt σ exp (- ( y i - θ T x i ) 2 2 σ 2)

$p(\epsilon_i) = \frac {1}{\sqrt{2\pi}\sigma} \exp({-\frac {\epsilon_i^2}{2 \sigma^2}}) = \frac {1}{\sqrt{2\pi}\sigma} \exp({-\frac {(y_i-\theta^Tx_i)^2}{2 \sigma^2}})$

我们的目的是最小化 $\vert \epsilon_i \vert$ ，而 $p(\epsilon_i)$ 在 $\epsilon_i$ 值趋近 $\mu = 0$ 时取得最大值，如上图。因此，已知 $x_i$ 和 $y_i$ ，我们可以使用最大似然估计来得到 $\theta$ ：

L (θ) = \prod i = 1 m p (y i | x i; θ) = \prod i = 1 m 1 2 π ‾ ‾ ‾ \sqrt σ exp (- ( y i - θ T x i ) 2 2 σ 2)

$L(\theta) = \prod\limits_{i=1}^m p(y_i|x_i;\theta) = \prod\limits_{i=1}^m \frac {1}{\sqrt{2\pi}\sigma} \exp({-\frac {(y_i-\theta^Tx_i)^2}{2 \sigma^2}})$
将

L(θ) $L(\theta)$ 取对数，得到

logL(θ) $\log L(\theta)$ ，连乘变成了累加，消掉了指数幂，减少了函数的复杂度。并且，

L(θ) $L(\theta)$ 与

logL(θ) $\log L(\theta)$ 具有相同的变化趋势。因此，求

L(θ) $L(\theta)$ 取得最大值时

θ $\theta$ 的取值等同于求

logL(θ) $\log L(\theta)$ 取得最大值时

θ $\theta$ 的取值。

log L (θ) = \sum i = 1 m log (1 2 π ‾ ‾ ‾ \sqrt σ exp (- ( y i - θ T x i ) 2 2 σ 2)) = \sum i = 1 m (log 1 2 π ‾ ‾ ‾ \sqrt σ - ( y i - θ T x i ) 2 2 σ 2) = \sum i = 1 m log 1 2 π ‾ ‾ ‾ \sqrt σ - \sum i = 1 m ( y i - θ T x i ) 2 2 σ 2

$\begin{align*} & \log L(\theta) = \sum\limits_{i=1}^m \log (\frac {1}{\sqrt{2\pi}\sigma} \exp({-\frac {(y_i-\theta^Tx_i)^2}{2 \sigma^2}}))\\ & = \sum\limits_{i=1}^m (\log \frac {1}{\sqrt{2\pi}\sigma} -\frac {(y_i-\theta^Tx_i)^2}{2 \sigma^2}) \\ & = \sum\limits_{i=1}^m \log \frac {1}{\sqrt{2\pi}\sigma} - \sum\limits_{i=1}^m \frac {(y_i-\theta^Tx_i)^2}{2 \sigma^2} \end{align*}$
由于

∑i=1mlog12π√σ $\sum\limits_{i=1}^m \log \frac {1}{\sqrt{2\pi}\sigma}$ 和

2σ2 $2\sigma^2$ 是常量,

∑i=1m(yi−θTxi)2 $\sum\limits_{i=1}^m (y_i-\theta^Tx_i)^2$ 取得最小值时，

logL(θ) $\log L(\theta)$ 取得最大值。

于是，我们得到了最小二乘法函数：

J (θ) = \sum i = 1 m (y i - θ T x i) 2

$J(\theta) = \sum\limits_{i=1}^m (y_i - \theta^Tx_i)^2$

J(θ) $J(\theta)$ 在其偏微分取值为零时取得最小值。因此，我们对

J(θ) $J(\theta)$ 偏微分，求其取值为零时

θ $\theta$ 的值。

J θ = \partial J \partial θ = [\partial J \partial θ 0 \partial J \partial θ 1 \dots \partial J \partial θ n] T = 0

$J_\theta = \frac {\partial J}{\partial \theta} = \begin{bmatrix} \frac {\partial J}{\partial \theta_0} & \frac {\partial J}{\partial \theta_1} & \dots & \frac {\partial J}{\partial \theta_n} \end{bmatrix}^T = \mathbf 0$
我们以

∂J∂θ0 $\frac {\partial J}{\partial \theta_0}$ 为例：

\partial J \partial θ 0 = \partial \partial θ 0 \sum i = 1 m (y i - θ T x i) 2 = \partial \partial θ 0 [(y 1 - θ T x 1) 2 + (y 2 - θ T x 2) 2 + \dots + (y m - θ T x m) 2] = \partial \partial θ 0 [(y 1 - \sum j = 1 n θ j x j 1) 2 + (y 2 - \sum j = 1 n θ j x j 2) 2 + \dots + (y m - \sum j = 1 n θ j x j m) 2] = 2 (y 1 - \sum j = 1 n θ j x j 1) (- x 01) + 2 (y 2 - \sum j = 1 n θ j x j 2) (- x 02) + \dots + 2 (y m - \sum j = 1 n θ j x j m) (- x 0 m) = - 2 [y 1 - \sum j = 1 n θ j x j 1 y 2 - \sum j = 1 n θ j x j 2 \dots y m - \sum j = 1 n θ j x j m] [x 01 x 02 \dots x 0 m] T = - 2 [y 1 - θ T x 1 y 2 - θ T x 2 \dots y m - θ T x m] [x 01 x 02 \dots x 0 m] T

$\begin{align*} & \frac {\partial J}{\partial \theta_0} = \frac {\partial}{\partial \theta_0} \sum\limits_{i=1}^m(y_i - \theta^Tx_i)^2 \\ & = \frac {\partial}{\partial \theta_0}[(y_1 - \theta^Tx_1)^2 + (y_2 - \theta^Tx_2)^2 + \dots + (y_m - \theta^Tx_m)^2] \\ & = \frac {\partial}{\partial \theta_0}[(y_1 - \sum\limits_{j=1}^n \theta_jx_1^j)^2 + (y_2 - \sum\limits_{j=1}^n \theta_jx_2^j)^2 + \dots + (y_m - \sum\limits_{j=1}^n \theta_jx_m^j)^2] \\ & = 2(y_1 - \sum\limits_{j=1}^n \theta_jx_1^j)(-x_1^0) + 2(y_2 - \sum\limits_{j=1}^n \theta_jx_2^j)(-x_2^0) + \dots + 2(y_m - \sum\limits_{j=1}^n \theta_jx_m^j)(-x_m^0) \\ & = -2 \begin{bmatrix} y_1 - \sum\limits_{j=1}^n\theta_jx_1^j & y_2 - \sum\limits_{j=1}^n\theta_jx_2^j & \dots & y_m - \sum\limits_{j=1}^n\theta_jx_m^j \end{bmatrix} \begin{bmatrix} x_1^0 & x_2^0 & \dots & x_m^0 \end{bmatrix}^T \\ & = -2 \begin{bmatrix} y_1 - \theta^Tx_1 & y_2 - \theta^Tx_2 & \dots & y_m - \theta^Tx_m \end{bmatrix} \begin{bmatrix} x_1^0 & x_2^0 & \dots & x_m^0 \end{bmatrix}^T \end{align*}$
同理，我们可以求得

∂J∂θ1 $\frac {\partial J}{\partial \theta_1}$ 至

∂J∂θn $\frac {\partial J}{\partial \theta_n}$ 的偏微分：

\partial J \partial θ 0 = - 2 [y 1 - θ T x 1 y 2 - θ T x 2 \dots y m - θ T x m] [x 01 x 02 \dots x 0 m] T \partial J \partial θ 1 = - 2 [y 1 - θ T x 1 y 2 - θ T x 2 \dots y m - θ T x m] [x 11 x 12 \dots x 1 m] T ⋮ \partial J \partial θ n = - 2 [y 1 - θ T x 1 y 2 - θ T x 2 \dots y m - θ T x m] [x n 1 x n 2 \dots x n m] T

$\frac {\partial J}{\partial \theta_0} = -2 \begin{bmatrix} y_1 - \theta^Tx_1 & y_2 - \theta^Tx_2 & \dots & y_m - \theta^Tx_m \end{bmatrix} \begin{bmatrix} x_1^0 & x_2^0 & \dots & x_m^0 \end{bmatrix}^T \\ \frac {\partial J}{\partial \theta_1} = -2 \begin{bmatrix} y_1 - \theta^Tx_1 & y_2 - \theta^Tx_2 & \dots & y_m - \theta^Tx_m \end{bmatrix} \begin{bmatrix} x_1^1 & x_2^1 & \dots & x_m^1 \end{bmatrix}^T \\ \vdots \\ \frac {\partial J}{\partial \theta_n} = -2 \begin{bmatrix} y_1 - \theta^Tx_1 & y_2 - \theta^Tx_2 & \dots & y_m - \theta^Tx_m \end{bmatrix} \begin{bmatrix} x_1^n & x_2^n & \dots & x_m^n \end{bmatrix}^T$
将其改写成矩阵的形式：

J θ = \partial J \partial θ = ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ \partial J \partial θ 0 \partial J \partial θ 1 ⋮ \partial J \partial θ n ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ = - 2 [y i - θ T x 1 y 2 - θ T x 2 \dots y m - θ T x m] ⎡ ⎣ ⎢ ⎢ ⎢ ⎢ ⎢ x 01 x 02 ⋮ x 0 m x 11 x 12 x 1 m \dots \dots \dots x n 1 x n 2 x n m ⎤ ⎦ ⎥ ⎥ ⎥ ⎥ ⎥ = - 2 ([y 1 y 2 \dots y m] - [θ T x 1 θ T x 2 \dots θ T x m]) X = - 2 (Y T - θ T [x 1 x 2 \dots x m]) X = - 2 (Y T - θ T X T) X

$\begin{align*} & J_\theta = \frac {\partial J}{\partial \theta} = \begin{bmatrix} \frac {\partial J}{\partial \theta_0} \\ \frac {\partial J}{\partial \theta_1} \\ \vdots \\ \frac {\partial J}{\partial \theta_n} \end{bmatrix} = -2 \begin{bmatrix} y_i - \theta^Tx_1 & y_2 - \theta^Tx_2 & \dots & y_m - \theta^Tx_m \end{bmatrix} \begin{bmatrix} x_1^0 & x_1^1 & \dots & x_1^n \\ x_2^0 & x_2^1 & \dots & x_2^n \\ \vdots \\ x_m^0 & x_m^1 & \dots & x_m^n \end{bmatrix} \\ & = -2( \begin{bmatrix} y_1 & y_2 & \dots & y_m \end{bmatrix} - \begin{bmatrix} \theta^Tx_1 & \theta^Tx_2 & \dots & \theta^Tx_m \end{bmatrix} ) \mathbf X \\ & = -2(\mathbf Y^T - \theta^T \begin{bmatrix} x_1 & x_2 & \dots & x_m \end{bmatrix}) \mathbf X \\ & = -2(\mathbf Y^T - \theta^T \mathbf X^T) \mathbf X \end{align*}$
令

Jθ=−2(YT−θTXT)X=0 $J_\theta = -2(\mathbf Y^T - \theta^T \mathbf X^T) \mathbf X = \mathbf 0$ ，得到：

Y T X = θ T X T X θ T = Y T X (X T X) - 1 = Y T X X - 1 (X T) - 1 = Y T (X T) - 1 = Y T (X - 1) T θ = (θ T) T = X - 1 Y

$\mathbf Y^T \mathbf X = \theta^T \mathbf X^T \mathbf X \\ \theta^T = \mathbf Y^T \mathbf X(\mathbf X^T \mathbf X)^{-1} = \mathbf Y^T \mathbf X \mathbf X^{-1} (\mathbf X^T)^{-1} = \mathbf Y^T (\mathbf X^T)^{-1} = \mathbf Y^T (\mathbf X^{-1})^T \\ \theta = (\theta^T)^T = \mathbf X^{-1} \mathbf Y$

因此，当 $\theta = \mathbf X^{-1} \mathbf Y$ 时， $\hat y_i = \theta^Tx_i$ 能最好地描述 $\mathbf X$ 与 $\mathbf Y$ 之间的线形相关性。

然而，现实中的数据并非都能通过求偏导来求极值。例如： $z=y^2 - x^2$ 。如下图，当 $\frac {\partial z}{\partial y} = 0; \frac {\partial z}{\partial x} = 0$ 时， $y=0; x=0$ （鞍点）却非极值点。因此，实际中多采用梯度下降法或牛顿法来求得目标函数极值的近似解。通过线性回归得到回归参数后，可以通过计算判定系数 $R^2$ 来评估回归函数的拟合优度。

多元线性回归推导 — Multi-Variable Linear Regression Derivation

猜你喜欢