符号(Notations)

(1) $\boldsymbol{f}$ 表示多个函数的组合 $\left[{ \begin{array}{center} f_1(\boldsymbol{x})\\ f_2(\boldsymbol{x})\\ \vdots\\ f_m(\boldsymbol{x}) \end{array}}\right]$
(2) $\nabla f(\boldsymbol{x})$ 表示函数 $f(\boldsymbol{x})$ 的梯度
(3)粗体符号表示矢量或者矩阵，比如 $\boldsymbol{x}$ 表示一个矢量， $\boldsymbol{H}$ 表示一个矩阵。

2. 梯度

定义对于任意点（ $\boldsymbol{x}\in \mathbb{R}^n$ ）的映射 $\boldsymbol{f}: \mathbb{R}^n\rightarrow \mathbb{R}^m$

\begin{aligned} (156) & f (x) = [\begin{array}{cr} f_{1} (x) \\ f_{2} (x) \\ ⋮ \\ f_{m} (x) \end{array}] = {[f_{1} (x), f_{2} (x), \dots, f_{m} (x)]}^{T} \end{aligned}

$\begin{align} \boldsymbol{f}({\boldsymbol{x}})=\left[{ \begin{array}{center} f_1(\boldsymbol{x})\\ f_2(\boldsymbol{x})\\ \vdots\\ f_m(\boldsymbol{x}) \end{array}}\right]= \left[{f_1(\boldsymbol{x}),f_2(\boldsymbol{x}),\cdots,f_m(\boldsymbol{x})}\right]^T \end{align}$
其中

f_{i} (x)

$f_i(\boldsymbol{x})$ 是一个

R^{n} \to R

$\mathbb{R}^n\rightarrow \mathbb{R}$ 的映射。

\frac{\partial f (x)}{\partial x_{j}}

$\frac{\partial \boldsymbol{f} (\boldsymbol{x})}{\partial x_j}$ 定义为

\begin{aligned} (157) & \frac{\partial f (x)}{\partial x_{j}} = [\begin{array}{cr} \frac{\partial f_{1} (x)}{\partial x_{j}} \\ \frac{\partial f_{2} (x)}{\partial x_{j}} \\ ⋮ \\ \frac{\partial f_{m} (x)}{\partial x_{j}} \end{array}] = {[\frac{\partial f_{1} (x)}{\partial x_{j}}, \frac{\partial f_{2} (x)}{\partial x_{j}}, \dots, \frac{\partial f_{m} (x)}{\partial x_{j}}]}^{T} \end{aligned}

$\begin{align} \frac{\partial \boldsymbol{f} (\boldsymbol{x})}{\partial x_j}=\left[{ \begin{array}{center} \frac{\partial f_1(\boldsymbol{x})}{\partial x_j}\\ \frac{\partial f_2(\boldsymbol{x})}{\partial x_j}\\ \vdots\\ \frac{\partial f_m(\boldsymbol{x})}{\partial x_j} \end{array}}\right]=\left[{\frac{\partial f_1(\boldsymbol{x})}{\partial x_j},\frac{\partial f_2(\boldsymbol{x})}{\partial x_j},\cdots,\frac{\partial f_m(\boldsymbol{x})}{\partial x_j}}\right]^T \end{align}$
上面的矢量是曲线

f (x)

$\boldsymbol{f}(\boldsymbol{x})$ 的在点

x

$\boldsymbol{x}$ 处的切矢量，它可以通过固定其余的

x_{i} （ i \neq j ）

$x_i（i\ne j）$ 仅仅改变

x_{j}

$x_j$ 得到。

可导函数 $\boldsymbol{f}: \mathbb{R}^n\rightarrow \mathbb{R}^m$ 的导数定义为

$\begin{aligned} (158) & D f (x) = [\frac{\partial f (x)}{\partial x_{1}}, \frac{\partial f (x)}{\partial x_{2}}, \dots, \frac{\partial f (x)}{\partial x_{n}}] = [\begin{array}{cr} \nabla f_{1} (x)^{T} \\ \nabla f_{2} (x)^{T} \\ ⋮ \\ \nabla f_{m} (x)^{T} \end{array}] = [\begin{array}{cr} \frac{\partial f_{1} (x)}{\partial x_{1}}, \dots, \frac{\partial f_{1} (x)}{\partial x_{n}} \\ ⋮ \\ \frac{\partial f_{m} (x)}{\partial x_{1}}, \dots, \frac{\partial f_{m} (x)}{\partial x_{n}} \end{array}] \in R^{m \times n} \end{aligned}$ $\begin{align} D\boldsymbol{f}(\boldsymbol{x})=\left[{\frac{\partial \boldsymbol{f}(\boldsymbol{x})}{\partial x_1},\frac{\partial \boldsymbol{f}(\boldsymbol{x})}{\partial x_2},\cdots, \frac{\partial \boldsymbol{f}(\boldsymbol{x})}{\partial x_n}}\right] =\left[ \begin{array}{center} \nabla f_1(\boldsymbol{x})^T\\ \nabla f_2(\boldsymbol{x})^T\\ \vdots\\ \nabla f_m(\boldsymbol{x})^T \end{array} \right] =\left[{ \begin{array}{center} \frac{\partial f_1(\boldsymbol{x})}{\partial x_1}, \cdots, \frac{\partial f_1(\boldsymbol{x})}{\partial x_n}\\ \vdots\\ \frac{\partial f_m(\boldsymbol{x})}{\partial x_1}, \cdots, \frac{\partial f_m(\boldsymbol{x})}{\partial x_n} \end{array} }\right]\in \mathbb{R}^{m\times n} \end{align}$
$f: \mathbb{R}^n\rightarrow \mathbb{R}$ 是可导的，则函数在点 $\boldsymbol{x}$ 处的梯度梯度 $\nabla f(\boldsymbol{x})$ 可表示为

$\begin{aligned} (159) & \nabla f (x) & = D f (x)^{T} = [\begin{array}{cr} \frac{\partial f (x)}{\partial x_{1}} \\ \frac{\partial f (x)}{\partial x_{2}} \\ ⋮ \\ \frac{\partial f (x)}{\partial x_{n}} \end{array}] \in R^{n} \\ (160) & \nabla^{2} f (x) & = [\begin{array}{cr} \frac{\partial f^{2} (x)}{\partial x_{1}^{2}}, \dots, \frac{\partial f^{2} (x)}{\partial x_{1} \partial x_{n}} \\ ⋮ \\ \frac{\partial f^{2} (x)}{\partial x_{n} \partial x_{1}}, \dots, \frac{\partial f^{2} (x)}{\partial x_{n}^{2}} \end{array}] \in R^{n \times n} \end{aligned}$ $\begin{align} \nabla f(\boldsymbol{x})&=D f(\boldsymbol{x})^T=\left[{ \begin{array}{center} \frac{\partial f(\boldsymbol{x})}{\partial x_1}\\ \frac{\partial f(\boldsymbol{x})}{\partial x_2}\\ \vdots\\ \frac{\partial f(\boldsymbol{x})}{\partial x_n} \end{array} }\right]\in \mathbb{R}^n\\ \nabla^2 f(x)&=\left[{ \begin{array}{center} \frac{\partial f^2(\boldsymbol{x})}{\partial x_1^2}, \cdots, \frac{\partial f^2(\boldsymbol{x})}{\partial x_1\partial x_n}\\ \vdots\\ \frac{\partial f^2(\boldsymbol{x})}{\partial x_n\partial x_1}, \cdots, \frac{\partial f^2(\boldsymbol{x})}{\partial x_n^2} \end{array} }\right]\in \mathbb{R}^{n\times n} \end{align}$

3. Example：线性高斯模型的最小二乘解

\begin{aligned} (731) & y = H x + n \end{aligned}

$\begin{align} \boldsymbol{y}=\boldsymbol{Hx}+\boldsymbol{n} \end{align}$
其中

x \in R^{n}

$\boldsymbol{x}\in \mathbb{R}^n$ ,

H \in R^{m \times n}

$\boldsymbol{H}\in \mathbb{R}^{m \times n}$ 是观测矩阵，

n \sim N (μ, σ^{2} I)

$\boldsymbol{n}\sim \mathcal{N}(\boldsymbol{\mu}, \sigma^2\mathbf{I})$ ，

y \in R^{m}

$\boldsymbol{y}\in \mathbb{R}^m$ 是观测向量。
解：最小二乘估计器为

\begin{aligned} (732) & \hat{x} = \underset{x}{arg min} ‖ y - H x ‖ \end{aligned}

$\begin{align} \hat{\boldsymbol{x}}=\underset{\boldsymbol{x}}{\text{arg} \min}\|\boldsymbol{y}-\boldsymbol{Hx}\| \end{align}$
令

J = ‖ y - H x ‖^{2}

$J=\|\boldsymbol{y}-\boldsymbol{Hx}\|^2$

\begin{aligned} (733) & J & = (y - H x)^{T} (y - H x) \\ (734) & = y^{T} y - y^{T} H x - x^{T} H^{T} y + x^{T} H^{T} H x \end{aligned}

$\begin{align} J&=(\boldsymbol{y}-\boldsymbol{Hx})^T(\boldsymbol{y}-\boldsymbol{Hx})\\ &=\boldsymbol{y}^T\boldsymbol{y}-\boldsymbol{y}^T\boldsymbol{Hx}-\boldsymbol{x}^T\boldsymbol{H}^T\boldsymbol{y}+\boldsymbol{x}^T\boldsymbol{H}^T\boldsymbol{Hx} \end{align}$
求梯度

\begin{aligned} (735) & \nabla J & = - 2 H^{T} y + 2 H^{T} H x \end{aligned}

$\begin{align} \nabla J&=-2\boldsymbol{H}^T\boldsymbol{y}+2\boldsymbol{H}^T\boldsymbol{H}\boldsymbol{x} \end{align}$
令梯度等于0，有

\begin{aligned} (736) & \hat{x} = (H^{T} H)^{- 1} H^{T} y \end{aligned}

$\begin{align} \hat{\boldsymbol{x}}=(\boldsymbol{H}^T\boldsymbol{H})^{-1}\boldsymbol{H}^T\boldsymbol{y} \end{align}$
因此线性高斯模型的最小二乘估计器为

\hat{x} = (H^{T} H)^{- 1} H^{T} y

$\hat{\boldsymbol{x}}=(\boldsymbol{H}^T\boldsymbol{H})^{-1}\boldsymbol{H}^T\boldsymbol{y}$ 。

Remarks：最小二乘估计器的优点就是不用考虑噪声 $\boldsymbol{n}$ 的分布，当噪声能量很小时，最小二乘估计器的性能会逐渐趋于克拉美-罗下限（CRLB），但，随着噪声的能量增大，最小二乘估计器的性能会逐渐变差。

4. Examp: 求 $f(\boldsymbol{x})$ 的梯度

求 $f(\boldsymbol{x})$ 的梯度

\begin{aligned} (332) & f (x) = a^{T} x \end{aligned}

$\begin{align} f(\boldsymbol{x})=\boldsymbol{a}^T\boldsymbol{x} \end{align}$
其中

f : R^{n} \to R

$f: \mathbb{R}^n\rightarrow \mathbb{R}$ ,

a \in R^{n}

$\boldsymbol{a}\in \mathbb{R}^n$ 是常数，

x \in R^{n}

$\boldsymbol{x}\in\mathbb{R}^n$ 是自变量矢量。现在求

f (x)

$f(\boldsymbol{x})$ 的梯度
解：

\begin{aligned} (333) & f (x) = a^{T} x = a_{1} x_{1} + a_{2} x_{2} + \dots + a_{n} x_{n} \end{aligned}

$\begin{align} f(\boldsymbol{x})=\boldsymbol{a}^T\boldsymbol{x}=a_1x_1+a_2x_2+\cdots+a_nx_n \end{align}$

\begin{aligned} (334) & \nabla f (x) = [\begin{array}{cr} \frac{\partial f (x)}{\partial x_{1}} \\ \frac{\partial f (x)}{\partial x_{2}} \\ ⋮ \\ \frac{\partial f (x)}{\partial x_{n}} \end{array}] = [\begin{array}{cr} a_{1} \\ a_{2} \\ ⋮ \\ a_{n} \end{array}] = a \in R^{n} \end{aligned}

$\begin{align} \nabla f(\boldsymbol{x})=\left[{ \begin{array}{center} \frac{\partial f(\boldsymbol{x})}{\partial x_1}\\ \frac{\partial f(\boldsymbol{x})}{\partial x_2}\\ \vdots\\ \frac{\partial f(\boldsymbol{x})}{\partial x_n} \end{array} }\right]=\left[{ \begin{array}{center} a_1\\ a_2\\ \vdots\\ a_n \end{array} }\right]=\boldsymbol{a}\in \mathbb{R}^n \end{align}$

梯度及最小二乘估计器

符号(Notations)

2. 梯度

3. Example：线性高斯模型的最小二乘解

4. Examp: 求 $f(\boldsymbol{x})$ 的梯度

猜你喜欢

梯度及最小二乘估计器

符号(Notations)

2. 梯度

3. Example：线性高斯模型的最小二乘解

4. Examp: 求 f(x) f ( x ) f(\boldsymbol{x})的梯度

猜你喜欢

4. Examp: 求 $f(\boldsymbol{x})$ 的梯度