矩阵微分

矩阵微分

标签（空格分隔）： 矩阵微分

---

在学习机器学习算法时,发现对矩阵求导很不熟悉,去看了张贤达的矩阵分析,发现标量对矩阵求导的问题说的很清楚.关于如何求解 hessian矩阵,日后再来补上.
重要的事情说三遍
损失函数是标量函数!!!
损失函数是标量函数!!!
损失函数是标量函数!!!

主要介绍实值函数相对于实向量变量或者矩阵变量的偏导.这里首先对变元和函数符号做统一的规定以便后面介绍.
$\pmb x = [x_1, ..., x_m]^T \in R^m$为实向量变元
$\pmb X = [\pmb x_1, ..., \pmb x_m]^T \in R^{m \times n}$为矩阵变元
$f(\pmb x) \in R 为实值标量函数,其变元\pmb x \in R^m,记做f:R^{m} \to R$
$f(\pmb X) \in R 为实值标量函数,其变元\pmb X \in R^{m \times n},记做f:R^{m \times n} \to R$
$\pmb f(\pmb x) \in R^p 为p维实列向量函数,其变元\pmb x \in R^m,记做f:R^{m} \to R^p$
$\pmb f(\pmb X) \in R^p 为p维实列向量函数,其变元\pmb X \in R^{m \times n},记做f:R^{m \times n} \to R^p$
$\pmb F(\pmb x) \in R^{p \times q} 为p \times q 实矩阵函数,其变元\pmb x \in R^m,记做f:R^{m} \to R^{p \times q}$
$\pmb F(\pmb X) \in R^{p \times q} 为p \times q 实矩阵函数,其变元\pmb X \in R^{m \times n},记做f:R^{m \times n} \to R^{p \times q}$

Jacobian 矩阵

采用$1 \times m$行向量作为偏导算子,记为

$$D_{\pmb x} \overset{def}{=} [ \cfrac {\partial}{\partial x_1}, ..., \cfrac {\partial }{\partial x_m}]$$

实值标量函数$f(\pmb x) 在 \pmb x 的偏导向量为 1\times m 行向量,定义如下$

$$D_{\pmb x} f(\pmb x ) = \cfrac {\partial f(\pmb x)}{\partial \pmb x ^T} = [ \cfrac {\partial f(\pmb x)}{\partial x_1}, ..., \cfrac {\partial f(\pmb x)}{\partial x_m}]$$

当实值标量函数$f(\pmb X)的变元是p \times q$维矩阵的时候,他有两种定义:Jacobian矩阵和行向量偏导.他的Jacobian矩阵定义为如下

$$D_{\pmb X} f(\pmb X) = \cfrac {\partial f(\pmb X)}{\partial \pmb X^T}$$ 而他的行向量偏导定义为 $$D_{vec \pmb X } f(\pmb X) = \cfrac {\partial f(\pmb X)}{\partial vec(\pmb X)^T}= [ \cfrac {\partial f(\pmb x)}{\partial x_1}, ...\cfrac {\partial f(\pmb x)}{\partial x_{m1}},..., \cfrac {\partial f(\pmb x)}{\partial x_{1n}},...,\cfrac {\partial f(\pmb x)}{\partial x_{mn}}]$$ 这里需要注意的是,实值标量函数 $f(\pmb X)$的Jacobian矩阵的转置 $D_{\pmb X}^Tf(\pmb X)$的列向量化后即为他的行向量偏导 $D_{vec \pmb X}f(\pmb X)$,这是后面介绍Jacobian矩阵相关知识的基础
$当\pmb F(\pmb X)为p \times q 实矩阵函数时,定义他的Jacobian矩阵如下$
$$D_{\pmb X} \pmb F(\pmb X)\overset{def}{=} \cfrac {\partial vec(\pmb F(\pmb X))}{\partial (vec \pmb X)^T}$$

梯度矩阵

采用列向量形式的偏导算子称为列向量偏导算子,也称为梯度算子
采用$1 \times m$向量作为偏导算子,记为

$$\nabla _{\pmb x} \overset{def}{=} [ \cfrac {\partial}{\partial x_1}, ..., \cfrac {\partial }{\partial x_m}]^T$$

实值标量函数$f(\pmb x) 在 \pmb x 的梯度向量为 m\times 1 列向量,定义如下$

$$\nabla _{\pmb x} f(\pmb x) = [ \cfrac {\partial f(\pmb x)}{\partial x_1}, ..., \cfrac {\partial f(\pmb x)}{\partial x_m}]^T$$

实值标量函数$f(\pmb X) 的变元 \pmb X 列向量化后,可以定义其矩阵变元\pmb X 的梯度向量为$

$$\nabla _{vec \pmb X} f(\pmb X) = \cfrac {\partial f(\pmb X)}{\partial vec(\pmb X)}= [ \cfrac {\partial f(\pmb x)}{\partial x_1}, ...\cfrac {\partial f(\pmb x)}{\partial x_{m1}},..., \cfrac {\partial f(\pmb x)}{\partial x_{1n}},...,\cfrac {\partial f(\pmb x)}{\partial x_{mn}}]^T$$
定义 $f(\pmb X) 的其关于矩阵变元\pmb X 的梯度矩阵为$
$$\nabla _{\pmb X} f(\pmb X) = \cfrac {\partial f(\pmb X)}{\partial \pmb X}$$ 比较 $f(\pmb X)$的梯度矩阵和Jacobian矩阵,可以发现梯度矩阵即为Jacobian矩阵的转置
当实值标量函数数 $f(\pmb X)的变元是p \times q维矩阵的时候,他的梯度向量定义为如下$
$$\nabla_{\pmb X} \pmb F(\pmb X)\overset{def}{=} \cfrac {\partial vec(\pmb F(\pmb X))}{\partial (vec \pmb X)^T}$$

标量函数$f(\pmb x)$与Jacobian矩阵

以向量为变元的标量函数$f(\pmb x)$的全微分形式可以写为

$$df(\pmb x) = \cfrac {\partial f(\pmb x)}{\partial x_1} dx_1 + ... + \cfrac {\partial f(\pmb x)}{\partial x_m} dx_m = \cfrac {\partial f(\pmb x)}{\partial \pmb x ^T} d \pmb x$$
$记\pmb A = \cfrac {\partial f(\pmb x)}{\partial \pmb x ^T}$,则有如下等价关系
$$df(\pmb x) = tr(Ad \pmb x) \iff D_{\pmb x} f(\pmb x ) = \cfrac {\partial f(\pmb x)}{\partial \pmb x ^T} = A$$
也就是标量函数 $f(\pmb x)$的Jacobian矩阵和微分矩阵存在着等价关系

标量函数$f(\pmb X)$与Jacobian矩阵

标量函数$f(\pmb X)$和上面类似,其全微分可以写成如下形式

$$\begin{align*} df(\pmb X) & = \cfrac {\partial f(\pmb X)}{\partial \pmb x_1^T} d \pmb x_1 + ... + \cfrac {\partial f(\pmb X)}{\partial \pmb x_n^T} d \pmb x_n \\ &= \cfrac {\partial f(\pmb X)}{\partial vec^T \pmb (X)} d (vec \pmb X) \\ &= D_{vec \pmb X} f(\pmb X) d(vec \pmb X) \end{align*}$$
这里再利用行向量偏导和Jacobian矩阵的关系 $D_{vec \pmb X} f(\pmb X) = (vec (D_{\pmb X}^T f(\pmb X)))^T，并令A=D_{\pmb X}^T f(\pmb X)$可以得到
$$\begin{align*} df(\pmb X) & = (vec(A^T))^T d(vec \pmb X) \end{align*}$$
由向量化算子vec与迹函数的关系式 $tr(B^TC) = (vec(B))^Tvec(C),令B = A^T, C = d \pmb X$,则上式可以重写为
$$df(\pmb X) = tr(\pmb A d \pmb X)$$

综合以上,可以得到如下结论:
Jacobian矩阵可以通过以下式子等价确定

$$df(\pmb x) = tr(A d \pmb x) \iff D_{\pmb x} f(\pmb x) = A \\ df(\pmb X) = tr(A d \pmb X) \iff D_{\pmb X} f(\pmb X) = A$$

矩阵微分$df(\pmb X)$可以通过简单的变化转化为矩阵微分的标准形式$df(\pmb X) = tr(A d \pmb X)$.再由Jacobian和梯度矩阵的关系,进一步可以得到梯度矩阵.因而对求解梯度矩阵可以由矩阵微分的标准形式$df(\pmb X) = tr(A d \pmb X)$得到.
如下:
对于$tr(\pmb X^T \pmb X)$我们可以得到

$$\begin{align*} d tr(\pmb X^T \pmb X) & = tr(d(\pmb X^T \pmb X)) \\ &= tr(d(\pmb X )^T \pmb X + \pmb X^T d \pmb X)\\ &= tr(d(\pmb X )^T \pmb X) +tr( \pmb X^T d \pmb X) \\ &= tr(\pmb X ^ T d(\pmb X )) +tr( \pmb X^T d \pmb X) \\ &= tr(2 \pmb X ^ T d(\pmb X )) \end{align*}$$
由以上结论,可以得到 $\pmb X^T \pmb X$关于 $\pmb X$的梯度矩阵为
$$\cfrac{\partial tr(\pmb X^T \pmb X)}{\partial \pmb X} = (2 \pmb X ^T)^T = 2 \pmb X$$
求解梯度矩阵是进行一阶优化算法的基础部分.

参考文献&学习资料
矩阵分析与应用 -张贤达
矩阵求导术-知乎
The Matrix Cookbook.