介绍矩阵求导法则，以及常用的求导公式、迹函数、行列式求导结论

矩阵求导法则

矩阵求导应该分为标量求导、向量求导、矩阵求导三个方面来介绍，公式繁多，但仔细看看其实是有规律可循的。

标量求导

无论是矩阵、向量对标量求导，或者是标量对矩阵、向量求导，其结论都是一样的：等价于对矩阵（向量）的每个分量求导，并且保持维数不变。

例如，我们可以计算标量对向量求导：

设 $y$ 为一个元素， $x^T = [x_1...x_q]$ 是 $q$ 维行向量，则：

\frac{\partial y}{\partial x^{T}} = [\frac{\partial y}{\partial x_{1}} . . . \frac{\partial y}{\partial x_{q}}]

$\frac{\partial y}{\partial x^T} = [\frac{\partial y}{\partial x_1}...\frac{\partial y}{\partial x_q}]$

向量求导

对于向量求导，我们可以先将向量看做一个标量，然后使用标量求导法则，最后将向量形式化为标量进行。

例如，我们可以计算行向量对列向量求导：

设 $y^T=[y_1...y_n]$ 是 $n$ 维行向量， $x=[x_1,...,x_p]$ 是 $p$ 维列向量，则：

\begin{aligned} \frac{\partial y^{T}}{\partial x} & = & [\frac{\partial y_{1}}{\partial x} . . . \frac{\partial y_{n}}{\partial x}] \\ = & [\begin{matrix} \frac{\partial y_{1}}{\partial x_{1}} & . . . & \frac{\partial y_{n}}{\partial x_{1}} \\ . . . & . . . & . . . \\ \frac{\partial y_{1}}{\partial x_{p}} & . . . & \frac{\partial y_{n}}{\partial x_{p}} \end{matrix}] \end{aligned}

$\begin{align}\frac{\partial y^T}{\partial x} &=& [\frac{\partial y_1}{\partial x}...\frac{\partial y_n}{\partial x}] \nonumber \\ &=&\begin{bmatrix}\frac{\partial y_1}{\partial x_1}& ... &\frac{\partial y_n}{\partial x_1} \\ ... & ... &... \\\frac{\partial y_1}{\partial x_p} &... &\frac{\partial y_n}{\partial x_p} \end{bmatrix} \nonumber\end{align}$

矩阵求导

与向量求导类似，先将矩阵化当做一个标量，再使用标量对矩阵的运算进行。

例如，我们可以计算矩阵对列向量求导：

设 $Y =\begin{bmatrix}y_{11} &... & y_{1n} \\... &... &... \\ y_{m1} & ...&y_{mn} \end{bmatrix}$ 是 $m\times n$ 矩阵， $x=[x_1,...,x_p]$ 是 $p$ 维列向量，则：

\frac{\partial Y}{\partial x} = [\frac{\partial Y}{\partial x_{1}}, . . ., \frac{\partial Y}{\partial x_{p}}]

$\frac{\partial Y}{\partial x} = [\frac{\partial Y}{\partial x_1},...,\frac{\partial Y}{\partial x_p}]$

矩阵微积分

常见求导性质

实值函数相对于实向量的梯度

设 $f(x) = x = [x_1,...,x_n]^T$

\frac{\partial f (x)}{\partial x^{T}} = \frac{\partial x}{\partial x^{T}} = I_{n \times n}

$\frac{\partial f (x)}{\partial x^T} = \frac{\partial x}{\partial x^T} = I_{n\times n}$

\frac{\partial (f (x))^{T}}{\partial x} = \frac{\partial x^{T}}{\partial x} = I_{n \times n}

$\frac{\partial (f (x))^T}{\partial x} = \frac{\partial x^T}{\partial x} = I_{n\times n}$

\frac{\partial f (x)}{\partial x} = \frac{\partial x}{\partial x} = v e c (I_{n \times n})

$\frac{\partial f (x)}{\partial x} = \frac{\partial x}{\partial x} = vec(I_{n\times n})$

\frac{\partial (f (x))^{T}}{\partial x^{T}} = \frac{\partial x^{T}}{\partial x^{T}} = v e c (I_{n \times n})^{T}

$\frac{\partial (f (x))^T}{\partial x^T} = \frac{\partial x^T}{\partial x^T} = vec(I_{n\times n})^T$

其中， $vec$ 表示向量化矩阵，按列将矩阵表示为向量，具体可见Wikipedia。

常见性质

$f(x) = Ax$ ，则

$\frac{\partial f (x)}{\partial x^{T}} = \frac{\partial (A x)}{\partial x^{T}} = A$ $\frac{\partial f (x)}{\partial x^T} = \frac{\partial (Ax)}{\partial x^T} =A$
$f(x) = x^TAx$ ，则

$\frac{\partial f (x)}{\partial x} = \frac{\partial (x^{T} A x)}{\partial x} = A x + A^{T} x$ $\frac{\partial f (x)}{\partial x} = \frac{\partial (x^TAx)}{\partial x} =Ax+A^Tx$
$f(x) = a^Tx$ ，则

$\frac{\partial a^{T} x}{\partial x} = \frac{\partial x^{T} a}{\partial x} = a$ $\frac{\partial a^Tx}{\partial x} = \frac{\partial x^Ta}{\partial x} =a$
$f(x) = x^TAy$ ，则

$\frac{\partial x^{T} A y}{\partial x} = A y$ $\frac{\partial x^TAy}{\partial x} = Ay$

$\frac{\partial x^{T} A y}{\partial A} = x y^{T}$ $\frac{\partial x^TAy}{\partial A} = xy^T$
$d f (X) = t r ((\frac{\partial f (X)}{\partial X})^{T} d X)$ $df(X) = tr((\frac{\partial f(X)}{\partial X})^T d X)$
矩阵微分也满足线性法则、乘积法则。
矩阵的逆的微分

$d (X^{- 1}) = - X^{- 1} (d X) X^{- 1}$ $d(X^{-1}) = -X^{-1}(dX)X^{-1}$

迹函数

迹函数相对于矩阵的梯度

\frac{\partial (t r (Z Z^{T}))}{\partial Z} = \frac{\partial (t r (Z^{T} Z))}{\partial Z} = 2 Z

$\frac{\partial (tr (ZZ^T))}{\partial Z} = \frac{\partial (tr (Z^TZ))}{\partial Z} = 2Z$

矩阵微分算子和迹算子的可交换性

d (t r (X)) = t r (d (X)) = \sum_{i = 1}^{n} d x_{i i}

$d(tr(X)) = tr(d(X)) = \sum\limits_{i=1}^{n} dx_{ii}$

常见性质

$\frac{\partial t r (A)}{\partial A} = I_{n \times n}$ $\frac{\partial tr(A)}{\partial A} = I_{n\times n}$
$\frac{\partial t r (A B)}{\partial A} = B^{T}$ $\frac{\partial tr(AB)}{\partial A} = B^T$
$d (t r (A X B)) = t r (A (d X) B) = t r (B A (d x))$ $d(tr(AXB)) = tr(A(dX)B) = tr(BA(dx))$

$\frac{\partial t r (A X B)}{\partial X} = (B A)^{T} = A^{T} B^{T}$ $\frac{\partial tr(AXB)}{\partial X} = (BA)^T = A^TB^T$
$d (t r (A X^{- 1} B)) = t r (A (d X^{- 1}) B) = - t r (A X^{- 1} (d X) X^{- 1} B) = - t r (X^{- 1} B A X^{- 1} d X)$ $d(tr(AX^{-1}B)) = tr(A(dX^{-1})B) = -tr(AX^{-1}(dX)X^{-1}B) = -tr(X^{-1}BAX^{-1}dX)$

$\frac{\partial t r (A X^{- 1} B)}{\partial X} = - (X^{- 1} B A X^{- 1})^{T} = - X^{- T} A^{T} B^{T} X^{- T}$ $\frac{\partial tr(AX^{-1}B)}{\partial X} = -(X^{-1}BAX^{-1})^T = -X^{-T}A^TB^TX^{-T}$
$\frac{\partial t r (X^{T} X)}{\partial X} = 2 X$ $\frac{\partial tr(X^TX)}{\partial X} = 2X$

行列式

行列式相对于矩阵的梯度

\frac{\partial | Z |}{\partial Z} = | Z | (Z^{- 1})^{T}

$\frac{\partial |Z|}{\partial Z} = |Z|(Z^{-1})^T$

微分形式

d | X | = t r (| X | X^{- 1} d X)

$d|X| = tr(|X| X^{-1} dX)$

常见性质

$\begin{aligned} d | A X B | & = & t r (| A X B | (A X B)^{- 1} d (A X B)) \\ = & t r (| A X B | (A X B)^{- 1} A (d X) B) \\ = & t r (| A X B | B (A X B)^{- 1} A (d X)) \end{aligned}$ $\begin{align} d|AXB| &=& tr(|AXB| (AXB)^{-1}d(AXB)) \nonumber\\&=& tr(|AXB| (AXB)^{-1}A(dX)B) \nonumber\\&=& tr(|AXB| B(AXB)^{-1}A(dX))\nonumber\end{align}$

$\frac{\partial | A X B |}{\partial X} = | A X B | A^{T} (B^{T} X^{T} A^{T})^{- 1} B^{T}$ $\frac{\partial |AXB|}{\partial X} = |AXB|A^T(B^TX^TA^T)^{-1}B^T$
$\frac{\partial | X |}{\partial X} = | X | X^{- T}$ $\frac{\partial |X|}{\partial X} = |X| X^{-T}$
$\frac{\partial | X X^{T} |}{\partial X} = 2 | X X^{T} | (X X^{T})^{- 1} X$ $\frac{\partial |XX^T|}{\partial X} = 2|XX^T| (XX^{T})^{-1}X$

reference

矩阵的导数与迹