Reading notesFirst Order Motion Model for Image Animation

The article solves the problem of picture animation. Assuming that there are source pictures and driving videos, and the objects in them are of the same type, the method in the article makes the objects in the source pictures move according to the actions of the objects in the driving video.
The method in the article only requires a video set of similar objects and does not require additional annotations.

method

This method is based on the self-supervised strategy. The main method is to reconstruct the training video based on a frame of image in the training video and the learned action representation. Among them, the action representation consists of motion-specific keypoints and local affine transformations. Note that because it is a self-supervised method, the key points here are learned by the algorithm, unlike the key points in the face key point detection algorithm, which are artificially designated with specific meanings.

The frame diagram is shown in the figure above and consists of two parts, one is the motion estimation module and the other is the image generation module. The purpose of the motion estimation module is to estimate a frame D ∈ R 3 × H × W
from the driving video \mathbf D \in \mathbb R^{3\times H \times W}$\mathbf{D}\in {\mathbb{R}}^{3\times H\times W}$ to source image$\mathbf{S}\in R^{3\times H\times W}$ dense motion field. Sports field${\mathcal{T}}_{\mathbf{S}\leftarrow \mathbf{D}}:{\mathbb{R}}^2\to R^2$将$Each\; pixel\; position\; in\; D$ is mapped to the corresponding$\mathbf{S}$。$T_{S\leftarrow D}$Also called backward optical flow. Reverse optical flow is used instead of forward optical flow because backward warping can be achieved efficiently in a differentiable manner using bilinear sampling.

Affine transformation

Let’s first recall the radial transformation (Affine transformation).
On homogeneous coordinates, the affine transformation can be expressed by the following formula:
$$\left[\begin{array}{c}\vec{y}\\ 1\end{array}\right]=\left[\begin{array}{cc}\mathbf{B}& \vec{b}\\ 0,\dots ,0& 1\end{array}\right]\left[\begin{array}{c}\vec{x}\\ 1\end{array}\right]$$ Because the last row of the operation matrix is ​​used for operation completion, the affine transformation on the 2-dimensional image is determined by the matrix$\mathbf{A}=[\mathbf{B},\vec{b}]\in {\mathbb{R}}^{2\times 3}$ definition.

motion estimation module

The motion estimation module is divided into two parts.

coarse motion estimation

Coarse motion estimation predicts motion patterns at key points, that is, reverse optical flow ${\mathcal{T}}_{\mathbf{S}\leftarrow \mathbf{D}}$。$T_{S\leftarrow D}$Approximated by first-order Taylor expansion near key points.

Suppose there is an abstract reference frame $R.$ _ In this way, we need to estimate two transformations: from$R$到$\mathbf{S}$（${\mathcal{T}}_{\mathbf{S}\leftarrow \mathbf{R}}$) and from $R$ to$\mathbf{D}$（$T_{\mathbf{D}\leftarrow \mathbf{R}}$). The advantage of abstract reference frames is that they allow us to process D independently $D$和$S.$ _
For convenience of description, use$X$ represents$S$ or$\mathbf{D}$，用$p_1,\cdots ,p_K$Represents an abstract reference frame R \mathbf RThe coordinates of the key points on$R$$z$ represents the coordinates of points on other frames. We estimate that at key points$p_1,\cdots ,p_K$TX around ${\mathcal{T}}_{\mathbf{X}\leftarrow \mathbf{R}}$. Specifically, we consider $T_{X\leftarrow R}$At key points $p_1,\cdots ,p_K$Additionally:
$$T_{X\leftarrow R}(p)=T_{X\leftarrow R}(p_k)+(\frac{d{\mathcal{T}}_{\mathbf{X}\leftarrow \mathbf{R}}(p)}{dp}{|}_{p=p_k})(p-p_k)+o(\Vert p-p_k\Vert )$$ This can be seen as an affine transformation${\mathbf{A}}_{\mathbf{X}\leftarrow \mathbf{R}}^k\in {\mathbb{R}}^{2\times 3}$ ,${\mathcal{T}}_{X\leftarrow R}(p_k)$ is the translation parameter,$\frac{d{T}_{X\leftarrow R}\left(p\right)}{dp}{|}_{p=p_k}$are the parameters of linear mapping.

$T_{X\leftarrow R}$The function K is a free-form Jacobian function.
TX ← R ( p ) ≈ { { TX ← R ( p 1 ) , d TX ← R ( p ) dp ∣ p = p 1 } , ⋯ , { TX ← R ( p K ) , d TX ← R ( p ) dp ∣ p = p K } } \mathcal T_{\mathbf X \leftarrow \mathbf R}(p) \approx \{\{ \mathcal T_{\mathbf X \leftarrow\mathbf R}(p_1),\frac{d\mathcal T_{\mathbf X \leftarrow\mathbf R}(p)}{dp}|_{p=p_1}\}, \cdots,\{ \mathcal T_{\mathbf X \leftarrow \mathbf R}(p_K),\frac{d \mathcal T_{\mathbf X \leftarrow \mathbf R}(p)}{dp}|_{p=p_K}\} \}TX←R​(p)≈{ { TX←R​(p1​),dpdTX←R​(p)​∣p=p1​​},⋯,{ TX←R​(pK​),dpdTX←R​(p)​∣p=pK​​}}
We assume that${\mathcal{T}}_{X\leftarrow R}$The locality at each keypoint is a bijection. Then for $T_{\mathbf{S}\leftarrow \mathbf{D}}$，我们有
$$T_{S\leftarrow D}=T_{\mathbf{S}\leftarrow \mathbf{R}}\circ T_{\mathbf{D}\leftarrow \mathbf{R}}^{-1}$$In the case of the boundary layer
$$\begin{gathered} {\mathcal{T}}_{\mathbf{S}\leftarrow \mathbf{D}}(z)\approx T_{\mathbf{S}\leftarrow \mathbf{R}}(p_k)+J_k(z-T_{\mathbf{D}\leftarrow \mathbf{R}}(p_k)) \\ {J}_k=(\frac{d{\mathcal{T}}_{S\leftarrow R}(p)}{dp}{|}_{p=p_k})(\frac{d{\mathcal{T}}_{\mathbf{D}\leftarrow \mathbf{R}}\left(p\right)}{dp}{|}_{p=p_k}{)}^{-1} \end{gathered}$$ ${\mathcal{T}}_{\mathbf{S}\leftarrow \mathbf{R}}(p_k)$和$T_{D\leftarrow R}\left(p_k\right)$ is predicted using the keypoint predictor networkbased on U-Net. Predict a heatmap for each key point, and predict K heatmaps in total. The last layer of the U-Net decoder uses softmax to predict the confidence map of each key point (keypoint confidence map), which is the confidence of the key point at each pixel position, satisfying ∑ z${\sum }_{z\in \mathcal{Z}}{\mathbf{W}}^{to}\left(of\right)=1$ , where$Z$ represents all pixel positions.
${\mathcal{T}}_{\mathbf{S}\leftarrow \mathbf{R}}(p_k)$和$T_{\mathbf{D}\leftarrow \mathbf{R}}\left(p_k\right)$ is equivalent to the translation parameter in affine transformation. Note that it is two-dimensional (z includes x and y). The translation parameters are calculated weighted by the key point confidence map:
$$b^k=\sum _{z\in \mathcal{Z}}{\mathbf{W}}^k\left(z\right)z$$ $\frac{d{\mathcal{T}}_{\mathbf{S}\leftarrow \mathbf{R}}(p)}{dp}{|}_{p=p_k}$和$\frac{d{T}_{\mathbf{D}\leftarrow \mathbf{R}}\left(p\right)}{dp}{|}_{p=p_k}$Equivalent to the linear transformation part in the affine transformation, they are estimated as the remaining 4 parameters in the affine transformation using the additional 4 channels of the keypoint predictor network, with 4 additional channels for each keypoint. Use $P_{ij}^k\in {\mathbb{R}}^{H\times W}$ represents the estimated value of one of the channels, wherei, j ∈ { 1, 2 } i,j\in\{1,2\}i,j∈{ 1,2 } is the coordinate of the affine transformation. The parameters of the linear transformation are weighted and fused using key point confidence maps:
$${\mathbf{B}}^k[i,j]=\sum _{z\in \mathcal{Z}}{\mathbf{W}}^k(z)P_{ij}^k(z)$$

dense motion estimation

Dense motion estimation predicts the motion pattern T ^ S ← D of each pixel of the entire image \hat{\mathcal T}_{\mathbf S \leftarrow \mathbf D}${\hat{\mathcal{T}}}_{\mathbf{S}\leftarrow \mathbf{D}}$。

We use a convolutional network from $Taylor\; expansion\; TS\; at\; K$ key points$T_{S\leftarrow D}\left(z\right)$ and source image frame$Ssing$ T ^ S ← D \hat{\mathcal T}_{\mathbf S \leftarrow \mathbf D}${\hat{\mathcal{T}}}_{\mathbf{S}\leftarrow \mathbf{D}}$.
Distort the source image frame S using transformations at key points $S$ , you can get$K$ transformed images${\mathbf{S}}^1,\cdots ,S^{K.}$ _ Additionally, consider an additional image$S^0=S$ as background. Calculate heatmap H k ( z ) \mathbf H_k(z)
for each key point${\mathbf{H}}_k\left(z\right)$ Specify the smoothness in the intrinsic motion
$${\mathbf{H}}_k(z)=exp(\frac{({\mathcal{T}}_{\mathbf{D}\leftarrow \mathbf{R}}(p_k)-z{)}^2}{p})-exp\left(\frac{\left(T_{\mathbf{S}\leftarrow \mathbf{R}}\right(p_k)-z{)}^2}{p}\right)$$
将$H_k$和${\mathbf{S}}^0,\cdots ,S^{The\; K}$ splicing input isa dense motion network. dense motion network estimates$K+1$ mask${\mathbf{M}}_k,k=0,\cdots ,K$ indicates which local transformation is used for each position, satisfying${\sum }_{k=0}^K{M}^{to}\left(of\right)=1.$ Let us define the equation in the following way:
$${\hat{\mathcal{T}}}_{\mathbf{S}\leftarrow \mathbf{D}}(z)={\mathbf{M}}_{0}z+\sum _{k=1}^K{M}_k({\mathcal{T}}_{\mathbf{S}\leftarrow \mathbf{R}}(p_k)+J_k(z-{\mathcal{T}}_{\mathbf{D}\leftarrow \mathbf{R}}\left(p_k\right)$$
.
$$\mathbf{O}(z)=M^0(z)z+\sum _{k=1}^K{M}^k(z){\mathbf{A}}_{\mathbf{S}\leftarrow \mathbf{D}}^k\left[\begin{array}{c}z\\ 1\end{array}\right]$$

Image generation module

1. According to the predicted ${\hat{\mathcal{T}}}_{\mathbf{S}\leftarrow \mathbf{D}}$to SSThe feature map of$S$$X\in {\mathbb{R}}^{H^{\prime }\times W^{\text{'}}}$ Use warp operation.
2. InSSWhen there is occlusion in$S$$D^{\text{'}}$ cannot be completely obtained by warp source image, but requires inpaint. So, predict an occlusion map${\hat{\mathcal{O}}}_{S\leftarrow D}\in [0,1{]}^{H^{\prime }\times W^{\prime }}$ , indicating the area of ​​the source image that needs to be inpainted. The occlusion map is predicted by adding a layer after the dense motion network.
The transformed feature map can be expressed as:
$$X^{\prime }={\hat{\mathcal{O}}}_{\mathbf{S}\leftarrow \mathbf{D}}\odot f_w(\xi ,{\hat{\mathcal{T}}}_{S\leftarrow D})$$ $f_w$Represents the back-warping operation. The converted feature map is input to the later layer of the image generation module for processing, and finally an image is generated.

train

The training loss consists of multiple items. The first is reconstruction loss based on perceptual loss. This loss uses the pre-trained VGG-19 network as a feature extractor to compare the feature differences between the reconstructed frames and the real frames driving the video.

In addition, considering that the learning of key points is unlabeled, which will lead to unstable performance, Equivariance constraint is introduced for use in unsupervised key point learning. Assume picture $X$ undergoes a known transformation$T_{\mathbf{X}\leftarrow \mathbf{Y}}$Get $Y$。Equivariance constraint solution:
$${\mathcal{T}}_{\mathbf{X}\leftarrow \mathbf{R}}\equiv T_{\mathbf{X}\leftarrow \mathbf{Y}}\circ T_{\mathbf{Y}\leftarrow \mathbf{R}}$$By performing first-order Taylor expansion on both sides, and using L1 loss to constrain the values ​​and Jacobian at the key points respectively.

References

《First Order Motion Model for Image Animation》
《Motion Representations for Articulated Animation》