Paper notes NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

NeRF uses neural networks to represent scenes. Given a scene, input pictures from some perspectives of the scene, and NeRF can synthesize pictures from new perspectives of the scene.

neural radiation field

Neural radiance field (NeRF) uses 5D vector-valued functions to represent a scene.
The input is continuous 5D coordinates (including position $\mathbf{x}=(x,y,z)$ and viewing angle direction$\mathbf{d}=(i,\varphi )$ ), the output is the luminous color$\mathbf{c}=(r,g,b)$ and volume density$\sigma$ .
Specifically, use a fully connected network to approximate this scenario, that is, learn$F_{}:(\mathbf{x},\mathbf{d})\to (\mathbf{c},\sigma )$ .
The authors encourage making bulk density dependent only on location. So the network structure is to first input the position$x$ , output$\sigma$ and an eigenvector. Then the feature vector and the viewing angle direction are spliced, and finally mapped to$c$ color.
Note that one NeRF needs to be trained for each scenario.

position encoding

Using high-frequency functions to map the input into a higher-dimensional space before passing it to the network can provide a better fit to data containing high-frequency variation. Therefore, similar to Transformer, the author proposes to transform $\mathbf{x},d$ is mapped into a high-dimensional space using trigonometric functions, and then input into the network:

$p$ is the value of each coordinate, such as$xneutralx$ , y , zx,y,z$x,y,z$。

Stereoscopic rendering using radiant fields

In order to match the radiation field, the author uses volume rendering to render the image.
For stereoscopic rendering, please refer to https://zhuanlan.zhihu.com/p/595117334
Volume density $\sigma \left(x\right)$ can be interpreted as the light at positionx \mathbf xThe differential probability of ending at infinitesimal particle at$x\; .$
In stereoscopic rendering, camera ray$\mathbf{r}(t)=\mathbf{o}+td$ is in the range$[t_n,t_f]$ Desired color$C(\mathbf{r})$如下计算：
$$\begin{gathered} C(\mathbf{r})={\int }_{t_n}^{t_f}T(t)\sigma (\mathbf{r}(t))\mathbf{c}(\mathbf{r}(t),\mathbf{d})dt \\ whereT(t)=\exp (-{\int }_{t_n}^t\sigma (r(s))ds) \end{gathered}$$ Rendering a view from a continuous neural radiation field requires tracing every pixel on the camera ray of the desired virtual camera to estimate the integral$C(\mathbf{r})$。

The integral of the above formula is actually calculated numerically, which requires sampling and summation. However, if sampling is fixed at certain points, the resolution of the representation will be limited. In order to solve this problem, the author proposes to use stratified sampling method. First, $[t_n,t_f]$ Divide it equally into N buckets of the same size, and then randomly sample a sample in each bucket:
$$t_i\sim \mathcal{U}[t_n+\frac{i-1}{N}(t_f-t_n),t_n+\frac{i}{N}(t_f-t_n)]$$ Although the sampled samples are still discrete, the optimization process is cyclic and requires multiple samplings. Each sampling can sample to a different position, so it is equivalent to optimizing at continuous positions. Estimate C ( r ) C(\mathbf r)using the sampled samples$C(\mathbf{r})$的方法如下：
$$\begin{gathered} \hat{C}(\mathbf{r})=\sum _i^N{T}_i(1-\exp (-p_i{d}_i)){\mathbf{c}}_i \\ where{T}_i=\exp (-\sum _{j=1}^{i-1}{p}_j{d}_j)\text{\hspace{1em}(1)} \end{gathered}$$其中$d_i=t_{i+1}-t_i$. Additionally, estimate $The\; method\; of\; C\left(r\right)$ is differentiable, so the parameters can be easily optimized.

Hierarchical stereo sampling

If the NeRF values ​​are computed densely at N query points along each camera ray, such a rendering strategy is inefficient because free space and occlusion areas that do not contribute to the rendered image will be oversampled.
In order to solve this problem, the author proposed to train two networks, one is coarse-grained (coarse) and the other is fine-grained (fine). First, the coarse-grained network is hierarchically sampled $N_c$points, and then calculate ${\hat{C}}_c\left(r\right)$ .
Standardization${\hat{w}}_i=\frac{w_i}{{\sum }_j{w}_j}$, the probability density function of a distribution can be obtained. According to this distribution, use inverse transform sampling to obtain $N_f$point. This $N_f$Points are obtained based on their importance to the rendering.
Then, use a fine-grained network to calculate this $N_c+N_f$value of points, use formula (1) to calculate the final light color ${\hat{C}}_f\left(r\right)$ .
This Hierarchical method can sample more points in the visible part.

train

Each scenario requires a separate NeRF training. In addition to a set of RGB images of the scene, the input also includes its corresponding camera position, camera internal parameters, and scene boundaries (which can be estimated using a structure-from-motion library, such as COLMAP).
The trained loss function is the squared error between the rendered pixels and the real pixels:

where $R$ is a batch collection of rays. In each loop of training, a batch of camera rays is randomly sampled from pixels.
Although the final rendered image is generated by the fine-grained network, the coarse-grained network also needs to be trained because the coarse-grained network provides the location of the sampling points.