97、Text2NeRF: Text-Driven 3D Scene Generation with Neural Radiance Fields

Introduction

Paper address

Use a diffusion model to infer text-related images as content priors, and use a monocular depth estimation method to provide geometric priors, and introduce a progressive scene Drawing and updating strategies to ensure texture and geometry consistency between different views

Implementation process

In simple terms:

Text-image diffusion model generates an initial image$I_0$，将$I_0$Distortion, get multiple pictures of the same z plane, that is, Support set $S_0$, be careful, be careful$S_0$Koreyu$I_0$ is distorted, so there are many gaps, but we can calculate it according to $S_0$Reconstruct the initial NeRF model.

Use the initial NeRF model to render the new perspective picture. This is incomplete, but it can be completed by the diffusion model. Note that in order to maintain the consistency of the scene, the perspective is from $I_0$A small offset next to allows the diffusion model to move as much as possible from $I_0$Get information from it, and then you can update the NeRF model.

Due to the influence of image distortion, image scale gaps and distance gaps will inevitably result (reflected in the differences in the depth of spatial points at different viewing angles). For this purpose, a deep alignment strategy is adopted.

Support Set

采用了 DIBR(Depth-image-based rendering (dibr), compression, and transmission for a new approach on 3d-tv) 方法生成 $S_0$

Specifically:
Obtain initial image from diffusion model $I_0$, and then obtain the depth through the depth prediction network $D_0$，对于 $I-Each\; pixel\; q\; of\; 0$ and its depth z are converted using the following formula to obtain $S_0$。

$K$ Sum $P_i$is the intrinsic matrix and camera pose in view i.

为了在大视野范围内生成3D场景，将相机位置设置在辐射场内部，并使相机向外看，但是该方法不能像其他设置相机查看内部的方法那样生成单独的3D对象。

With current camera position $P_0$as the center, generate a surrounding circle with a radius r, with the same z coordinate, uniformly sample n points as the camera position, and use the same camera direction as the current view to generate a warped view that supports concentration, generally r=0.2, n=8, the offset directions are generally up, down, left, right, up left, down left, up right and down right.

At this point you can begin to reconstruct the initial 3D model.

Text-Driven Inpainting

Except initial view $I_0$Rendering results outside the rendering results will inevitably have missing content. At this time, the text-driven image filling method based on the pre-trained diffusion model can be used.

First, render a new perspective $P_1$ Statue $I_k^R$，通过对率$I_0$Twisted to$P_1$Later image sum$I_k^R$, we get the mask $M_k$. Then it is thrown to the diffusion model, which expands the scene information.

However, the generation quality of the diffusion model is not necessarily very good, so multiple drawing processes are used to evaluate the image encoder of CLIP, compare the gap between the completed image and the initial image, and select optimal. The paper uses 30 candidates.

Depth Alignment

The completed image and the initial image will have depth conflicts in the overlapping parts. It is reflected as:

Scale gap: The distance between the spatial points corresponding to the sofa and the wall in the image should be unique, but there may be differences in different views The spatial points fitted by different views are inconsistent
Distance gap:

The paper globally aligns the two depth maps by compensating for average scale and distance differences.

corresponds to the rendered image and the completed image, expressed as$\{(x_j^R,x_j^E){\}}_{j=1}^M$, calculate the mean scale fraction s and the depth offset δ to approximate the mean scale and distance differences

Scaleed point ${\hat{x}}_j^E=s\cdot x_j^E$, z(x) represents the prediction depth

The global depth is defined here $D_k^{global}=s\cdot D_k^E+\delta$, minimize rendering depth close to global depth

Progressive Inpainting and Updating

In order to ensure the consistency of views during the scene rendering process and avoid the ambiguity of geometry and appearance, a progressive rendering and update strategy is adopted to update the brightness field view by view.

Update the brightness field after each completion. This means that what was previously drawn will be reflected in subsequent renderings, and these parts will be considered known areas and will not be drawn again in other views.

Inspired by (Zeroshot text-guided object generation with dream fields), a depth-sensing transmission loss is designed $L_T$, to prompt the NeRF network to generate null density before the camera ray reaches the expected depth

m(t) is a mask, when t<$\widehat{With}$ when, m(t) = 1, no 为0,$\widehat{With}$This is the depth $\stackrel{The\; pixel-by-pixel\; depth\; value\; in\; ^}{D}$, T (T) is the cumulative transmittance

Effect