NeRF has made great progress in the synthesis of new perspectives, ECCV2020 best paper, but the fly in the ointment is that the reasoning speed is very slow, which seriously restricts its application.
In order to render a pixel in the image, NeRF projects a ray through the pixel from the center of the camera, samples hundreds of points on this ray, calculates the neural radiation field of each point, and uses volume rendering technology to render the ray on this ray The neural radiation field is accumulated for the color of this pixel. To render a picture with a resolution of 800×800, NeRF often takes half a minute to several minutes.
In order to improve the rendering speed of NeRF, NSVF proposes to use a sparse voxel field to organize the scene, allowing NeRF to sample and perform volume rendering only in places that are not empty in the scene. The rendering speed of NSVF is 10 times that of NeRF, but it still cannot meet the real-time requirements. Works such as PlenOctree, FastNeRF, BakingNeRF, and NeX propose to pre-calculate and cache the neural radiation field in the scene, so as to accelerate rendering. Since the radiation field is different from each angle, caching the radiation field with a 360-degree viewing angle requires a huge storage space. Therefore, these works use techniques such as spherical harmonics to decouple the radiation field and the effect of depending on the viewing angle, and only cache does not depend on the viewing angle. radiation field. But these methods require a lot of additional storage space for each scene. DONeRF tries to use a multi-layer perceptron to estimate the depth of pixels, and then sample a small number of points near the depth to achieve acceleration, but this method requires the supervision of depth images.
For students who are not familiar with NeRF, it is recommended to learn NeRF-based 3D content generation first
4. MobileNeRF training code mobilenerf
