Hair rendering system based on deep neural network
A deep neural network based hair rendering system is presented to model high frequency component of furry objects. Compared with existing approaches, the present method can generate photo-realistic rendering results. An acceleration method is applied in our framework, which can speed up training and rendering processes. In addition, a patch-based training scheme is introduced, which significantly increases the quality of outputs and preserves high frequency details.
1. A method of rendering an object, comprising:
capturing a plurality of images of the object from a plurality of views;
obtaining an alpha mask for each of the plurality of images;
obtaining a 3D proxy geometry through a structure from a silhouette algorithm;
rendering a depth map for each of the plurality of images using a 3D proxy;
training a neural radiance field-based (NeRF-based) deep neural network with images with depth; and
rendering a plurality of images for a plurality of new views.
2. The method according to claim 1 , further comprising:
training the NeRF-based deep neural network with pixels in an area of the image with depth.
3. The method according to claim 2 , wherein a near bound t n and a far bound t r of the area is calculated by:
t n =d s −d f *d r ; t f =d s +d f (1− d r ),
wherein d s is the depth of the image, d f is a sampling range, and d r is a ratio of sampling numbers.
4. The method according to claim 3 , wherein d f and d r are set based on a nearest depth and farthest depth rendered using the 3D proxy.
5. The method according to claim 2 , wherein the pixels in the images with depth used in training comprises no more than 10% of the pixels in the plurality of images.
6. The method according to claim 1 , further comprising:
dividing the images with depth into a plurality of patches; and
training NeRF-based deep neural network with the images with depth in the patch with a perceptual loss function for supervision.
7. The method of according to claim 1 , further comprising:
training the NeRF-based deep neural network using a red-green-blue (RGB) supervision and an alpha supervision using a loss function.
8. The method of according to claim 1 , wherein the NeRF-based deep neural network comprises a multi-layer UNet convolutional network.
9. The method according to claim 1 , wherein the NeRF-based deep neural network comprises a generative adversarial network, and the method further comprises:
training the generative adversarial network using a RGB supervision using a loss function.
10. The method according to claim 9 , further comprising:
training the generative adversarial network using the RGB supervision and a pseudo-supervision using the loss function.
11. The method according to claim 1 , wherein the object comprises hair.
12. A device for rendering an object, comprising:
a processor; and
a memory configured with computer instructions executable by the processor, wherein, upon being executed by the processor, the computer instructions cause the processor to perform operations, comprising:
obtaining an alpha mask for each of a plurality of images captured of the object from a plurality of views;
obtaining a 3D proxy geometry through a structure from a silhouette algorithm;
rendering a depth map for each of the plurality of images using a 3D proxy;
training a NeRF-based deep neural network with images with depth; and
rendering a plurality of images for a plurality of new views.
13. The device according to claim 12 , wherein the operation of training the NeRF-based deep neural network with the images with depth comprises:
training the NeRF-based deep neural network with pixels in an area of the image with depth.
14. The device according to claim 13 , wherein a near bound t n and a far bound t f of the area is calculated by:
t n =d s −d f *d r ; t f =d s +d f (1− d r ),
wherein d s is the depth of the image, d f is a sampling range, and d r is a ratio of sampling numbers.
15. The device according to claim 14 , wherein d f and d r are set based on a nearest depth and farthest depth rendered using the 3D proxy.
16. The device according to claim 13 , wherein the pixels in the images with depth used in training comprises no more than 10% of the pixels in the plurality of images.
17. The device according to claim 12 , wherein the operation of training the NeRF-based deep neural network with the images with depth comprises:
dividing the images with depth into a plurality of patches; and
training the NeRF-based deep neural network with the images with depth in the patch with a perceptual loss function for supervision.
18. The device according to claim 12 , wherein training the NeRF-based deep neural network with the images with depth comprises:
training the NeRF-based deep neural network using a RGB supervision and an alpha supervision using a loss function.
19. The device of according to claim 12 , wherein the NeRF-based deep neural network comprises a multi-layer UNet convolutional network.
20. The device of according to claim 12 , wherein the NeRF-based deep neural network comprises a generative adversarial network, and the operation of training the NeRF-based deep neural network with the images with depth comprises:
training the generative adversarial network using a RGB supervision using a loss function.
21. The device according to claim 20 , wherein the operation of training the NeRF-based deep neural network with the images with depth comprises:
training the generative adversarial network using the RGB supervision and a pseudo-supervision using the loss function.
22. The device according to claim 12 , wherein the object comprises hair.