Method for rendering relighted 3D portrait of person and computing device for the same
The disclosure provides a method for generating relightable 3D portrait using a deep neural network and a computing device implementing the method. A possibility of obtaining, in real time and on computing devices having limited processing resources, realistically relighted 3D portraits having quality higher or at least comparable to quality achieved by prior art solutions, but without utilizing complex and costly equipment is provided. A method for rendering a relighted 3D portrait of a person, the method including: receiving an input defining a camera viewpoint and lighting conditions, rasterizing latent descriptors of a 3D point cloud at different resolutions based on the camera viewpoint to obtain rasterized images, wherein the 3D point cloud is generated based on a sequence of images captured by a camera with a blinking flash while moving the camera at least partly around an upper body, the sequence of images comprising a set of flash images and a set of no-flash images, processing the rasterized images with a deep neural network to predict albedo, normals, environmental shadow maps, and segmentation mask for the received camera viewpoint, and fusing the predicted albedo, normals, environmental shadow maps, and segmentation mask into the relighted 3D portrait based on the lighting conditions.
1. A method for rendering a relighted 3D portrait, the method comprising:
capturing a sequence of frames by a camera with a blinking flash while moving the camera around a person;
generating a 3D point cloud based on the sequence of frames;
receiving an input defining a camera viewpoint and lighting conditions;
obtaining feature information regarding the 3D point cloud for the camera viewpoint;
obtaining, via a neural network, at least one 2D image for the camera viewpoint based on the feature information; and
fusing the at least one 2D image into the relighted 3D portrait according to the lighting conditions.
2. The method of claim 1 , wherein the at least one 2D image includes albedo map, normal map, environmental lighting map, and segmentation mask.
3. The method of claim 2 , wherein the obtaining feature information comprises obtaining rasterized images based on the feature information, and
wherein the obtaining the at least one 2D image comprises processing the rasterized images with the neural network to the albedo map, the normal map, the environmental lighting map, and the segmentation mask for the camera viewpoint.
4. The method of claim 3 , wherein the 3D point cloud is generated based on the sequence of images captured by the camera while moving the camera around an upper body of the person.
5. The method of claim 4 , wherein the sequence of images comprising a set of flash images and a set of no-flash images.
6. The method of claim 1 , wherein the generating the 3D point cloud further comprises estimating camera viewpoints with which the sequence of images is captured.
7. The method of claim 6 , wherein the received camera viewpoint and the lighting conditions differ from camera viewpoints and lighting conditions with which the sequence of images is captured.
8. The method of claim 6 , wherein the generating the 3D point cloud and the estimating camera viewpoints are performed using Structure-from-Motion (SfM), wherein the 3D point cloud comprises points corresponding to an upper body.
9. The method of claim 1 , wherein the generating the 3D point cloud further comprises:
processing each image of the sequence by segmenting a foreground; and
filtering the 3D point cloud based on the segmented foreground by a segmentation neural network to obtain the filtered 3D point cloud.
10. The method of claim 1 , wherein the obtaining feature information is performed using Z-buffering.
11. The method of claim 5 , wherein flash images of the set of flash images are alternated with no-flash images of the set of no-flash images in said sequence of images.
12. The method of claim 1 , wherein the obtaining feature information comprises rasterizing latent descriptors of the 3D point cloud at different resolutions according to the camera viewpoint, and
wherein the latent descriptor includes a multi-dimensional latent vector characterizing properties of a corresponding point in the 3D point cloud.
13. The method of claim 1 , wherein the method further comprises training the neural network, and
wherein the training the neural network comprises:
randomly sampling an image from the captured sequence of images, the camera viewpoint corresponding to the image, and
obtaining a predicted image by the neural network for the camera viewpoint, wherein the training stage is carried out iteratively.
14. The method of claim 13 , wherein the neural network is trained by backpropagation of a loss to weights of the neural network, latent descriptors, and auxiliary parameters,
wherein the loss is calculated based on one or more of: main loss, segmentation loss, room shading loss, symmetry loss, albedo color matching loss, normal loss.
15. The method of claim 14 , wherein the auxiliary parameters include one or more of room lighting color temperature, flashlight color temperature, and albedo half-texture.
16. The method of claim 15 , further comprising predicting auxiliary face meshes with corresponding texture mapping by a 3D face mesh reconstruction network for each of the images in the captured sequence,
wherein the corresponding texture mapping includes a specification of two-dimensional coordinates in the fixed, predefined texture space for every vertex of the mesh.
17. The method of claim 14 , wherein the main loss is calculated as a mismatch between predicted image and the sampled image by a combination of non-perceptual and perceptual loss functions.
18. The method of claim 14 , wherein the segmentation loss is calculated as a mismatch between a predicted segmentation mask and a segmented foreground of an image of the sequence of frames.
19. The method of claim 14 , wherein the room shading loss is calculated as a penalty for a sharpness of predicted environmental shadow maps, wherein the penalty increases as the sharpness increases.
20. A computer-readable non-transitory storage medium having stored therein instructions that, when executed, cause the electronic device to perform the method of claim 1 .