Neural opacity point cloud

1. A method of rendering an object, the method comprising:

encoding a feature vector to each point in a point cloud for an object, wherein the feature vector comprises an alpha matte;

projecting each point in the point cloud and the corresponding feature vector to a target view to compute a feature map; and

using a neural rendering network to decode the feature map into a RGB image and the alpha matte and to update the feature vector.

2. The method of claim 1 , further comprising:

rasterizing a point in the point cloud into a patch on the feature map, wherein a size of the patch is determined by a depth value of the point.

3. The method of claim 1 , further comprising:

associating each point in the point cloud with a set of pixels on the feature map.

4. The method of claim 1 , further comprising:

back-propagating a gradient of a loss function in the feature map to a gradient on the point cloud.

5. The method of claim 1 , wherein the neural rendering network comprises a first encoder and a first decoder for processing the RGB image, and a second encoder and a second decoder for processing the alpha matte.

6. The method of claim 5 , wherein the first encoder adopts U-Net architecture while replacing a convolutional layer in U-Net architecture with a gated convolution layer.

7. The method of claim 6 , wherein the first encoder comprises one input convolution block and four down-sampling blocks.

8. The method of claim 6 , wherein the first decoder comprises one output convolution block and four upsampling blocks.

9. The method of claim 5 , further comprising:

obtaining an input to each upsampling block of the first decoder by concatenating intermediate feature maps from both the first encoder and a previous upsampling block.

10. The method of claim 5 , wherein the second encoder comprises one input convolution block and two down-sampling blocks.

11. The method of claim 10 , wherein the second decoder comprises one output convolution block and two upsampling blocks.

12. The method of claim 10 , further comprising:

obtaining an input to each upsampling block of the second decoder by concatenating intermediate feature maps from both the first encoder and the second encoder.

13. The method of claim 1 , further comprising:

capturing a plurality of images of the object by a plurality of cameras from a plurality of viewpoints; and

constructing the point cloud using an image-based visual hull construction.

14. The method of claim 13 , further comprising:

rotating the object at a constant speed; and

capturing the plurality of images of the object at a fixed interval by the plurality of cameras from a plurality of viewpoints.

15. The method of claim 14 , further comprising:

capturing images of a calibration pattern rotating at the constant speed by a calibration camera at the fixed interval.

16. The method of claim 13 , further comprising:

calibrating the plurality of images of the object by calculating an extrinsic parameter of the calibration camera via a structure-from-motion (SFM) technique.

17. The method of claim 13 , further comprising:

generating the alpha matte for each image by obtaining a foreground binary mask and refining the foreground binary mask by a deep neural network.

18. The method of claim 17 , further comprising:

generating a trimap from the foreground binary mask.

19. The method of claim 17 , wherein the deep neural network comprises two parallel encoder branches for extracting local feature and global contextual information.

20. The method of claim 17 , further comprising:

constructing the point cloud of the object from the foreground binary mask using the image-based visual hull construction.

21. The method of claim 17 , further comprising:

training the deep neural network using the images and the corresponding alpha mattes.

22. The method of claim 21 , wherein one of the plurality of the cameras is set as a target camera.

23. The method of claim 21 , further comprising:

back-propagating a gradient of a loss function of the images to the point cloud.

24. The method of claim 21 , further comprising:

performing two-dimensional image transformation to the images.

25. The method of claim 1 , wherein the object comprises hair.