Point cloud attribute compression with point cloud geometry decoupling
In one implementation, a method for reconstructing attributes of a point cloud is provided wherein a geometry feature map representative of a geometry of the point cloud is obtained, an attribute feature map is decoded from a bitstream, the attribute feature map being representative of attributes of points of the point cloud, and the decoded attribute feature map is updated with the geometry feature map. Attributes of points of the point cloud are reconstructed based on the updated decoded attribute feature map.
1 . A method for reconstructing attributes of a point cloud, the method comprising:
obtaining a geometry feature map representative of a geometry of the point cloud;
decoding an attribute feature map from a bitstream, providing a decoded attribute feature map, the attribute feature map being representative of attributes of points of the point cloud;
updating the decoded attribute feature map with the geometry feature map by providing the geometry feature map and the decoded attribute map to a neural network-based conditional feature decoder,
wherein the neural network-based conditional feature decoder comprises:
a concatenation layer that concatenates the decoded attribute feature map with the geometry feature map providing a concatenated feature map, and
one or more convolutional layers that mix features of the concatenated feature map to output an updated decoded attribute feature map; and
reconstructing attributes of points of the point cloud based on the updated decoded attribute feature map.
2 . The method of claim 1 ,
wherein the point cloud is represented with an octree structure, and
wherein the attribute feature map represents voxel attributes at a current level in the octree structure of the point cloud.
3 . The method of claim 2 , wherein a feature in the attribute feature map for a current voxel is representative of at least a set of voxels that are still to be reconstructed.
4 . The method of claim 2 , wherein reconstructing attributes of points reconstructs voxel attributes at the current level in the octree structure and comprises:
determining an attribute probability of a current voxel based on the updated decoded attribute feature map, and
decoding attribute information for the current voxel based on the attribute probability determined for the current voxel.
5 . The method of claim 2 , wherein reconstructing attributes of points reconstructs voxel attributes at the current level in the octree structure and comprises:
obtaining an attribute prediction for a current voxel from reconstructed voxel attributes at a previous level of the octree structure, and
reconstructing an attribute value for the current voxel based on the attribute prediction obtained for the current voxel and the updated decoded attribute feature map.
6 . The method of claim 1 , wherein the decoded attribute feature map is updated with the geometry feature map and a predicted feature map obtained from a reference point cloud frame.
7 . The method of claim 1 , wherein the geometry feature map is obtained as an output of a feature aggregator neural network that processes a decoded point cloud without attributes.
8 . An apparatus comprising one or more processors, coupled to a memory, the apparatus being configured to:
obtain a geometry feature map representative of a geometry of a point cloud;
decode an attribute feature map from a bitstream, providing a decoded attribute feature map, the attribute feature map being representative of attributes of points of the point cloud;
update the decoded attribute feature map with the geometry feature map by providing the geometry feature map and the decoded attribute map to a neural network-based conditional feature decoder,
wherein the neural network-based conditional feature decoder comprises:
a concatenation layer that concatenates the decoded attribute feature map with the geometry feature map providing a concatenated feature map, and
one or more convolutional layers that mix features of the concatenated feature map to output an updated decoded attribute feature map; and
reconstruct attributes of points of the point cloud based on the updated decoded attribute feature map.
9 . The apparatus of claim 8 , wherein the geometry feature map is obtained as an output of a feature aggregator neural network that processes a decoded point cloud without attributes.
10 . The apparatus of claim 8 ,
wherein the point cloud is represented with an octree structure, and
wherein the attribute feature map represents voxel attributes at a current level in the octree structure of the point cloud.
11 . The apparatus of claim 10 , wherein a feature in the attribute feature map for a current voxel is representative of at least a set of voxels that are still to be reconstructed.
12 . The apparatus of claim 10 , wherein reconstructing attributes of points reconstructs voxel attributes at the current level in the octree structure and comprises:
determining an attribute probability of a current voxel based on the updated decoded attribute feature map, and
decoding attribute information for the current voxel based on the attribute probability determined for the current voxel.
13 . The apparatus of claim 8 , wherein the decoded attribute feature map is updated with the geometry feature map and a predicted feature map obtained from a reference point cloud frame.
14 . A method for encoding attributes of a current point cloud, the method comprising:
obtaining a geometry feature map representative of a geometry of the point cloud;
obtaining an attribute feature map, the attribute feature map being representative of attributes of points of the point cloud;
refining the attribute feature map with the geometry feature map by providing the geometry feature map and the attribute feature map to a neural network-based conditional feature encoder, providing a refined attribute feature map;
encoding the refined attribute feature map;
obtaining an updated attribute feature map from a decoded version of the refined attribute feature map and the geometry feature map by providing the geometry feature map and the decoded version of the refined attribute feature map to a neural network-based conditional feature decoder,
wherein the neural network-based conditional feature decoder comprises:
a concatenation layer that concatenates the decoded version of the refined attribute feature map with the geometry feature map providing a concatenated feature map, and
one or more convolutional layers that mix features of the concatenated feature map to output the updated decoded attribute feature map; and
encoding attributes of points of the point cloud based on the updated attribute feature map.
15 . The method of claim 14 ,
wherein the point cloud is represented with an octree structure, and
wherein the attribute feature map represents voxel attributes at a current level in the octree structure of the point cloud.
16 . The method of claim 15 , wherein a feature in the attribute feature map for a current voxel is representative of at least a set of voxels that are still to be reconstructed.
17 . The method of claim 15 , wherein encoding attributes of points of the point cloud encodes voxel attributes at the current level in the octree structure and comprises:
determining an attribute probability of a current voxel based on the updated attribute feature map, and
encoding attribute information for the current voxel based on the attribute probability determined for the current voxel.
18 . An apparatus comprising one or more processors, coupled to a memory, the apparatus being configured to:
obtain a geometry feature map representative of a geometry of a point cloud;
obtain an attribute feature map, the attribute feature map being representative of attributes of points of the point cloud;
refine the attribute feature map with the geometry feature map by providing the geometry feature map and the attribute feature map to a neural network-based conditional feature encoder, providing a refined attribute feature map;
encode the refined attribute feature map;
obtain an updated attribute feature map from a decoded version of the refined attribute feature map and the geometry feature map by providing the geometry feature map and the decoded version of the refined attribute feature map to a neural network-based conditional feature decoder,
wherein the neural network-based conditional feature decoder comprises:
a concatenation layer that concatenates the decoded version of the refined attribute feature map with the geometry feature map providing a concatenated feature map, and
one or more convolutional layers that mix features of the concatenated feature map to output the updated decoded attribute feature map; and
encode attributes of points of the point cloud based on the updated attribute feature map.
19 . The apparatus of claim 18 ,
wherein the point cloud is represented with an octree structure, and
wherein the attribute feature map represents voxel attributes at a current level in the octree structure of the point cloud.
20 . The apparatus of claim 18 , wherein the geometry feature map is obtained as an output of a feature aggregator neural network that processes a decoded point cloud without attributes.