Granted Patent B2

US 12,238,272 · App. 17/739,484 · Granted Feb 25, 2025

Inventors: Alexandre Zaghetto (San Jose, CA); Ali Tabatabai (Cupertino, CA); Danillo Graziosi (Flagstaff, AZ)

Assignees: SONY GROUP CORPORATION; SONY CORPORATION OF AMERICA

H04N19/103H04N19/119H04N19/147H04N19/176H04N19/42H04N19/597

Abstract

Claims (89)

1. An electronic device, comprising:

circuitry configured to:

receive a three-dimensional (3D) point cloud geometry;

partition the 3D point cloud geometry into a set of 3D blocks;

determine, for a 3D block of the set of 3D blocks, mode decision information that comprises at least one of:

class information associated with the 3D point cloud geometry,

one or more operational conditions associated with an encoding stage of the 3D point cloud geometry, or

mode-related information associated with one or more 3D blocks of the set of 3D blocks;

select one or more modes for the 3D block of the 3D point cloud geometry from a plurality of modes, based on the mode decision information,

wherein each mode of the plurality of modes corresponds to an encoding function for the 3D block of the point cloud geometry,

the plurality of modes further corresponds to a plurality of density levels,

each density level of the plurality of density levels corresponds to a median of a distribution of local density values associated with 3D points in a 3D block of a calibration point cloud of a plurality of calibration point clouds, and

each of the local density values is a number of neighborhood points within a spherical volume around each 3D point in the 3D block of the calibration point cloud; and

encode the 3D block of the 3D point cloud geometry based on the selected one or more modes.

2. The electronic device according to claim 1 , wherein the class information associated with the 3D point cloud geometry includes at least one of a geometry bit-depth, a density level of the plurality of density levels, or a point distribution associated with the 3D point cloud geometry.

3. The electronic device according to claim 1 , wherein the one or more operational conditions associated with the encoding stage of the 3D point cloud geometry includes a target rate-distortion cost associated with 3D point cloud geometry.

4. The electronic device according to claim 1 , wherein the circuitry is further configured to:

load a table that maps the plurality of modes with different classes associated with the plurality of calibration point clouds and operational conditions associated with an encoding stage of the plurality of calibration point clouds, wherein each calibration point cloud of the plurality of calibration point clouds is different from the received 3D point cloud geometry; and

search the table based on the class information and the one or more operational conditions to select the one or more modes.

5. The electronic device according to claim 4 , wherein the circuitry is further configured to:

partition the calibration point cloud of the plurality of calibration point clouds into a plurality of 3D blocks;

encode the plurality of 3D blocks based on each of the plurality of modes to generate a plurality of encoded 3D blocks;

determine a rate-distortion cost associated with each of the generated plurality of encoded 3D blocks;

determine statistical information that indicates, for each mode of the plurality of modes, a fraction of the plurality of encoded 3D blocks for which the rate-distortion cost is a minimum for the plurality of modes;

determine, from the generated plurality of encoded 3D blocks, a subset of encoded 3D blocks for which the fraction of the plurality of encoded 3D blocks is above a threshold, based on the determined statistical information;

determine, from the plurality of modes, a subset of modes that is used in the generation of the subset of encoded 3D blocks; and

generate the table based on the determined subset of modes, the different classes, and the operational conditions, wherein the table is generated prior to the encode of the 3D block of the 3D point cloud geometry.

6. The electronic device according to claim 1 , wherein the circuitry is further configured to:

encode the 3D block of the 3D point cloud geometry based on each of the selected one or more modes to determine one or more encoded 3D blocks;

determine rate-distortion costs associated with the determined one or more encoded 3D blocks;

determine a mode of the selected one or more modes as an optimal mode for the encoding stage, based on a determination that a rate-distortion cost associated with the mode corresponds to a minimum of the determined rate-distortion costs; and

encode the 3D block of the 3D point cloud geometry based on the determined mode to generate an encoded 3D block.

7. The electronic device according to claim 1 , wherein the encoding function corresponds to a Deep Neural Network (DNN) model that is trained to encode the 3D block of the 3D point cloud geometry to generate an encoded 3D block.

8. The electronic device according to claim 7 , wherein each mode of the plurality of modes corresponds to an alpha parameter of a focal loss function used in a training stage of the DNN model, and

the focal loss function is configured to penalize a removal of non-empty voxels from the 3D block of the 3D point cloud geometry.

9. The electronic device according to claim 1 , wherein the circuitry is further configured to:

determine subsets of the set of 3D blocks, based on a scan of the set of 3D blocks in a defined scan order;

encode each 3D block of a first subset of the determined subsets, based on the plurality of modes to generate a plurality of encoded 3D blocks;

determine a rate-distortion cost associated with each encoded 3D block of the plurality of encoded 3D blocks; and

determine mode usage statistics associated with the first subset based on the determined rate-distortion cost associated with each encoded 3D block of the plurality of encoded 3D blocks,

wherein the mode-related information includes the determined mode usage statistics associated with the first subset.

10. The electronic device according to claim 9 , wherein the circuitry is further configured to select the one or more modes for a second subset that includes the 3D block of the 3D point cloud geometry, wherein

the second subset is included in the determined subsets, and

the second subset succeeds the first subset in accordance with the defined scan order.

11. The electronic device according to claim 1 , wherein the circuitry is further configured to determine, from the set of 3D blocks, a subset of 3D blocks that is in a neighborhood of the 3D block of the 3D point cloud geometry, based on a spatial arrangement of the set of 3D blocks in the 3D point cloud geometry, and

wherein the selection of the one or more modes is based on a usage of the one or more modes to encode each 3D block of the subset of 3D blocks into a respective encoded 3D block.

12. The electronic device according to claim 1 , wherein the circuitry is further configured to determine point cloud metrics including the class information associated with the 3D block of the 3D point cloud geometry, wherein

the one or more modes is selected further based on an application of a classifier model on the point cloud metrics, and

the classifier model is a machine learning model that is trained on a task of mode prediction.

13. The electronic device according to claim 1 , wherein the circuitry is further configured to determine point cloud metrics including the class information associated with the 3D block of the 3D point cloud geometry and a subset of 3D blocks in a neighborhood of the 3D block of the 3D point cloud geometry,

wherein the one or more modes is selected further based on an application of a classifier model on the point cloud metrics, and

the classifier model is a machine learning model that is trained on a task of mode prediction.

14. The electronic device according to claim 1 , wherein the circuitry is further configured to apply a convolutional neural network on the 3D block of the 3D point cloud geometry to generate a mode prediction for the 3D block of the 3D point cloud geometry,

wherein the mode prediction is included in the mode decision information and the one or more modes are selected based on the mode prediction.

15. The electronic device according to claim 1 , wherein the circuitry is further configured to apply a convolutional neural network on the 3D block of the 3D point cloud geometry and a subset of 3D blocks in a neighborhood of the 3D block of the 3D point cloud geometry, to generate a mode prediction for the 3D block of the 3D point cloud geometry,

wherein the mode prediction is included in the mode decision information and the one or more modes are selected based on the mode prediction.

16. A method, comprising:

in an electronic device:

receiving a three-dimensional (3D) point cloud geometry;

partitioning the 3D point cloud geometry into a set of 3D blocks;

determining, for a 3D block of the set of 3D blocks, mode decision information that comprises at least one of:

class information associated with the 3D point cloud geometry,

one or more operational conditions associated with an encoding stage of the 3D point cloud geometry, or

mode-related information associated with one or more 3D blocks of the set of 3D blocks;

selecting one or more modes for the 3D block of the point cloud geometry from a plurality of modes, based on the mode decision information,

wherein each mode of the plurality of modes corresponds to an encoding function for the 3D block of the 3D point cloud geometry,

the plurality of modes further corresponds to a plurality of density levels,

each density level of the plurality of density levels corresponds to a median of a distribution of local density values associated with 3D points in a 3D block of a calibration point cloud of a plurality of calibration point clouds, and

each of the local density values is a number of neighborhood points within a spherical volume around each 3D point in the 3D block of the calibration point cloud; and

encoding the 3D block of the 3D point cloud geometry based on the selected one or more modes.

17. The method according to claim 16 , further comprising:

loading a table that maps the plurality of modes with different classes associated with the plurality of calibration point clouds and operational conditions associated with an encoding stage of the plurality of calibration point clouds, wherein each calibration point cloud of the plurality of calibration point clouds is different from the received 3D point cloud geometry; and

searching the table based on the class information and the one or more operational conditions to select the one or more modes.

18. The method according to claim 16 , wherein the encoding function corresponds to a Deep Neural Network (DNN) model that is trained to encode the 3D block of the 3D point cloud geometry to generate an encoded 3D block.

19. The method according to claim 18 , wherein each mode of the plurality of modes corresponds to an alpha parameter of a focal loss function used in a training stage of the DNN, and

the focal loss function is configured to penalize a removal of non-empty voxels from the 3D block of the 3D point cloud geometry.

20. A non-transitory computer-readable medium having stored thereon, computer-executable instructions that when executed by an electronic device, causes the electronic device to execute operations, the operations comprising:

receiving a three-dimensional (3D) point cloud geometry;

partitioning the 3D point cloud geometry into a set of 3D blocks;

determining, for a 3D block of the set of 3D blocks, mode decision information that comprises at least one of:

class information associated with the 3D point cloud geometry,

one or more operational conditions associated with an encoding stage of the 3D point cloud geometry, or

mode-related information associated with one or more 3D blocks of the set of 3D blocks;

selecting one or more modes for the 3D block of the point cloud geometry from a plurality of modes, based on the mode decision information,

wherein each mode of the plurality of modes corresponds to an encoding function for the 3D block of the 3D point cloud geometry,

the plurality of modes further corresponds to a plurality of density levels,

each density level of the plurality of density levels corresponds to a median of a distribution of local density values associated with 3D points in a 3D block of a calibration point cloud of a plurality of calibration point clouds, and

each of the local density values is a number of neighborhood points within a spherical volume around each 3D point in the 3D block of the calibration point cloud; and

encoding the 3D block of the point cloud geometry based on the selected one or more modes.

Assignments (1)

Continuity (2)

References Cited (18)