Systems and methods for mesh geometry prediction for high efficiency mesh coding

1. A method comprising:

generating a low-resolution polygon mesh from a high-resolution polygon mesh, each of the low-resolution polygon mesh and the high-resolution polygon mesh comprising vertices and connections that represent 3D media content;

using a vertex occupancy prediction network to generate, from the vertices of the low-resolution polygon mesh, reconstructed vertices of the high-resolution polygon mesh, wherein the vertex occupancy prediction network is trained according to a first learning algorithm;

using a connectivity prediction network to generate, from the reconstructed vertices of the high-resolution polygon mesh, reconstructed connections of the high-resolution polygon mesh, wherein the connectivity prediction network is trained according to a second learning algorithm;

computing vertex errors between the reconstructed vertices and the vertices of the high-resolution polygon mesh;

computing connectivity errors between the reconstructed connections and the connections of the high-resolution polygon mesh; and

transmitting, to a receiver over a communication network, bitstreams of the low-resolution polygon mesh, the vertex errors, and the connectivity errors for reconstruction of the high-resolution polygon mesh and display of the 3D media content.

2. The method of claim 1 , wherein the using the vertex occupancy prediction network to generate the reconstructed vertices comprises:

computing a probability of occupancy for vertices of a mesh object, wherein the mesh object is a 3D structure defining all potential vertices for the high-resolution polygon mesh;

comparing the probability of occupancy for the vertices of the mesh object to a threshold value; and

including, as the reconstructed vertices of the high-resolution polygon mesh, vertices of the mesh object associated with a corresponding probability of occupancy greater than the threshold value.

3. The method of claim 2 , wherein the using the vertex occupancy prediction network to generate the reconstructed vertices further comprises:

using a 3D sparse convolutional neural network to generate initial feature channels.

4. The method of claim 3 , wherein the using the vertex occupancy prediction network to generate the reconstructed vertices further comprises:

using a first plurality of groups of parallel convolution layers and a downscaling convolution layer to produce, from the initial feature channels, expanded feature channels.

5. The method of claim 4 , wherein the using the vertex occupancy prediction network to generate the reconstructed vertices further comprises:

using a second plurality of groups of a upscaling convolution layer and the parallel convolution layers to produce, from the expanded feature channels, output feature channels.

6. The method of claim 5 , wherein the using the vertex occupancy prediction network to generate the reconstructed vertices further comprises:

using the 3D sparse convolutional neural network to generate, from the output feature channels, a final output feature.

7. The method of claim 6 , wherein the using the vertex occupancy prediction network to generate the reconstructed vertices further comprises:

using a binary cross entropy SoftMax layer to output, from the final output feature, the probability of occupancy for the vertices of the mesh object.

8. The method of claim 1 , wherein the using the connectivity prediction network to generate the reconstructed connections comprises:

computing a probability of connectivity for connections of the reconstructed vertices of the high-resolution polygon mesh;

comparing the probability of connectivity for the connections of the reconstructed vertices of the high-resolution polygon mesh to a threshold value; and

including, as the reconstructed connections of the high-resolution polygon mesh, connections of the reconstructed vertices of the high-resolution polygon mesh associated with a corresponding probability of connectivity greater than the threshold value.

9. The method of claim 8 , wherein the using the connectivity prediction network to generate the reconstructed connections further comprises:

using a 3D sparse convolutional neural network to generate initial feature channels;

using a first plurality of groups of parallel convolution layers and a downscaling convolution layer to produce, from the initial feature channels, expanded feature channels;

using a second plurality of groups of an upscaling convolution layer and the parallel convolution layers to produce, from the expanded feature channels, output feature channels; and

using the 3D sparse convolutional neural network to generate an intermediate output feature from the output feature channels.

10. The method of claim 9 , wherein the using the connectivity prediction network to generate the reconstructed connections further comprises:

using a transformer block to generate, from the intermediate output feature, a final output feature; and

using a binary cross entropy SoftMax layer to output the probability of connectivity for connections of the reconstructed vertices of the high-resolution polygon mesh.

11. A system comprising:

control circuitry configured to:

generate a low-resolution polygon mesh from a high-resolution polygon mesh, each of the low-resolution polygon mesh and the high-resolution polygon mesh comprising vertices and connections that represent 3D media content;

use a vertex occupancy prediction network to generate, from the vertices of the low-resolution polygon mesh, reconstructed vertices of the high-resolution polygon mesh, wherein the vertex occupancy prediction network is trained according to a first learning algorithm;

use a connectivity prediction network to generate, from the reconstructed vertices of the high-resolution polygon mesh, reconstructed connections of the high-resolution polygon mesh, wherein the connectivity prediction network is trained according to a second learning algorithm;

compute vertex errors between the reconstructed vertices and the vertices of the high-resolution polygon mesh; and

compute connectivity errors between the reconstructed connections and the connections of the high-resolution polygon mesh; and input/output circuitry configured to:

transmit, to a receiver over a communication network, bitstreams of the low-resolution polygon mesh, the vertex errors, and the connectivity errors for reconstruction of the high-resolution polygon mesh and display of the 3D media content.

12. The system of claim 11 , wherein the control circuitry is configured to generate the reconstructed vertices by:

computing a probability of occupancy for vertices of a mesh object, wherein the mesh object is a 3D structure defining all potential vertices for the high-resolution polygon mesh;

comparing the probability of occupancy for the vertices of the mesh object to a threshold value; and

including, as the reconstructed vertices of the high-resolution polygon mesh, vertices of the mesh object associated with a corresponding probability of occupancy greater than the threshold value.

13. The system of claim 12 , wherein the control circuitry is further configured to generate the reconstructed vertices by:

using a 3D sparse convolutional neural network to generate initial feature channels.

14. The system of claim 13 , wherein the control circuitry is further configured to generate the reconstructed vertices by:

using a first plurality of groups of parallel convolution layers and a downscaling convolution layer to produce, from the initial feature channels, expanded feature channels.

15. The system of claim 14 , wherein the control circuitry is further configured to generate the reconstructed vertices by:

using a second plurality of groups of an upscaling convolution layer and the parallel convolution layers to produce, from the expanded feature channels, output feature channels.

16. The system of claim 15 , wherein the control circuitry is further configured to generate the reconstructed vertices by:

using the 3D sparse convolutional neural network to generate, from the output feature channels, a final output feature.

17. The system of claim 16 , wherein the control circuitry is further configured to generate the reconstructed vertices by:

using a binary cross entropy SoftMax layer to output, from the final output feature, the probability of occupancy for the vertices of the mesh object.

18. The system of claim 11 , wherein the control circuitry is configured to generate the reconstructed connections by:

computing a probability of connectivity for connections of the reconstructed vertices of the high-resolution polygon mesh;

comparing the probability of connectivity for the connections of the reconstructed vertices of the high-resolution polygon mesh to a threshold value; and

including, as the reconstructed connections of the high-resolution polygon mesh, connections of the reconstructed vertices of the high-resolution polygon mesh associated with a corresponding probability of connectivity greater than the threshold value.

19. The system of claim 18 , wherein the control circuitry is further configured to generate the reconstructed connections by:

using a 3D sparse convolutional neural network to generate initial feature channels;

using a first plurality of groups of parallel convolution layers and a downscaling convolution layer to produce, from the initial feature channels, expanded feature channels;

using a second plurality of groups of an upscaling convolution layer and the parallel convolution layers to produce, from the expanded feature channels, output feature channels; and

using the 3D sparse convolutional neural network to generate, from the output feature channels, an intermediate output.

20. The system of claim 19 , wherein the control circuitry is further configured to generate the reconstructed connections by:

using a transformer block to generate a final output feature from the intermediate output feature; and

using a binary cross entropy SoftMax layer to output the probability of connectivity for the connections of the reconstructed vertices of the high-resolution polygon mesh.