Techniques for improving mesh accuracy using labeled inputs

1. A method for generating a three-dimensional (3D) representation of an object using semantic data, comprising:

receiving input data generated in response to captured video in a filming area;

assigning a label to at least one region of the input data based on an analysis thereof;

setting at least one parameter for each respective one of the at least one region of the input data based on its label; and

generating the 3D representation based on the at least one parameter and semantic data associated with the input data.

2. The method of claim 1 , wherein the assigning of a label to that at least one region of the input data further comprises:

using at least one of: a deep learning method and a blendshape process.

3. The method of claim 1 , wherein the at least one parameter is a mesh parameter, and wherein the mesh parameter includes a mesh density of a generated mesh.

4. The method of claim 3 , further comprising:

generating a mesh with a higher mesh density for at least one of the at least one region when the at least one of the at least one region is labeled as a face, the mesh density being higher relative to mesh density for at least one of the at least one region that is not labeled as a face.

5. The method of claim 1 , further comprising meshing the labeled input data, wherein the meshing further comprises:

selecting a respective mesh process for each of the at least one region based on their respective labels; and

creating a unified mesh by unifying the meshes created for each of the at least one region.

6. The method of claim 5 , wherein at least one of the at least one region is assigned a label of at least one of eyes or ears.

7. The method of claim 1 , further comprising:

generating a mesh of at least one of the at least one region by applying a human blendshape fitting method to generate a mesh.

8. The method of claim 1 , further comprising:

determining if a region of the at least one region in the input data is labeled as a rigid body part based on semantic data; and

tracking each region of the at least one region determined to be a rigid body part.

9. The method of claim 8 , wherein the tracking further comprises:

tracking at least one of movement, deformation, or other changes in the rigid body part across a time sequence.

10. The method of claim 1 , further comprising:

determining if a region of the at least one region in the labeled input data is labeled as a non-rigid body part based on semantic data; and

tracking each region of the at least one region determined to be a non-rigid body part.

11. The method of claim 10 , further comprising:

determining at least one property of each non-rigid body part; and

basing mesh creation of the at least one determined property, whereby mesh creation is improved.

12. The method of claim 1 , further comprising:

determining a set of compression parameters based at least an importance of each one of the at least one region; and

applying a compression process to a mesh generated for each one of the at least one region based on the set of the determined compression parameters.

13. A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute the method of claim 1 .

14. A system for receiving an input data generated in response to captured video in a filming area, comprising:

a processing circuitry; and

a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to:

receive input data generated in response to captured video in a filming area;

assign a label to at least one region of the input data based on an analysis thereof;

set at least one parameter for each respective one of the at least one region of the input data based on its label; and; and

generate the 3D representation based on the at least one parameter and semantic data associated with the input data.

15. The system of claim 14 , wherein, as to the assigning of a label to that at least one region of the input data, the system is further configured to:

label at least a portion of the input data using at least one of: a deep learning method and a blendshape process.

16. The system of claim 14 , wherein the at least one parameter is a mesh parameter, and wherein the mesh parameter includes a mesh density of a generated mesh.

17. The system of claim 16 , wherein the system is further configured to:

generate a mesh with a higher mesh density for at least one of the at least one region when the at least one of the at least one region is labeled as a face, the mesh density being higher relative to mesh density for at least one of the at least one region that is not labeled as a face.

18. The system of claim 14 , wherein the system is further configured to mesh the labeled input data by:

selecting a respective mesh process for each of the at least one region based on their respective labels; and

creating a unified mesh by unifying the meshes created for each of the at least one region.

19. The system of claim 18 , wherein at least one of the at least one region is assigned a label of at least one of eyes or ears.

20. The system of claim 14 , wherein the system is further configured to:

generate a mesh of at least one of the at least one region by applying a human blendshape fitting method to generate a mesh.

21. The system of claim 14 , wherein the system is further configured to:

determine if a region of the at least one region in the input data is labeled as a rigid body part based on semantic data; and

track each region of the at least one region determined to be a rigid body part.

22. The system of claim 21 , wherein the system is further configured to:

track at least one of movement, deformation, or other changes in the rigid body part across a time sequence.

23. The system of claim 14 , wherein the system is further configured to:

determine if a region of the at least one region in the labeled input data is labeled as a non-rigid body part based on semantic data; and

each region of the at least one region determined to be a non-rigid body part.

24. The system of claim 23 , wherein the system is further configured to:

determine at least one property of each non-rigid body part; and

base mesh creation of the at least one determined property, whereby mesh creation is improved.

25. The system of claim 14 , wherein the system is further configured to:

determine a set of compression parameters based at least an importance of each one of the at least one region; and

apply a compression process to a mesh generated for each one of the at least one region based on the set of the determined compression parameters.