Volumetric capture and mesh-tracking based machine learning 4D face/body deformation training

1. A method programmed in a non-transitory memory of a device comprising:

using a volumetric capture system for high-quality 4D scanning, wherein the volumetric capture system is configured for capturing high quality photos and video simultaneously;

using mesh-tracking to establish temporal correspondences across a 4D scanned human face and full-body mesh sequence;

using mesh registration to establish spatial correspondences between the 4D scanned human face and full-body mesh sequence and a 3D computer graphics physical simulator, wherein the 3D computer graphics physical simulator receives a surface representation and a hierarchical set of interconnected parts from a rigging implementation; and

training surface deformation as a delta from the 3D computer graphics physical simulator using machine learning including combining detail-rich implicit functions and parametric representations to construct 3D models.

2. The method of claim 1 further comprising acquiring multiple, separate 3D scans.

3. The method of claim 1 further comprising predicting and synthesizing the deformation for natural animation using a standard motion capture animation.

4. The method of claim 1 further comprising:

using single-view or multi-view 2D videos of motion capture actors as input;

solving 3D model parameters for animation; and

based on the 3D model parameters solved by 3D solving, predicting 4D surface deformation from machine learning training.

5. An apparatus comprising:

a non-transitory memory for storing an application, the application for:

using a volumetric capture system for high-quality 4D scanning, wherein the volumetric capture system is configured for capturing high quality photos and video simultaneously;

using mesh-tracking to establish temporal correspondences across a 4D scanned human face and full-body mesh sequence;

using mesh registration to establish spatial correspondences between the 4D scanned human face and full-body mesh sequence and a 3D computer graphics physical simulator, wherein the 3D computer graphics physical simulator receives a surface representation and a hierarchical set of interconnected parts from a rigging implementation; and

training surface deformation as a delta from the 3D computer graphics physical simulator using machine learning including combining detail-rich implicit functions and parametric representations to construct 3D models; and

a processor coupled to the memory, the processor configured for processing the application.

6. The apparatus of claim 5 wherein the application is further configured for acquiring multiple, separate 3D scans.

7. The apparatus of claim 5 wherein the application is further configured for predicting and synthesizing the deformation for natural animation using a standard motion capture animation.

8. The apparatus of claim 5 wherein the application is further configured for:

using single-view or multi-view 2D videos of motion capture actors as input;

solving 3D model parameters for animation; and

based on the 3D model parameters solved by 3D solving, predicting 4D surface deformation from machine learning training.

9. A system comprising:

a volumetric capture system for high-quality 4D scanning, wherein the volumetric capture system is configured for capturing high quality photos and video simultaneously; and

a computing device configured for:

using mesh-tracking to establish temporal correspondences across a 4D scanned human face and full-body mesh sequence;

using mesh registration to establish spatial correspondences between the 4D scanned human face and full-body mesh sequence and a 3D computer graphics physical simulator, wherein the 3D computer graphics physical simulator receives a surface representation and a hierarchical set of interconnected parts from a rigging implementation; and

training surface deformation as a delta from the 3D computer graphics physical simulator using machine learning including combining detail-rich implicit functions and parametric representations to construct 3D models.

10. The system of claim 9 wherein the computing device is further configured for predicting and synthesizing the deformation for natural animation using a standard motion capture animation.

11. The system of claim 9 wherein the computing device is further configured for:

using single-view or multi-view 2D videos of motion capture actors as input;

solving 3D model parameters for animation; and

based on the 3D model parameters solved by 3D solving, predicting 4D surface deformation from machine learning training.

12. A method programmed in a non-transitory memory of a device comprising:

using single-view or multi-view 2D videos of motion capture actors as input;

solving 3D model parameters for animation; and

based on the 3D model parameters solved by 3D solving, predicting 4D surface deformation from machine learning training using mesh-tracking to establish temporal correspondences across a 4D scanned human face and full-body mesh sequence;

using mesh registration to establish spatial correspondences between the 4D scanned human face and full-body mesh sequence and a 3D computer graphics physical simulator, wherein the 3D computer graphics physical simulator receives a surface representation and a hierarchical set of interconnected parts from a rigging implementation; and

training surface deformation as a delta from the 3D computer graphics physical simulator using the machine learning including combining detail-rich implicit functions and parametric representations to construct 3D models.