Advanced automatic rig creation processes
The disclosure provides methods and systems for automatically generating an animatable object, such as a 3D model. In particular, the present technology provides fast, easy, and automatic animatable solutions based on unique facial characteristics of user input. Various embodiments of the present technology include receiving user input, such as a two-dimensional image or three-dimensional scan of a user's face, and automatically detecting one or more features. The methods and systems may further include deforming a template geometry and a template control structure based on the one or more detected features to automatically generate a custom geometry and custom control structure, respectively. A texture of the received user input may also be transferred to the custom geometry. The animatable object therefore includes the custom geometry, the transferred texture, and the custom control structure, which follow a morphology of the face.
1. A method for animating a reconstructed head mesh, the method comprising:
using artificial intelligence to;
automatically detect one or more features of received user input for a face, the automatic detection comprising automatically determining a plurality of spatial coordinates for the face, each spatial coordinate of the plurality of spatial coordinates for the face being associated with one of the one or more features of the received user input for the face, the plurality of spatial coordinates for the face being determined using ray casting techniques;
automatically deform a template geometry based on one of image-based target points utilizing the one or more detected features, descriptor-based target points, and artificial intelligence based on a machine learning algorithm to automatically generate a custom geometry;
automatically transfer a texture of the received user input to custom geometry;
automatically deform a template control structure based on the one or more detected features to automatically generate a custom control structure;
automatically generate an animatable object having the custom geometry, the transferred texture, and the custom control structure; and
generate three-dimensional trajectories of mesh points to simulate a facial deformation.
2. The method of claim 1 , further comprising using a user-specified expression type.
3. The method of claim 2 , wherein the user-specified expression type is smiling, happy, sad or angry.
4. The method of claim 3 , further comprising automatically selecting head parts.
5. The method of claim 4 , the head parts including a facial structure.
6. The method of claim 5 , the facial structure relating to a gender.
7. The method of claim 5 , the facial structure reflecting differences in statistical information regarding a facial morphology of a group.
8. The method of claim 1 , further comprising using artificial intelligence to perform facial animation.
9. The method of claim 1 , further comprising using artificial intelligence to drive mesh deformation over time.
10. The method of claim 9 , further comprising using the machine learning algorithm trained from a set of real-world videos to simulate realistic facial deformations.
11. The method of claim 9 , further comprising using the machine learning algorithm trained from motion capture data to simulate realistic facial deformations.
12. The method of claim 1 , wherein the step of automatically deforming of the template geometry based on the image-based target points comprises matching a set of key points from a template head with their equivalent anatomical points detected on an input subject's head.
13. The method of claim 1 , wherein the step of automatically deforming of the template geometry based on the descriptor-based target points comprises specifying a set of descriptors by a user and converting the user specified descriptors into actual 3D points.
14. The method of claim 13 , wherein the set of descriptors specified by the user are one of quantitative descriptors or qualitative descriptors.
15. The method of claim 14 , wherein the conversion of the user specified descriptors into actual 3D points comprises automatically selecting a head part from a data store that best matches the user specified descriptors.
16. The method of claim 15 , wherein the automatic selection of the head part from the data store that best matches the user specified descriptors is based on numerical optimization of similarity measures corresponding to relevant head part comparisons.
17. The method of claim 1 , wherein the step of automatically detecting the one or more features of the received user input for the face further comprises establishing a first set of facial features used in the deformation of the template geometry, a second set of facial features to facilitate alignment and scale of the template geometry to be deformed and a third set of facial features to determine coloring of the template geometry to be deformed.
18. The method of claim 1 , wherein the automatic transfer of the texture of the received user input to the custom geometry is configured to adapt to an environment of a 3D model.
19. The method of claim 18 , wherein the adaptation comprises adjustment for illumination and context.
20. The method of claim 1 , wherein the automatic transfer of the texture of the received user input to the custom geometry includes mapping a plurality of pixels of the received user input to vertices of the custom geometry.