Systems and methods for generating computer ready animation models of a human head from captured data images
System and methods for computer animations of 3D models of heads generated from images of faces is disclosed. A 2D captured image that includes an image of a face can be received and used to generate a static 3D model of a head. A rig can be fit to the static 3D model to generate an animation-ready 3D generative model. Sets of rigs can be parameters that each map to particular sounds or particular facial movement observed in a video. These mappings can be used to generate a playlists of sets of rig parameters based upon received audio or video content. The playlist may be played in synchronization with an audio rendition of the audio content. Methods can receive a captured image, identify taxonomy attributes from the captured image, select a template model for the captured image, and perform a shape solve for the selected template model based on the identified taxonomy attributes.
1. A method for generating a three dimensional (3D) head model from a captured image, the method comprising:
receiving a captured image;
identifying a set of taxonomy attributes from the captured image by:
using a single encoder model to generate an embedding of the captured image;
using a set of one or more terminal models to analyze the generated embedding; and
identifying taxonomy attributes based on the classifications of the terminal models;
selecting a template model for the captured image; and
performing a shape solve for the selected template model based on the identified taxonomy attributes.
2. The method of claim 1 , wherein the set of taxonomy attributes comprises a set of one or more local taxonomy attributes and a set of one or more global taxonomy attributes.
3. The method of claim 2 further comprising performing a texture synthesis based on one or more of the taxonomy attributes.
4. The method of claim 2 , wherein the template model is selected based on the set of global taxonomy attributes.
5. The method of claim 2 , wherein identifying the set of global taxonomy attributes comprises using a classifier model to classify the captured image.
6. The method of claim 2 , wherein the set of global taxonomy attributes comprises at least one of an ethnicity and a gender.
7. The method of claim 2 , wherein the set of local taxonomy attributes comprise one or more of head shape, eyelid turn, eyelid height, nose width, nose turn, mustache, beard, sideburns, eye rotation, jaw angle, lip thickness, and chin shape.
8. The method of claim 1 , wherein each terminal model of the set of terminal models produces a score for an associated local taxonomy attribute, wherein performing the shape solve for a given local taxonomy attribute is based on the score calculated for the local taxonomy attribute.
9. The method of claim 1 further comprising:
training the single encoder model and the set of terminal models, wherein each terminal model is trained to classify a different taxonomy attribute.
10. The method of claim 9 , wherein training the encoder comprises:
generating an embedding from the single encoder model;
calculating a first loss for a first terminal model based on the generated embedding;
back propagating the first calculated loss through the encoder;
calculating a second loss for a second terminal model based on the generated embedding; and
back propagating the second calculated loss through the single encoder model.
11. The method of claim 1 , wherein receiving the captured image comprises capturing an image with a camera of a mobile device, wherein performing the shape solve is performed on the same mobile device.
12. A non-transitory machine readable medium containing processor instructions for generating a three dimensional (3D) head model from a captured image, where execution of the instructions by a processor causes the processor to perform a process that comprises:
receiving a captured image;
identifying a set of taxonomy attributes from the captured image by:
using a single encoder model to generate an embedding of the captured image;
using a set of one or more terminal models to analyze the generated embedding; and
identifying taxonomy attributes based on the classifications of the terminal models;
selecting a template model for the captured image; and
performing a shape solve for the selected template model based on the identified taxonomy attributes.
13. The non-transitory machine readable medium of claim 12 , wherein the set of taxonomy attributes comprises a set of one or more local taxonomy attributes and a set of one or more global taxonomy attributes, wherein the template model is selected based on the set of global attributes.
14. The non-transitory machine readable medium of claim 13 , wherein the set of global taxonomy attributes comprises at least one of an ethnicity and a gender.
15. The non-transitory machine readable medium of claim 13 , wherein execution of the instruction further causes the processor to perform a texture synthesis based on one or more of the taxonomy attributes.
16. The non-transitory machine readable medium of claim 12 , wherein the process further comprises training the single encoder model and the set of terminal models, wherein each terminal model is trained to classify a different taxonomy attribute.
17. The non-transitory machine readable medium of claim 16 , wherein training the encoder comprises:
generating an embedding from the single encoder model;
calculating a first loss for a first terminal model based on the generated embedding;
back propagating the first calculated loss through the encoder;
calculating a second loss for a second terminal model based on the generated embedding; and
back propagating the second calculated loss through the single encoder model.
18. The non-transitory machine readable medium of claim 12 , wherein receiving the captured image comprises capturing an image with a camera of a mobile device, wherein performing the shape solve is performed on the same mobile device.