Systems and methods for building a muscle-to-skin transformation in computer animation
An animation system wherein a machine learning model is adopted to learn a transformation relationship between facial muscle movements and skin surface movements. For example, for the skin surface representing “smile,” the transformation model derives movement vectors relating to what facial muscles are activated, what are the muscle strains, what is the joint movement, and/or the like. Such derived movement vectors may be used to simulate the skin surface “smile.”.
1. A computer-implemented method for learning a strain-to-skin transformation of a facial action in an animation system, the method comprising:
receiving a plurality of facial scans of a face of an actor representing a plurality of facial movements over a data bundle time period;
generating, from the plurality of facial scans, a data bundle comprising a first cache of a time-varying facial muscle strain vector over the data bundle time period, a second cache of a time-varying skin surface vector over the data bundle time period;
inputting the data bundle and anatomical data corresponding to the actor to a learning model;
generating, by the learning model, a predicted skin surface vector based on the first cache of a time-varying facial muscle strain vector over the data bundle time period and the anatomical data corresponding to the actor;
computing a metric based on the predicted skin surface vector and a ground truth skin surface from the second cache of a time-varying skin surface vector over the data bundle time period; and
updating parameters of the learning model by minimizing the computed metric.
2. The method of claim 1 , wherein the learning model is a linear regression model and is updated by solving a constrained least-square problem based on the computed metric.
3. The method of claim 1 , wherein the learning model is a fully-connected layer, and the computed metric is a L2 loss objective.
4. The method of claim 1 , further comprising:
generating a training dataset of a plurality of data bundles, wherein the plurality of data bundles corresponding to multiple types of facial movements including any combination of a facial action, a dialog, and a depiction of an emotion.
5. The method of claim 1 , further comprising:
deriving a transformational relationship between at least the time-varying facial muscle strain vector and the time-varying skin surface vector from the updated learning model.
6. The method of claim 5 , further comprising:
obtaining an adjusted facial muscle strain vector having strain values not obtained from any of the plurality of facial scans; and
generating, using the transformational relationship, an adjusted predicted skin surface vector from the adjusted facial muscle strain vector.
7. The method of claim 6 , further comprising:
determining that the adjusted predicted skin surface vector matches a desired skin surface vector; and
sending the adjusted facial muscle strain vector to an animation creation system for creating an animated skin surface based on the adjusted facial muscle strain vector.
8. The method of claim 1 , wherein the plurality of facial scans includes a first facial scan of a neutral pose of the actor and a second facial scan of a non-neutral pose of the actor, and wherein the second facial scan is a period of time apart from the first facial scan.
9. The method of claim 1 , wherein the plurality of facial scans are collected from the actor across a period of time, and the ground truth skin surface is obtained from facial scans that are averaged out among similar facial scans collected from the period of time.
10. The method of claim 1 , wherein the plurality of facial movements include one or more of a facial action, a dialog, and/or a depiction of an emotion.
11. A system for learning a strain-to-skin transformation of a facial action in an animation system, the system comprising:
a communication interface that receives a plurality of facial scans of a face of an actor representing a plurality of facial movements over a data bundle time period;
a memory that stores a plurality of processor-executable instructions; and
one or more hardware processors that read the plurality of processor-executable instructions to perform:
generating, from the plurality of facial scans, a data bundle comprising a first cache of a time-varying facial muscle strain vector over the data bundle time period, a second cache of a time-varying skin surface vector over the data bundle time period;
inputting the data bundle and anatomical data corresponding to the actor to a learning model;
generating, by the learning model, a predicted skin surface vector based on the first cache of a time-varying facial muscle strain vector over the data bundle time period and the anatomical data corresponding to the actor;
computing a metric based on the predicted skin surface vector and a ground truth skin surface from the second cache of a time-varying skin surface vector over the data bundle time period; and
updating parameters of the learning model by minimizing the computed metric.
12. The system of claim 11 , wherein the learning model is a linear regression model and is updated by solving a constrained least-square problem based on the computed metric.
13. The system of claim 11 , wherein the learning model is a fully-connected layer, and the computed metric is a L2 loss objective.
14. The system of claim 11 , wherein the one or more hardware processors read the plurality of processor-executable instructions to further perform:
generating a training dataset of a plurality of data bundles, wherein the plurality of data bundles corresponding to multiple types of facial movements including any combination of a facial action, a dialog, and a depiction of an emotion.
15. The system of claim 11 , wherein the one or more hardware processors read the plurality of processor-executable instructions to further perform:
deriving a transformational relationship between at least the time-varying facial muscle strain vector and the time-varying skin surface vector from the updated learning model.
16. The system of claim 15 , wherein the one or more hardware processors read the plurality of processor-executable instructions to further perform:
obtaining an adjusted facial muscle strain vector having strain values not obtained from any of the plurality of facial scans; and
generating, using the transformational relationship, an adjusted predicted skin surface vector from the adjusted facial muscle strain vector.
17. The system of claim 16 , wherein the one or more hardware processors read the plurality of processor-executable instructions to further perform:
determining that the adjusted predicted skin surface vector matches a desired skin surface vector; and
sending the adjusted facial muscle strain vector to an animation creation system for creating an animated skin surface based on the adjusted facial muscle strain vector.
18. The system of claim 11 , wherein the plurality of facial scans includes a first facial scan of a neutral pose of the actor and a second facial scan of a non-neutral pose of the actor, and wherein the second facial scan is a period of time apart from the first facial scan.
19. The system of claim 11 , wherein the plurality of facial scans are collected from the actor across a period of time, and the ground truth skin surface is obtained from facial scans that are averaged out among similar facial scans collected from the period of time.
20. A non-transitory processor-readable medium storing a plurality of processor-executable instructions for learning a strain-to-skin transformation of a facial action in an animation system, the processor-executable instructions being executed by one or more hardware processors to perform:
receiving a plurality of facial scans of a face of an actor representing a plurality of facial movements over a data bundle time period;
generating, from the plurality of facial scans, a data bundle comprising a first cache of a time-varying facial muscle strain vector over the data bundle time period, a second cache of a time-varying skin surface vector over the data bundle time period;
inputting the data bundle and anatomical data corresponding to the actor to a learning model;
generating, by the learning model, a predicted skin surf ace vector based on the first cache of a time-varying facial muscle strain vector over the data bundle time period and the anatomical data corresponding to the actor;
computing a metric based on the predicted skin surface vector and a ground truth skin surface from the second cache of a time-varying skin surface vector over the data bundle time period; and
updating parameters of the learning model by minimizing the computed metric.