Asymmetric facial expression recognition
The present disclosure describes techniques for facial expression recognition. A first loss function may be determined based on a first set of feature vectors associated with a first set of images depicting facial expressions and a first set of labels indicative of the facial expressions. A second loss function may be determined based on a second set of feature vectors associated with a second set of images depicting asymmetric facial expressions and a second set of labels indicative of the asymmetric facial expressions. The first loss function and the second loss function may be used to determine a maximum loss function. The maximum loss function may be applied during training of a model. The trained model may be configured to predict at least one asymmetric facial expression in a subsequently received image.
1. A method, comprising:
determining a first loss function based on a first set of feature vectors associated with a first set of images depicting facial expressions and a first set of labels indicative of the facial expressions;
determining a second loss function based on a second set of feature vectors associated with a second set of images depicting asymmetric facial expressions and a second set of labels indicative of the asymmetric facial expressions;
determining, based on the first loss function and the second loss function, a maximum loss function;
applying the maximum loss function during training of a model, wherein the trained model is configured to predict different types of asymmetric facial expressions in input images; and
controlling a facial animation using the predicted different types of asymmetric facial expressions.
2. The method of claim 1 , wherein the first set of labels comprises a first quantity of blend shape coefficients and the second set of labels comprises a second quantity of blend shape coefficients, the second quantity being smaller than the first quantity.
3. The method of claim 1 , wherein the second set of labels comprises blend shape coefficients indicative of the asymmetric facial expressions associated with at least one of an eye, a mouth, a jaw, an eyebrow, a cheek, a nose, or a tongue.
4. The method of claim 1 , wherein the second set of images correspond to a subset of the first set of images.
5. The method of claim 1 , further comprising:
augmenting the second set of images by flipping at least one portion of at least one image of the second set of images.
6. The method of claim 5 , further comprising:
augmenting the second set of labels based on at least one flipped portion of at least one image.
7. The method of claim 1 , further comprising:
determining the first set of feature vectors by inputting the first set of images into a neural network and performing regression on a different set of feature vectors with a higher dimension than the first set of feature vectors, the different set of feature vectors being associated with the first set of images.
8. A system, comprising:
at least one computing device in communication with a computer memory, the computer memory comprising computer-readable instructions that upon execution by the at least one computing device, configure the system to perform operations comprising:
determining a first loss function based on a first set of feature vectors associated with a first set of images depicting facial expressions and a first set of labels indicative of the facial expressions;
determining a second loss function based on a second set of feature vectors associated with a second set of images depicting asymmetric facial expressions and a second set of labels indicative of the asymmetric facial expressions;
determining, based on the first loss function and the second loss function, a maximum loss function;
applying the maximum loss function during training of a model, wherein the trained model is configured to predict different types of asymmetric facial expressions in input images; and
controlling a facial animation using the predicted different types of asymmetric facial expressions.
9. The system of claim 8 , wherein the first set of labels comprises a first quantity of blend shape coefficients and the second set of labels comprises a second quantity of blend shape coefficients, the second quantity being smaller than the first quantity.
10. The system of claim 8 , wherein the second set of labels comprises blend shape coefficients indicative of the asymmetric facial expressions associated with at least one of an eye, a mouth, a jaw, an eyebrow, a cheek, a nose, or a tongue.
11. The system of claim 8 , wherein the second set of images correspond to a subset of the first set of images.
12. The system of claim 8 , the operations further comprising:
augmenting the second set of images by flipping at least one portion of at least one image of the second set of images.
13. The system of claim 12 , the operations further comprising:
augmenting the second set of labels based on at least one flipped portion of at least one image.
14. The system of claim 8 , the operations further comprising:
determining the first set of feature vectors by inputting the first set of images into a neural network and performing regression on a different set of feature vectors with a higher dimension than the first set of feature vectors, the different set of feature vectors being associated with the first set of images.
15. A non-transitory computer-readable storage medium, storing computer-readable instructions that upon execution by a processor cause the processor to implement operations comprising:
determining a first loss function based on a first set of feature vectors associated with a first set of images depicting facial expressions and a first set of labels indicative of the facial expressions;
determining a second loss function based on a second set of feature vectors associated with a second set of images depicting asymmetric facial expressions and a second set of labels indicative of the asymmetric facial expressions;
determining, based on the first loss function and the second loss function, a maximum loss function;
applying the maximum loss function during training of a model, wherein the trained model is configured to predict different types of asymmetric facial expressions in input images; and
controlling a facial animation using the predicted different types of asymmetric facial expressions.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the first set of labels comprises a first quantity of blend shape coefficients and the second set of labels comprises a second quantity of blend shape coefficients, the second quantity being smaller than the first quantity.
17. The non-transitory computer-readable storage medium of claim 15 , the operations further comprising:
determining the first set of feature vectors by inputting the first set of images into a neural network and performing regression on a different set of feature vectors with a higher dimension than the first set of feature vectors, the different set of feature vectors being associated with the first set of images.