Semantic deep face models
Techniques are disclosed for training and applying nonlinear face models. In embodiments, a nonlinear face model includes an identity encoder, an expression encoder, and a decoder. The identity encoder takes as input a representation of a facial identity, such as a neutral face mesh minus a reference mesh, and outputs a code associated with the facial identity. The expression encoder takes as input a representation of a target expression, such as a set of blendweight values, and outputs a code associated with the target expression. The codes associated with the facial identity and the facial expression can be concatenated and input into the decoder, which outputs a representation of a face having the facial identity and expression. The representation of the face can include vertex displacements for deforming the reference mesh.
1. A computer-implemented method for generating a representation of a face, the computer-implemented method comprising:
receiving a representation of a facial identity and a representation of a facial expression;
processing the representation of the facial identity using a first variational autoencoder (VAE) encoder to generate an encoded representation of the facial identity;
processing the representation of the facial expression using a second VAE encoder to generate an encoded representation of the facial expression; and
performing one or more decoding operations based on the encoded representation of the facial identity and the encoded representation of the facial expression to generate the representation of the face having the facial identity and the facial expression.
2. The computer-implemented method of claim 1 , wherein the representation of the facial identity includes a mesh of the face having a neutral expression, and the computer-implemented method further comprises generating the encoded representation of the facial identity by:
subtracting a reference mesh from the mesh of the face to determine a set of differences between the mesh of the face and the reference mesh; and
encoding the set of differences to generate the encoded representation of the facial identity.
3. The computer-implemented method of claim 1 , wherein the representation of the facial identity comprises the encoded representation of the facial identity.
4. The computer-implemented method of claim 1 , further comprising generating the encoded representation of the facial identity by adding noise to an encoded representation of another facial identity.
5. The computer-implemented method of claim 1 , wherein the representation of the facial expression includes one or more two-dimensional (2D) landmarks of facial features associated with an image depicting the face, and
the computer-implemented method further comprises generating the encoded representation of the facial expression by mapping the one or more 2D landmarks to the encoded representation of the facial expression.
6. The computer-implemented method of claim 1 ,
wherein the representation of the facial expression includes one or more blendweight values, and the computer-implemented method further comprises performing one or more encoding operations based on the one or more blendweight values to generate the encoded representation of the facial expression.
7. The computer-implemented method of claim 6 , further comprising determining the one or more blendweight values based on one or more images.
8. The computer-implemented method of claim 6 , further comprising:
receiving the one or more blendweight values via a user interface; or
determining the one or more blendweight values based on one or more three-dimensional meshes.
9. The computer-implemented method of claim 1 , wherein the representation of the facial expression comprises the encoded representation of the facial expression.
10. The computer-implemented method of claim 1 , further comprising rendering at least one image based on the representation of the face.
11. A non-transitory computer-readable storage medium including instructions that, when executed by a processing unit, cause the processing unit to perform steps for generating a representation of a face, the steps comprising:
receiving a representation of a facial identity and a representation of a facial expression;
processing the representation of the facial identity using a first variational autoencoder (VAE) encoder to generate an encoded representation of the facial identity; and
processing the representation of the facial expression using a second VAE encoder to generate an encoded representation of the facial expression; and
processing the encoded representation of the facial identity and the encoded representation of the facial expression using a decoder to generate the representation of the face having the facial identity and the facial expression.
12. The non-transitory computer-readable storage medium of claim 11 wherein the representation of the face includes one or more vertex displacements for deforming a reference mesh.
13. The computer-readable storage medium of claim 11 , wherein processing the encoded representation of the facial identity and the encoded representation of the facial expression using the decoder comprises:
concatenating the encoded representation of the facial identity and the encoded representation of the facial expression to generate an encoded representation of the facial identity and the facial expression; and
inputting the encoded representation of the facial identity and the facial expression into the decoder.
14. The computer-readable storage medium of claim 11 , wherein the representation of the facial identity includes a mesh of the face having a neutral expression, and the steps further comprise generating the encoded representation of the facial identity by:
subtracting a reference mesh from the mesh of the face, the to determine a set of differences between the mesh of the face and the reference mesh; and
encoding the set of differences to generate the encoded representation of the facial identity.
15. The computer-readable storage medium of claim 11 , wherein the representation of the facial identity comprises the encoded representation of the facial identity, and the encoded representation of the facial identity is generated by adding noise to an encoded representation of another facial identity.
16. The computer-readable storage medium of claim 11 ,
wherein the representation of the facial expression includes one or more two-dimensional (2D) landmarks of facial features associated with an image depicting the face, and
the steps further comprise generating the encoded representation of the facial expression by mapping the one or more 2D landmarks to the encoded representation of the facial expression.
17. The computer-readable storage medium of claim 11 , wherein the representation of the facial expression includes one or more blendweight values, and the steps further comprise performing one or more encoding operations based on the one or more blendweight values to generate the encoded representation of the facial expression.
18. The computer-readable storage medium of claim 17 , the steps further comprising:
receiving the one or more blendweight values via a user interface; or
determining the one or more blendweight values based on at least one image or three-dimensional mesh.
19. The computer-readable storage medium of claim 11 , the steps further comprising rendering at least one image based on the representation of the face.
20. A computer-implemented method of training a model for generating a representation of a face, the computer-implemented method comprising:
receiving a data set that includes meshes of neutral faces that respectively represent a face having a neutral expression and meshes of faces that respectively represent a face having a facial expression; and
training a model based on the data set, wherein the model comprises:
a first variational autoencoder (VAE) encoder configured to map representations of neutral faces to encoded representations of facial identities,
a second VAE encoder configured to map representations of faces having facial expressions to encoded representations of the facial expressions, and
a decoder configured to map encoded representations of facial identities and encoded representations of facial expressions to representations of faces.
21. The computer-implemented method of claim 20 , wherein training the model based on the data set comprises:
subtracting a reference mesh from each of the meshes of neutral faces to determine a respective set of differences;
determining a respective set of blendweight values corresponding to a facial expression in each of the meshes of faces having facial expressions; and
training the model based on the sets of differences and the sets of blendweight values.