Face reconstruction using a mesh convolution network
Embodiment of the present invention sets forth techniques for performing face reconstruction. The techniques include generating an identity mesh based on an identity encoding that represents an identity associated with a face in one or more images. The techniques also include generating an expression mesh based on an expression encoding that represents an expression associated with the face in the one or more images. The techniques also include generating, by a machine learning model, an output mesh of the face based on the identity mesh and the expression mesh.
1. A computer-implemented method for performing reconstruction of a face, the computer-implemented method comprising:
generating an identity mesh based on an identity encoding that represents an identity associated with a face in one or more images;
generating an expression mesh based on an expression encoding that represents an expression associated with the face in the one or more images; and
generating, by a machine learning model, an output mesh of the face via an upsampling operation associated with at least one of the identity mesh or the expression mesh.
2. The computer-implemented method of claim 1 , wherein the expression mesh associates one or more expression features with one or more locations of a mesh topology, and the identity mesh associates one or more identity features with one or more locations of the mesh topology.
3. The computer-implemented method of claim 1 , further comprising generating, based on the one or more images of the face, one or more camera parameters associated with the one or more images.
4. The computer-implemented method of claim 3 , further comprising adjusting one or both of the identity mesh or the expression mesh based on a feature selection, the feature selection being based on the one or more camera parameters.
5. The computer-implemented method of claim 3 , further comprising training the machine learning model based on one or more losses, the one or more losses including one or more of,
an identity loss based on the generated identity mesh and a ground truth identity mesh of the face,
an expression loss based on the generated expression mesh and a ground truth expression mesh of the face,
an output mesh loss based on the generated output mesh and a ground truth mesh, or
a camera parameter loss based on the one or more camera parameters and one or more ground truth camera parameters.
6. The computer-implemented method of claim 1 , wherein each of the expression mesh and the identity mesh includes one or both of a set of vertex coordinates or a set of vertex displacement vectors.
7. The computer-implemented method of claim 1 , wherein a resolution of the output mesh is higher than a resolution of one or both of the identity mesh or the expression mesh.
8. The computer-implemented method of claim 1 , further comprising training the machine learning model based on an identity consistency loss, wherein the identity consistency loss is based on identity encodings associated with each of the one or more images.
9. The computer-implemented method of claim 1 , further comprising normalizing a set of vertices in one or both of the identity mesh or the expression mesh, wherein the normalizing is based on a difference between the one or both of the identity mesh or the expression mesh and an average mesh.
10. One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
generating an identity mesh based on an identity encoding that represents an identity associated with a face in one or more images;
generating an expression mesh based on an expression encoding that represents an expression associated with the face in the one or more images; and
generating, by a machine learning model, an output mesh of the face via an upsampling operation associated with at least one of the identity mesh or the expression mesh.
11. The one or more non-transitory computer readable media of claim 10 , wherein the expression mesh associates one or more expression features with one or more locations of a mesh topology, and the identity mesh associates one or more identity features with one or more locations of the mesh topology.
12. The one or more non-transitory computer readable media of claim 10 , further comprising generating, based on the one or more images of the face, one or more camera parameters associated with the one or more images.
13. The one or more non-transitory computer readable media of claim 12 , further comprising adjusting one or both of the identity mesh or the expression mesh based on a feature selection, the feature selection being based on the one or more camera parameters.
14. The one or more non-transitory computer readable media of claim 12 , wherein the instructions further cause the one or more processors to perform the step of training the machine learning model based on one or more losses, the one or more losses including one or more of,
an identity loss based on the generated identity mesh and a ground truth identity mesh of the face,
an expression loss based on the generated expression mesh and a ground truth expression mesh of the face,
an output mesh loss based on the generated output mesh and a ground truth mesh, or
a camera parameter loss based on the one or more camera parameters and one or more ground truth camera parameters.
15. The one or more non-transitory computer readable media of claim 10 , wherein each of the expression mesh and the identity mesh includes one or both of a set of vertex coordinates or a set of vertex displacement vectors.
16. The one or more non-transitory computer readable media of claim 10 , wherein a resolution of the output mesh is higher than a resolution of one or both of the identity mesh or the expression mesh.
17. The one or more non-transitory computer readable media of claim 10 , wherein the instructions further cause the one or more processors to perform the step of training the machine learning model based on an identity consistency loss, wherein the identity consistency loss is based on identity encodings associated with each of the one or more images.
18. The one or more non-transitory computer readable media of claim 10 , wherein the instructions further cause the one or more processors to perform the step of normalizing a set of vertices in one or both of the identity mesh or the expression mesh, wherein the normalizing is based on a difference between the one or both of the identity mesh or the expression mesh and an average mesh.
19. A system, comprising:
one or more memories that store instructions, and
one or more processors that are coupled to the one or more memories and,
when executing the instructions, are configured to:
generate an identity mesh based on an identity encoding that represents an identity of a face in one or more images;
generate an expression mesh based on an expression encoding that represents an expression of the face in the one or more images of the face; and
generate, by a machine learning model, an output mesh of the face via an upsampling operation associated with at least one of the identity mesh or the expression mesh.
20. The system of claim 19 , wherein the one or more processors, when executing the instructions, are configured to train the machine learning model based on one or more losses, the one or more losses including one or more of,
an identity loss based on the generated identity mesh and a ground truth identity mesh of the face,
an expression loss based on the generated expression mesh and a ground truth expression mesh of the face,
an output mesh loss based on the generated output mesh and a ground truth mesh, or
a camera parameter loss based on one or more camera parameters and one or more ground truth camera parameters.