Face reconstruction from a learned embedding
The present disclosure provides systems and methods that perform face reconstruction based on an image of a face. In particular, one example system of the present disclosure combines a machine-learned image recognition model with a face modeler that uses a morphable model of a human's facial appearance. The image recognition model can be a deep learning model that generates an embedding in response to receipt of an image (e.g., an uncontrolled image of a face). The example system can further include a small, lightweight, translation model structurally positioned between the image recognition model and the face modeler. The translation model can be a machine-learned model that is trained to receive the embedding generated by the image recognition model and, in response, output a plurality of facial modeling parameter values usable by the face modeler to generate a model of the face.
1. A computing system comprising:
one or more processors;
one or more non-transitory computer readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining an embedding, wherein the embedding was generated by processing an image of a face with a machine-learned image recognition model;
processing the embedding with a machine-learned translation model to generate a plurality of facial modeling parameter values, wherein the plurality of facial modeling parameter values are descriptive of a plurality of facial attributes of the face in the image, and wherein the machine-learned translation model was trained on example embeddings generated from training data comprising a plurality of face morphs, wherein the plurality of face morphs were generated by: processing a plurality of training images to determine a plurality of facial landmarks, determining an average set of facial landmarks, and generating the plurality of face morphs based on warping the plurality of training images to the average set of facial landmarks;
processing the plurality of facial modeling parameter values with a face modeler to generate a model of the face; and
processing the model of the face with a face renderer to generate a rendering of the face.
2. The computing system of claim 1 , wherein the model of the face comprises a three-dimensional model of the face, and wherein the rendering of the face comprises a two-dimensional rendering of the face.
3. The computing system of claim 1 , wherein the machine-learned translation model was trained on a set of training data that includes a plurality of example embeddings labeled with a plurality of example facial modeling parameter values usable by a face modeler to generate a model of a face.
4. The computing system of claim 3 , wherein the example embeddings are obtained by inputting example faces into the machine-learned image recognition model, said example faces being generated by the face modeler using the respective example facial modeling parameter values.
5. The computing system of claim 1 , wherein the operations further comprise:
processing the rendering of the face with an attribute measurer to determine one or more measured facial attributes; and
processing the one or more measured facial attributes with an artistic renderer to generate an artistic rendering of the face.
6. The computing system of claim 1 , wherein the operations further comprise:
providing the rendering of the face as an avatar in a virtual reality environment.
7. The computing system of claim 1 , wherein the machine-learned translation model has been trained on a training dataset comprising a plurality of example images of faces respectively associated with a plurality of example sets of facial modeling parameter values, wherein at least a portion of the training dataset was generated with a morphable model training set to generate faces that have different facial modeling parameter values.
8. The computing system of claim 1 , wherein obtaining an embedding comprises:
obtaining the image of the face;
inputting the image of the face into a face reconstruction system, wherein the face reconstruction system comprises:
a machine-learned convolutional neural network configured to receive and process the image of the face to generate the embedding for the face.
9. The computing system of claim 1 , wherein the plurality of facial modeling parameter values comprises one or more facial modeling parameter values descriptive of landmark positions for the face.
10. The computing system of claim 1 , further comprising:
a machine-learned face reconstruction model that is operable to receive the embedding for the face and, in response to receipt of the embedding, output a reconstructed representation of the face.
11. The computing system of claim 10 , wherein the machine-learned face reconstruction model comprises the machine-learned translation model and the face renderer.
12. A computer-implemented method, the method comprising:
obtaining, by a computing system comprising one or more processors, an embedding, wherein the embedding was generated by processing an image of a face with a machine-learned image recognition model;
processing, by the computing system, the embedding with a face reconstruction model to generate a model of the face, wherein processing the embedding with the face reconstruction model to generate the face comprises:
processing, by the computing system, the embedding with a machine-learned translation model to generate a plurality of facial modeling parameter values, wherein the plurality of facial modeling parameter values are descriptive of a plurality of facial attributes of the face in the image, and wherein the machine-learned translation model was trained on example embeddings generated from training data comprising a plurality of face morphs, wherein the plurality of face morphs were generated by: processing a plurality of training images to determine a plurality of facial landmarks, determining an average set of facial landmarks, and generating the plurality of face morphs based on warping the plurality of training images to the average set of facial landmarks; and
processing, by the computing system, the plurality of facial modeling parameter values with a face modeler to generate a model of the face.
13. The method of claim 12 , further comprising:
processing, by the computing system, the model of the face with a face renderer to generate a rendering of the face.
14. The method of claim 12 , wherein the machine-learned translation model comprises a regression neural network.
15. The method of claim 12 , wherein the machine-learned translation model comprises one or more fully connected layers.
16. The method of claim 12 , wherein the face reconstruction model comprises a machine-learned variational autoencoder.
17. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:
obtaining an embedding, wherein the embedding was generated by processing an image of a face with a machine-learned image recognition model;
processing the embedding with a machine-learned translation model to generate a plurality of facial modeling parameter values, wherein the plurality of facial modeling parameter values are descriptive of a plurality of facial attributes of the face in the image, and wherein the machine-learned translation model was trained on example embeddings generated from training data comprising a plurality of face morphs, wherein the plurality of face morphs were generated by: processing a plurality of training images to determine a plurality of facial landmarks, determining an average set of facial landmarks, and generating the plurality of face morphs based on warping the plurality of training images to the average set of facial landmarks;
processing the plurality of facial modeling parameter values with a face modeler to generate a model of the face; and
processing the model of the face with a face renderer to generate a rendering of the face.
18. The one or more non-transitory computer-readable media of claim 17 , wherein the rendering comprises one or more personalized emojis.
19. The one or more non-transitory computer-readable media of claim 17 , wherein the image is descriptive of a side of the face with uneven lighting, and wherein the rendering comprises a front-facing, evenly-lit rendering of the face.
20. The one or more non-transitory computer-readable media of claim 17 , wherein the image of the face comprises a controlled image of the face with front-facing pose.