Automatic avatar generation using semi-supervised machine learning
Avatar generation from an image is performed using semi-supervised machine learning. An image space model undergoes unsupervised training from images to generate latent image vectors responsive to image inputs. An avatar parameter space model undergoes unsupervised training from avatar parameter values for avatar parameters to generate latent avatar parameter vectors responsive to avatar parameter value inputs. A cross-modal mapping model undergoes supervised training on image-avatar parameter pair inputs corresponding to the latent image vectors and the latent avatar parameter vectors. The trained image space model generates a latent image vector from an image input. The trained cross-modal mapping model translates the latent image vector to a latent avatar parameter vector. The trained avatar parameter space model generates avatar parameter values from the latent avatar parameter vector. The latent avatar parameter vector can be used to render an avatar having features corresponding to the input image.
1 . A method for avatar generation performed by one or more processors, the method comprising:
training an image space model on a set of images, wherein the trained image space model generates latent image vectors responsive to image inputs;
training an avatar parameter space model on a set of avatar parameter values for avatar parameters, wherein the trained avatar parameter space model generates latent avatar parameter vectors responsive to avatar parameter value inputs; and
training a cross-modal mapping model on the latent image vectors and the latent avatar parameter vectors, the latent image vectors and the latent avatar parameter vectors respectively generated using the trained image space model and the trained avatar parameter space model responsive to image-avatar parameter pair inputs.
2 . The method of claim 1 , further comprising generating the set of avatar parameter values by modifying an initial set of avatar parameter values for an avatar.
3 . The method of claim 1 , further comprising generating the image-avatar parameter pair inputs by receiving modifications to avatar parameter values such that avatars generated as a result of the modifications correspond to images, wherein the modified avatar parameter values and corresponding images form image-avatar parameter pairs used as the image-avatar parameter pair inputs.
4 . The method of claim 1 , wherein the avatar parameter space model is trained independently from the image space model.
5 . The method of claim 1 , wherein training the cross-modal mapping model comprises minimizing an alignment loss between cross-modal mapping model output vectors and the latent avatar parameter vectors.
6 . The method of claim 1 , wherein the trained cross-modal mapping model is configured to translate a latent image vector from a latent image vector space to a latent avatar parameter vector of a latent avatar parameter vector space.
7 . The method of claim 1 , wherein:
the image space model is trained using unsupervised learning;
the avatar parameter space model is trained using unsupervised learning; and
the cross-modal mapping model is trained using supervised learning.
8 . The method of claim 1 , wherein:
the image space model comprises a convolutional neural network;
the avatar parameter space model comprises a first multilayer perceptron; and
the cross-modal mapping model comprises a second multilayer perceptron.
9 . The method of claim 1 , wherein:
the trained image space model comprises a latent image vector encoder; and
the trained avatar parameter space model comprises a latent avatar parameter vector decoder.
10 . One or more computer storage media storing computer readable instructions thereon that, when executed by a processor, cause the processor to perform a method for avatar generation, the method comprising:
accessing latent image vectors generated from image inputs using a trained image space model;
accessing latent avatar parameter vectors generated from avatar parameter value inputs using a trained avatar parameter space model, wherein the image inputs and the avatar parameter value inputs from image-avatar parameter pairs; and
training a cross-modal mapping model on the latent image vectors and the latent avatar parameter vectors.
11 . The media of claim 10 , wherein training the cross-modal mapping model comprises minimizing an alignment loss between cross-modal mapping model output vectors and the latent avatar parameter vectors.
12 . The media of claim 10 , wherein the trained cross-modal mapping model is configured to translate a latent image vector from a latent image vector space to a latent avatar parameter vector of a latent avatar parameter vector space.
13 . The media of claim 10 , wherein the cross-modal mapping model is trained using supervised learning.
14 . The media of claim 10 , wherein the cross-modal mapping model comprises a multilayer perceptron.
15 . A system for avatar generation, the system comprising:
at least one processor; and
one or more computer storage media storing computer readable instructions thereon that when executed by the at least one processor cause the at least one processor to perform operations comprising:
generating a latent image vector using a trained image space model, the latent image vector generated by the trained image space model responsive to an image input;
translating the latent image vector into a latent avatar parameter vector using a trained cross-modal mapping model; and
generating avatar parameter values using a trained avatar parameter space model, the avatar parameter values generated by the trained avatar parameter space model responsive to a latent avatar parameter vector input comprising the latent avatar parameter vector.
16 . The system of claim 15 , further comprising rendering an avatar from the avatar parameter values.
17 . The system of claim 15 , wherein:
the trained image space model comprises a latent image vector encoder; and
the trained avatar parameter space model comprises a latent avatar parameter vector decoder.
18 . The system of claim 15 , wherein:
the trained image space model is configured to generate the latent image vector based on unsupervised training from a set of images;
the trained cross-modal mapping model is configured to translate the latent image vector based on supervised training from image-avatar parameter pairs; and
the trained avatar parameter space model is configured to generate the avatar parameter values based on unsupervised training from a set of avatar parameter values for avatar parameters.
19 . The system of claim 15 , wherein:
the trained image space model comprises a convolutional neural network;
the trained avatar parameter space model comprises a first multilayer perceptron; and
the trained cross-modal mapping model comprises a second multilayer perceptron.
20 . The system of claim 15 , wherein the image input comprises a face, and the avatar parameter values define an avatar comprising facial features corresponding to the face.