Real-time avatars using dynamic textures

1. A system for generating real-time facial avatars, the system comprising a processor and memory, the processor executing instructions which cause the processor to:

train a generative adversarial neural network for application in creating a set of predetermined key expressions for an input image by causing the processor to:

employ a generative network within the generative adversarial neural network to iteratively create facial textures;

employ a discriminator network within the generative adversarial neural network to iteratively approve or refuse the facial textures as adequately corresponding to a training data set or not adequately corresponding to the training data set; and

apply the generative adversarial neural network to create the set of predetermined key expressions after training is complete; and

generate a facial avatar by causing the processor to:

receive a single image of a human face in a neutral facial pose;

generate a morphable three-dimensional model of the human face and a texture map based upon the single image;

generate a set of created images for the human face from the texture map, each of the created images representing a predetermined facial expression;

generate blendshape coefficients and texture maps for each of the set of created images to thereby create the set of predetermined key expressions; and

generate a series of facial action coding system (FACS) texture maps corresponding to a predetermined set of FACS expressions and an activation mask comprising a per-vertex deformation relative to the neutral facial pose for each desired FACS based upon the predetermined key expressions.

2. The system of claim 1 , wherein the predetermined set of FACS expressions are transmitted to a mobile device including a second processor and a second memory, the second processor executing instructions which cause the second processor to:

store the predetermined set of FACS expressions;

receive data representing a facial pose to synthesize for the human face;

generate a synthesized three-dimensional model of the facial pose and applying a facial texture linearly combined between the nearest two of the predetermined set of FACS expressions to the three-dimensional model; and

output the combined synthesized three-dimensional model with the facial texture.

3. The system of claim 2 :

the processor executing instructions which cause the processor to

pre-calculate a set of eye textures for use in conjunction with the facial texture and the three-dimensional model, each of the set of eye textures corresponding to a particular orientation of an iris, and

transmit the set of eye textures and the three-dimensional model to the mobile device; and

the second processor executing instructions which cause the second processor to

linearly combine at least two of the set of eye textures that best correspond to the facial pose to thereby create composite eye textures, and

apply the composite eye textures to the three-dimensional model.

4. The system of claim 2 :

the processor executing instructions which cause the processor to

pre-generate a set of mouth textures in multiple geometric configurations for the human face, and

transmit the set of mouth textures and the three-dimensional model to the mobile device; and

the second processor executing instructions which cause the second processor to

select a subset of the set of mouth textures that is closest to a mouth visible in the facial pose,

perform per-pixel weighting of the subset of the set of mouth textures to generate a synthesized mouth texture,

apply a sharpening filter to the synthesized mouth texture to generate a sharpened synthesized mouth texture, and

apply the sharpened synthesized mouth texture to the three-dimensional model.

5. The system of claim 2 wherein receipt of the facial pose includes completing a translation of a received two-dimensional image into a set of blendshape coefficients.

6. The system of claim 2 wherein the second processor operates to repeat the instructions for multiple frames of captured video, with each frame representing the facial pose.

7. A method of generating real-time facial avatars comprising:

employing a generative network within the generative adversarial neural network to iteratively create facial textures;

employing a discriminator network within the generative adversarial neural network to iteratively approve or refuse the facial textures as adequately corresponding to a training data set or not adequately corresponding to the training data set;

applying the generative adversarial neural network to create a set of predetermined key expressions after training is complete;

receiving a single image of a human face in a neutral facial pose;

generating a morphable three-dimensional model of the human face and a texture map based upon the single image;

generating a set of created images for the human face from the texture map, each of the created images representing a predetermined facial expression;

generating blendshape coefficients and texture maps for each of the set of created images to thereby create the set of predetermined key expressions; and

generating a series of facial action coding system (FACS) texture maps corresponding to a predetermined set of FACS expressions and an activation mask comprising a per-vertex deformation relative to the neutral facial pose for each desired FACS based upon the predetermined key expressions.

8. The method of claim 7 further comprising:

storing the predetermined set of FACS expressions;

receiving data representing a facial pose to synthesize for the human face;

generating a synthesized three-dimensional model of the facial pose and applying a facial texture linearly combined between the nearest two of the predetermined set of FACS expressions to the three-dimensional model; and

outputting the combined synthesized three-dimensional model with the facial texture.

9. The method of claim 8 further comprising:

pre-calculating a set of eye textures for use in conjunction with the facial texture and the three-dimensional model, each of the set of eye textures corresponding to a particular orientation of an iris;

transmitting the set of eye textures and the three-dimensional model to a mobile device;

linearly combining at least two of the set of eye textures that best correspond to the facial pose to thereby create composite eye textures; and

applying the composite eye textures to the three-dimensional model.

10. The method of claim 8 further comprising:

pre-generating a set of mouth textures in multiple geometric configurations for the human face;

transmitting the set of mouth textures and the three-dimensional model to a mobile device;

selecting a subset of the set of mouth textures that is closest to a mouth visible in the facial pose;

performing per-pixel weighting of the subset of the set of mouth textures to generate a synthesized mouth texture;

applying a sharpening filter to the synthesized mouth texture to generate a sharpened synthesized mouth texture; and

applying the sharpened synthesized mouth texture to the three-dimensional model.

11. The method of claim 8 wherein receipt of the facial pose includes completing a translation of a received two-dimensional image into a set of blendshape coefficients.

12. The method of claim 8 further comprising repeating the steps of claim 8 for multiple frames of captured video, with each frame representing the facial pose.

13. An apparatus comprising a storage medium storing a program having instructions which when executed by a processor will cause the processor to:

train a generative adversarial neural network for application in creating a set of predetermined key expressions for an input image by causing the processor to:

employ a generative network within the generative adversarial neural network to iteratively create facial textures;

employ a discriminator network within the generative adversarial neural network to iteratively approve or refuse the facial textures as adequately corresponding to a training data set or not adequately corresponding to the training data set; and

apply the generative adversarial neural network to create the set of predetermined key expressions after training is complete; and

generate a facial avatar by causing the processor to:

receive a single image of a human face in a neutral facial pose;

generate a morphable three-dimensional model of the human face and a texture map based upon the single image;

generate a set of created images for the human face from the texture map, each of the created images representing a predetermined facial expression;

generate blendshape coefficients and texture maps for each of the set of created images to thereby create the set of predetermined key expressions; and

generate a series of facial action coding system (FACS) texture maps corresponding to a predetermined set of FACS expressions and an activation mask comprising a per-vertex deformation relative to the neutral facial pose for each desired FACS based upon the predetermined key expressions.

14. The apparatus of claim 13 , wherein the instructions further cause the processor to:

store the predetermined set of FACS expressions;

receive data representing a facial pose to synthesize for the human face;

generate a synthesized three-dimensional model of the facial pose and applying a facial texture linearly combined between the nearest two of the predetermined set of FACS expressions to the three-dimensional model; and

output the combined synthesized three-dimensional model with the facial texture.

15. The apparatus of claim 14 wherein the instructions further cause the processor to:

pre-calculate a set of eye textures for use in conjunction with the facial texture and the three-dimensional model, each of the set of eye textures corresponding to a particular orientation of an iris;

transmit the set of eye textures and the three-dimensional model to a mobile device; and

linearly combine at least two of the set of eye textures that best correspond to the facial pose to thereby create composite eye textures; and

apply the composite eye textures to the three-dimensional model.

16. The apparatus of claim 14 wherein the instructions further cause the processor to:

pre-generate a set of mouth textures in multiple geometric configurations for the human face;

transmit the set of mouth textures and the three-dimensional model to a mobile device;

select a subset of the set of mouth textures that is closest to a mouth visible in the facial pose;

perform per-pixel weighting of the subset of the set of mouth textures to generate a synthesized mouth texture;

apply a sharpening filter to the synthesized mouth texture to generate a sharpened synthesized mouth texture; and

apply the sharpened synthesized mouth texture to the three-dimensional model.

17. The apparatus of claim 13 further comprising:

the processor;

a memory; and

wherein the processor and the memory comprise circuits and software for performing the instructions on the storage medium.