System apparatus and method for providing facial expression to avatars

1. A system for providing a facial expression to one or more video frames of a virtual avatar comprising:

a training system configured to train a neural network system to replace a face of the virtual avatar with a source face and to provide a facial expression of the source face to the face of the virtual avatar in real-time; and

an inference system configured to use the trained neural network system to provide one or more facial expressions of the source face to the face of the avatar in real-time to cause the one or more facial expressions of the avatar to imitate the one or more facial expressions of a source face media which is represented by the virtual avatar by reconstructing a three dimensional (3D) model of a face of the virtual avatar, animate the face of the virtual avatar by reconstructing geometry of the 3D model, render the animated face of the virtual avatar and apply to the rendered virtual avatar the trained neural network system in a post-processing phase.

2. The system of claim 1 , wherein the training of the neural network system comprises:

train at least one neural network of the neural network system to replace a face of at least one image received from a first database with at least one source face image received from a second database to generate the trained neural network system.

3. The system of claim 2 , wherein the first database comprises a plurality of videos and images of faces.

4. The system of claim 2 , wherein the second database comprises a plurality of videos and images of faces that are configured to be used as source faces.

5. The system of claim 2 , wherein the inference system is configured to:

use the trained neural network system to replace a face in a target image from the first database with a source face image from the second database, add facial expressions to the source face image and to provide the source face with facial expressions on the virtual avatar.

6. The system of claim 2 , wherein images of the first database and images of the second database comprise video with one or more frames.

7. The system of claim 2 , wherein the inference system is configured to:

use the trained neural network system to replace a face in a target image from the first database with a predetermined face image from the second database to provide the source face with facial expressions on the virtual avatar.

8. The system of claim 2 comprises a third database, wherein the third database comprises a plurality of images that serve as facial expressions.

9. The system of claim 8 , wherein the training of the neural network system comprises:

train at least one neural network of the neural network system to replace a face of at least one image received from the first database with at least one source face image received from the second database and images received from the third database to drive face expressions to generate a trained neural network system.

10. The system of claim 9 , wherein the inference system is configured to:

use the trained neural network system to replace a face in a target image from the first database with a source face image from the second database, and

use the plurality of images from the third database to drive face expressions to the source face of the virtual avatar.

11. A method for providing a facial expression to one or more video frames of a virtual avatar processed by a neural network system comprising:

training a neural network system to replace a face of the virtual avatar with a source face and to provide a facial expression of the source face to the face of the avatar in a real-time at a training phase using a neural network training system; and

providing by an inference system configured to use the trained neural network system to one or more facial expressions of the source face to the face of the avatar in real-time to cause the one or more facial expressions of the avatar to imitate the one or more facial expressions of a source face media which is represented by the virtual avatar by reconstructing a three dimensional (3D) model of a face of the virtual avatar, animating the face of the virtual avatar by reconstructing geometry of the 3D model, rendering the animated face of the virtual avatar and apply to the rendered virtual avatar the trained neural network system in a post-processing phase.

12. The method of claim 11 , wherein the training of the neural network system comprises:

at the training phase, training at least one neural network of the neural network system to replace a face of at least one image received from a first database with at least one source face image received from a second database; and

generating the trained neural network system.

13. The method of claim 12 , wherein the first database comprises a plurality of videos and images of faces.

14. The method of claim 12 , wherein the second database comprises a plurality of videos and images of faces that are configured to be used as source faces.

15. The method of claim 12 , processed by the inference system, comprises:

using the trained neural network system to replace a face in a target image from the first database with a source face image from the second database, add facial expressions to the source face image and to provide the source face with facial expressions on the virtual avatar.

16. The method of claim 12 , wherein images of the first database and images of the second database comprise video with one or more frames.

17. The method of claim 12 , proposed by the inference system, comprises:

using the trained neural network system to replace a face in a target image from the first database with a predetermined face image from the second database to provide the source face with facial expressions on the virtual avatar.

18. The method of claim 12 , wherein the system comprises a third database containing a plurality of images that serve as facial expressions.

19. The method of claim 18 , wherein the training of the neural network system comprises:

training at least one neural network of the neural network system to replace a face of at least one image received from the first database with at least one source face image received from the second database and images received from the third database to drive face expressions to generate a trained neural network system.

20. The method of claim 19 , processed by the inference system, comprises:

using the trained neural network system to replace a face in a target image from the first database with a source face image from the second database, and

using the plurality of images from the third database to drive face expressions to the source face of the virtual avatar.