Avatar animation system
Avatar animation systems disclosed herein provide high quality, real-time avatar animation that is based on the varying countenance of a human face. In some example embodiments, the real-time provision of high quality avatar animation is enabled at least in part, by a multi-frame regressor that is configured to map information descriptive of facial expressions depicted in two or more images to information descriptive of a single avatar blend shape. The two or more images may be temporally sequential images. This multi-frame regressor implements a machine learning component that generates the high quality avatar animation from information descriptive of a subject's face and/or information descriptive of avatar animation frames previously generated by the multi-frame regressor. The machine learning component may be trained using a set of training images that depict human facial expressions and avatar animation authored by professional animators to reflect facial expressions depicted in the set of training images.
1. An avatar animation system comprising: a memory; and at least one processor coupled to the memory and configured to: receive image data comprising three or more temporally sequential frames depicting three or more temporally sequential of facial expressions of a subject; identify a set of landmark points within each facial expression of the three or more temporally sequential facial expressions; generate a mesh based on each set of landmark points, thereby creating three or more temporally sequential meshes corresponding to the three or more temporally sequential facial expressions; generate a set of blend shape weights for a single frame of avatar animation at least in part by mapping, via a multi-frame regressor, the three or more temporally sequential meshes to the set of blend shape weights; and provide the set of blend shape weights to a user interface component, thereby causing the user interface component to render the single frame of avatar animation.
2. The avatar animation system of claim 1 , wherein the image data includes at least one of two-dimensional image data and three-dimensional image data.
3. The avatar animation system of claim 1 , wherein the at least one processor is configured to generate the set of blend shape weights at least in part by mapping, via the multi-frame regressor, the three or more temporally sequential meshes and at least one previously generated blend shape weight to the set of blend shape weights.
4. The avatar animation system of claim 1 , further comprising an avatar client component including the user interface component and configured to acquire the image data.
5. The avatar animation system of claim 1 , wherein the multi-frame regressor comprises an artificial neural network.
6. The avatar animation system of claim 5 , wherein the artificial neural network is configured to: process the three or more temporally sequential meshes via a plurality of input nodes; and generate the set of blend shape weights via a plurality, of output nodes.
7. The avatar animation system of claim 6 , wherein each input node of the plurality of input nodes is configured to receive either a coordinate value or a blend shape weight.
8. The avatar animation system of claim 6 , wherein each output node of the plurality of output nodes is configured to identify a blend shape weight.
9. A method of generating avatar animation using a system, the method comprising: receiving image data comprising three or more temporally sequential frames depicting three or more temporally sequential facial expressions of a subject; identifying a set of landmark points within each facial expression of the three or more temporally sequential facial expressions; generating a mesh based on each set of landmark points, thereby creating three or more temporally sequential meshes corresponding to the three or more temporally sequential facial expressions;
generating a set of blend shape weights for a single frame of avatar animation at least in part by mapping, via a multi-frame regressor, the three or more temporally sequential meshes to the set of blend shape weights; and providing the set of blend shape weights to a user interface component, thereby causing the user interface component to render the single frame of avatar animation.
10. The method of claim 9 , wherein generating the set of blend shape weights comprises mapping the three or more temporally sequential meshes and at least one previously generated blend shape weight to the set of blend shape weights.
11. The method of claim 9 , further comprising acquiring the image data.
12. The method of claim 9 , wherein mapping the three or more temporally sequential meshes includes mapping the three or more temporally sequential of meshes via an artificial neural network.
13. The method of claim 12 , wherein the artificial neural network includes a plurality of input nodes and a plurality of output nodes and the method further comprises: processing the three or more temporally sequential meshes via the plurality of input nodes; and generating the set of blend shape weights via the plurality of output nodes.
14. The method of claim 13 , further comprising receiving, at each input node of the plurality of input nodes, either a coordinate value or a blend shape weight.
15. The method of claim 13 , further comprising identifying, at each output node of the plurality of output nodes, a blend shape weight.
16. The method of claim 9 , wherein receiving the image data comprises receiving at least one of two-dimensional image data and three-dimensional image data.
17. A non-transient computer program product encoded with instructions that when executed by one or more processors cause a process of animating avatars to be carried out, the process comprising: receiving image data comprising three or more temporally sequential frames depicting three or more temporally sequential facial expressions of a subject; identifying a set of landmark points within each facial expression of the three or more temporally sequential facial expressions; generating a mesh based on each set of landmark points, thereby creating three or more temporally sequential meshes corresponding to the three or more temporally sequential facial expressions; generating a set of blend shape weights for a single frame of avatar animation at least in part by mapping, via a multi-frame regressor, the three or more temporally sequential meshes to the set of blend shape weights; and providing the set of blend shape weights to a user interface component, thereby causing the user interface component to render the single frame of avatar animation.
18. The computer program product of claim 17 , the process further comprising acquiring the image data.
19. The computer program product of claim 17 , wherein mapping the three or more temporally sequential meshes includes mapping the three or more temporally sequential meshes via an artificial neural network.
20. The computer program product of claim 19 , wherein the artificial neural network includes a plurality of input nodes and a plurality of output nodes and the process further comprises: processing the three or more temporally sequential meshes via the plurality of input nodes; and generating the set of blend shape weights via the plurality of output nodes.
21. The computer program product of claim 20 , the process further comprising receiving, at each input node of the plurality of input nodes, either a coordinate value or a blend shape weight.
22. The computer program product of claim 20 , the process further comprising identifying, at each output node of the plurality of output nodes, a blend shape weight.
23. The computer program product of claim 17 , wherein generating the set of blend shape weights comprises mapping the three or more temporally sequential meshes and at least one previously generated blend shape weight to the set of blend shape weights.
24. The computer program product of claim 17 , wherein the image data includes at least one of two-dimensional image data and three-dimensional image data.