3D face modeling based on neural networks

1. An electronic device, comprising:

circuitry configured to:

receive a two-dimensional (2D) color image of a human face with a first face expression;

obtain a first three-dimensional (3D) mesh of the human face with the first face expression, based on the received 2D color image;

generate first texture information for the obtained first 3D mesh based on the received 2D color image;

generate a first set of displacement maps based on the obtained first 3D mesh, wherein

the first set of displacement maps comprises a first displacement map along a first orthogonal axis, a second displacement map along a second orthogonal axis, and a third displacement map along a third orthogonal axis, and

each of the first set of displacement maps comprises geometric information associated with vertices on a surface of the first 3D mesh along a corresponding orthogonal axis;

obtain a mean 3D mesh of the human face with the first face expression, wherein

the geometric information of the first displacement map includes a difference between a vertex position on the first 3D mesh along the first orthogonal axis and a corresponding vertex position on the mean 3D mesh along the first orthogonal axis;

feed an input comprising the generated first texture information and the generated first set of displacement maps to a neural network, which is trained on an image-to-image translation task;

receive an output of the neural network for the fed input, wherein the received output comprises second texture information and a second set of displacement maps;

deform the obtained first 3D mesh of the human face based on UV mapping the second set of displacement maps onto the first 3D mesh; and

generate, based on the deformed first 3D mesh of the human face, a second 3D mesh of the human face with a second face expression which is different from the first face expression.

2. The electronic device according to claim 1 , wherein the first face expression is a neutral face expression, in which face muscles are in a relaxed position and are visibly devoid of an emotional expression.

3. The electronic device according to claim 1 , wherein the second face expression is a face expression, which is different from a neutral face expression and indicates one of a plurality of emotional states.

4. The electronic device according to claim 1 , wherein the circuitry is further configured to:

calculate a residual 3D mesh based on subtraction of the obtained mean 3D mesh from the obtained first 3D mesh; and

generate the first set of displacement maps based on a UV unwrapping of the calculated residual 3D mesh.

5. The electronic device according to claim 1 , wherein the neural network is an image-to-image Generative Adversarial Network (GAN).

6. The electronic device according to claim 1 , wherein the circuitry is further configured to:

texture the generated second 3D mesh based on the second texture information; and

render the textured second 3D mesh as a first blend-shape.

7. The electronic device according to claim 6 , wherein the circuitry is further configured to render a blend-shape-based facial animation of the human face based on application of a blending operation on the first blend-shape and a second blend-shape consisting of the first 3D mesh.

8. The electronic device according to claim 1 , wherein the circuitry is further configured to:

generate a training dataset comprising a plurality of training samples, each of which comprises a first pair of displacement maps and texture information corresponding to a neutral face expression and a second pair of displacement maps and texture information corresponding to a face expression different from the neutral face expression; and

train the neural network on each of the plurality of training samples of the generated training dataset.

9. The electronic device according to claim 8 , wherein the generated training dataset comprises the first set of displacement maps, the first texture information, the second set of displacement maps, and the second texture information as a first training sample of the plurality of training samples of the training dataset.

10. A method, comprising:

receiving a two-dimensional (2D) color image of a human face with a first face expression;

obtaining a first three-dimensional (3D) mesh of the human face with the first face expression, based on the received 2D color image;

generating first texture information for the obtained first 3D mesh based on the received 2D color image;

generating a first set of displacement maps based on the obtained first 3D mesh, wherein

the first set of displacement maps comprises a first displacement map along a first orthogonal axis, a second displacement map along a second orthogonal axis, and a third displacement map along a third orthogonal axis, and

each of the first set of displacement maps comprises geometric information associated with vertices on a surface of the first 3D mesh along a corresponding orthogonal axis;

obtaining a mean 3D mesh of the human face with the first face expression, wherein

the geometric information of the first displacement map includes a difference between a vertex position on the first 3D mesh along the first orthogonal axis and a corresponding vertex position on the mean 3D mesh along the first orthogonal axis;

feeding an input comprising the generated first texture information and the generated first set of displacement maps to a neural network, which is trained on an image-to-image translation task;

receiving an output of the neural network for the fed input, wherein the received output comprises second texture information and a second set of displacement maps;

deforming the obtained first 3D mesh of the human face based on UV mapping the second set of displacement maps onto the first 3D mesh; and

generating, based on the deformed first 3D mesh of the human face, a second 3D mesh of the human face with a second face expression which is different from the first face expression.

11. The method according to claim 10 , further comprising:

calculating a residual 3D mesh based on subtraction of the obtained mean 3D mesh from the obtained first 3D mesh; and

generating the first set of displacement maps based on a UV unwrapping of the calculated residual 3D mesh.

12. The method according to claim 10 , wherein the neural network is an image-to-image Generative Adversarial Network (GAN).

13. The method according to claim 10 , further comprising:

texturing the generated second 3D mesh based on the second texture information; and

rendering the textured second 3D mesh as a first blend-shape.

14. The method according to claim 13 , further comprising rendering a blend-shape-based facial animation of the human face based on application of a blending operation on the first blend-shape and a second blend-shape consisting of the first 3D mesh.

15. The method according to claim 11 , further comprising:

generating a training dataset comprising a plurality of training samples, each of which comprises a first pair of displacement maps and texture information corresponding to a neutral face expression and a second pair of displacement maps and texture information corresponding to a face expression different from the neutral face expression; and

training the neural network on each of the plurality of training samples of the generated training dataset.

16. A non-transitory computer-readable storage medium configured to store instructions that, in response to being executed, causes an electronic device to perform operations, the operations comprising:

receiving a two-dimensional (2D) color image of a human face with a first face expression;

obtaining a first three-dimensional (3D) mesh of the human face with the first face expression, based on the received 2D color image;

generating first texture information for the obtained first 3D mesh based on the received 2D color image;

generating a first set of displacement maps based on the obtained first 3D mesh, wherein

the first set of displacement maps comprises a first displacement map along a first orthogonal axis, a second displacement map along a second orthogonal axis, and a third displacement map along a third orthogonal axis, and

each of the first set of displacement maps comprises geometric information associated with vertices on a surface of the first 3D mesh along a corresponding orthogonal axis;

obtaining a mean 3D mesh of the human face with the first face expression, wherein

the geometric information of the first displacement map includes a difference between a vertex position on the first 3D mesh along the first orthogonal axis and a corresponding vertex position on the mean 3D mesh along the first orthogonal axis;

feeding an input comprising the generated first texture information and the generated first set of displacement maps to a neural network, which is trained on an image-to-image translation task;

receiving an output of the neural network for the fed input, wherein the received output comprises second texture information and a second set of displacement maps;

deforming the obtained first 3D mesh of the human face based on UV mapping the second set of displacement maps onto the first 3D mesh; and

generating, based on the deformed first 3D mesh of the human face, a second 3D mesh of the human face with a second face expression which is different from the first face expression.

17. The non-transitory computer-readable storage medium according to claim 16 , wherein the neural network is an image-to-image Generative Adversarial Network (GAN).