Method and an apparatus for generating a 3D face comprising at least one deformed region
A method and an apparatus for generating a 3D face comprising at least one region deformed according to a deformation style are provided. 3D data representative of at least one region of a first 3D face is provided as input to a neural network-based geometry deformation generator. Data representative of the deformation style is provided as input to the neural network-based geometry deformation generator. The 3D face comprising the at least one deformed region is obtained from the neural network-based geometry deformation generator, wherein the at least one deformed region includes geometry deformations representative of the deformation style.
1 . A method for generating a three-dimensional (3D) face comprising at least one region deformed according to a deformation style, comprising:
providing 3D data representative of at least one region of a first 3D face as input to a neural network-based geometry deformation generator;
providing a style code representative of the deformation style as input to the neural network-based geometry deformation generator; and
obtaining from an output of the neural network-based geometry deformation generator, the 3D face comprising the at least one deformed region wherein the at least one deformed region includes geometry deformations representative of the deformation style.
2 . The method of claim 1 , wherein the style code is obtained as an output of a neural network-based style encoder taking as input 3D data representative of the deformation style.
3 . The method of claim 2 , further comprising transmitting the style code.
4 . The method of claim 2 , wherein at least one of a neural network-based content encoder of the neural network-based geometry deformation generator, a neural network-based decoder of the neural network-based geometry deformation generator or the neural network-based style encoder comprises a set of convolutional layers, and wherein at least one of the layers uses a 3D convolutional operator.
5 . The method of claim 4 , wherein the 3D convolutional operator is based on a spiral path determined for each vertex of the 3D mesh provided as input to the at least one layer, the spiral path comprising a number of neighboring vertices of the vertex to which a convolutional kernel is applied in the at least one layer.
6 . The method of claim 1 , wherein the style code is obtained from a second 3D face.
7 . The method of claim 1 , wherein the neural network-based geometry deformation generator comprises:
a neural network-based content encoder taking as input the 3D data representative of the at least one region of the first 3D face and outputting a content code representative of the at least one region of the first 3D face that is style-invariant; and
a neural network-based decoder taking as input the content code and the style code, the neural network-based decoder outputting the 3D face comprising the at least one region deformed with the deformation style.
8 . The method of claim 7 , further comprising transmitting the content code.
9 . The method of claim 1 , wherein the 3D data representative of the at least one region of the first 3D face or the 3D data representative of the deformation style is a 3D mesh.
10 . The method of claim 9 , wherein the neural network-based geometry deformation generator is trained using at least one of the following loss including combinations thereof:
an adversarial loss using a discriminator that provides a binary classification indicating whether the input 3D mesh of the first 3D face is real mesh having its corresponding style or a 3D mesh produced by the neural network-based geometry deformation generator;
a cycle consistency loss guaranteeing that the output of the neural network-based geometry deformation generator preserves a style invariant characteristic of the input 3D mesh of the first 3D face when applying a style deformation distinct from the style deformation of the input 3D mesh of the first 3D face; or
a reconstruction loss preserving the style invariant characteristic of the input 3D mesh of the first 3D face when applying the style code obtained from a neural network-based style encoder using as input the 3D mesh of the first 3D face.
11 . A non-transitory computer readable storage medium having stored thereon instructions for causing one or more processors to perform the method of claim 1 .
12 . An apparatus for generating a three-dimensional (3D) face comprising at least one region deformed according to a deformation style, comprising one or more processors configured for:
providing 3D data representative of at least one region of a first 3D face as input to a neural network-based geometry deformation generator;
providing a style code representative of the deformation style as input to the neural network-based geometry deformation generator; and
obtaining from an output of the neural network-based geometry deformation generator, the 3D face comprising the at least one deformed region wherein the at least one deformed region includes geometry deformations representative of the deformation style.
13 . The apparatus of claim 12 , wherein the style code is obtained as an output of a neural network-based style encoder taking as input 3D data representative of the deformation style.
14 . The apparatus of claim 13 , wherein the neural network-based geometry deformation generator comprises:
a neural network-based content encoder taking as input the 3D data representative of the at least one region of the first 3D face and outputting a content code representative of the at least one region of the first 3D face that is style-invariant; and
a neural network-based decoder taking as input the content and the style code, the neural network-based decoder outputting the 3D face comprising the at least one region deformed with the deformation style.
15 . The apparatus of claim 14 , wherein the one or more processors are further configured for transmitting the content code.
16 . The apparatus of claim 14 , wherein at least one of the neural network-based content encoder, the neural network-based decoder or the neural network-based style encoder comprises a set of convolutional layers, and wherein at least one of the layers uses a 3D convolutional operator.
17 . The apparatus of claim 16 , wherein the 3D convolutional operator is based on a spiral path determined for each vertex of the 3D mesh provided as input to the at least one layer, the spiral path comprising a number of neighboring vertices of the vertex to which a convolutional kernel is applied in the at least one layer.
18 . The apparatus of claim 12 , wherein the style code is obtained from a second 3D face.
19 . The apparatus of claim 12 , wherein the 3D data representative of the at least one region of the first 3D face or the 3D data representative of the deformation style is a 3D mesh.
20 . The apparatus of claim 12 , wherein the one or more processors are further configured for transmitting the style code.