Face image generation method and device for generating fully-controllable talking face
Provided are face image generation method and device for generating a controllable talking face image. The method includes: a face image generation method for generating a controllable talking face image, the method comprising: receiving a source image and a series of driving images, sampled from the same video, and input audio; acquiring a style latent code including a source latent code and a driving latent code by encoding the source image and the series of driving images into a visual space by a visual encoder; acquiring an audio feature including an audio latent code by encoding the input audio by an audio encoder; acquiring a canonical code by mapping the source latent code to a canonical space by a canonical encoder; acquiring a motion code by combining the driving latent code with the audio latent code, and mapping the combined code to a multimodal motion space by a multimodal motion encoder; acquiring a multimodal fused latent code by combining the canonical code with the motion code; and generating a talking face image by transferring the multimodal fused latent code to a generative adversarial network (GAN).
1 . A face image generation method for generating a controllable talking face image, the method comprising:
receiving a source image and a series of driving images, sampled from the same video, and input audio;
acquiring a style latent code including a source latent code and a driving latent code by encoding the source image and the series of driving images into a visual space by a visual encoder;
acquiring an audio feature including an audio latent code by encoding the input audio by an audio encoder;
acquiring a canonical code by mapping the source latent code to a canonical space by a canonical encoder;
acquiring a motion code by combining the driving latent code with the audio latent code, and mapping the combined code to a multimodal motion space by a multimodal motion encoder;
acquiring a multimodal fused latent code by combining the canonical code with the motion code; and
generating a talking face image by transferring the multimodal fused latent code to a generative adversarial network (GAN).
2 . The method of claim 1 , wherein
in the acquiring of the multimodal fused latent code,
the multimodal fused latent code is acquired by combining the canonical code with the motion code by a linear operation.
3 . The method of claim 2 , wherein
in the acquiring of the multimodal fused latent code,
the multimodal fused latent code is acquired through 2-stage motion transfer according to Equation 1 below:
z
s
→
d
=
z
s
→
c
+
z
c
→
d
(
Equation
1
)
where z s→d is the multimodal fused latent code, z s→c is the canonical code and represents conversion from the source image to the canonical image, and z c→d is the motion code and represents conversion from the canonical image to the series of driving images.
4 . The method of claim 1 , wherein
in the acquiring of the motion code,
the motion code is acquired according to Equation 2 below in order for the combined code to include only a motion feature when combining the driving latent code with the audio latent code:
z
c
→
d
=
E
m
(
E
a
(
x
a
)
⊕
E
inv
(
x
d
)
)
(
Equation
2
)
where z c→d is the motion code, E m is the multimodal motion encoder, E a is the audio encoder, E inv is the visual encoder, x a is the input audio, x d represents the series of driving images, and ⊕ represents channel-wise concatenation.
5 . The method of claim 1 , further comprising
imposing a constraint to ensure orthogonality in which an identity feature and a motion feature are distinguished from each other between the canonical code and the motion code.
6 . The method of claim 5 , wherein
in the imposing of the constraint,
orthogonality loss is introduced according to Equation 3 below to ensure the orthogonality:
L
ortho
=
1
N
∑
(
z
s
→
c
⊙
z
c
→
d
)
(
Equation
3
)
where N is the number of channels of the latent code, z s→c is the canonical code, z c→d is the motion code, and ⊙ represents Hadamard product.
7 . The method of claim 1 , wherein
the canonical encoder includes two multilayer perceptrons (MLPs).
8 . The method of claim 1 , wherein
the multimodal motion encoder includes three multilayer perceptrons.
9 . The method of claim 1 , wherein
in the generating of the talking face image,
the talking face image is generated by transferring the multimodal fused latent code refined through a temporal fusion layer to the GAN.
10 . The method of claim 9 , wherein
the temporal fusion layer includes a one-dimensional convolution layer.
11 . A face image generation device for generating a controllable talking face image, which executes a program code loaded in at least one memory device by at least one processor,
wherein the program code is executed to
receive a source image and a series of driving images, sampled from the same video, and input audio,
acquire a style latent code including a source latent code and a driving latent code by encoding the source image and the series of driving images into a visual space by a visual encoder,
acquire an audio feature including an audio latent code by encoding the input audio by an audio encoder,
acquire a canonical code by mapping the source latent code to a canonical space by a canonical encoder,
acquire a motion code by combining the driving latent code with the audio latent code, and mapping the combined code to a multimodal motion space by a multimodal motion encoder,
acquire a multimodal fused latent code by combining the canonical code with the motion code, and
generate a talking face image by transferring the multimodal fused latent code to a generative adversarial network (GAN).
12 . The device of claim 11 , wherein
to acquire the multimodal fused latent code includes
that the multimodal fused latent code is acquired by combining the canonical code with the motion code by a linear operation.
13 . The device of claim 12 , wherein
to acquire the multimodal fused latent code includes
that the multimodal fused latent code is acquired through 2-stage motion transfer according to Equation 1 below:
z
s
→
d
=
z
s
→
c
+
z
c
→
d
(
Equation
1
)
where z s→d is the multimodal fused latent code, z s→c is the canonical code and represents conversion from the source image to the canonical image, and z c→d is the motion code and represents conversion from the canonical image to the series of driving images.
14 . The device of claim 11 , wherein
to acquire the motion code includes
that the motion code is acquired according to Equation 2 below in order for the combined code to include only a motion feature when combining the driving latent code with the audio latent code:
z
c
→
d
=
E
m
(
E
a
(
x
a
)
⊕
E
inv
(
x
d
)
)
(
Equation
2
)
where z c→d is the motion code, E m is the multimodal motion encoder, E a is the audio encoder, E inv is the visual encoder, x a is the input audio, x d represents the series of driving images, and ⊕ represents channel-wise concatenation.
15 . The device of claim 11 , wherein
the program code is executed
to impose a constraint to ensure orthogonality in which an identity feature and a motion feature are distinguished from each other between the canonical code and the motion code.
16 . The device of claim 15 , wherein
to impose the constraint includes
that orthogonality loss is introduced according to Equation 3 below to ensure the orthogonality:
L
ortho
=
1
N
∑
(
z
s
→
c
⊙
z
c
→
d
)
(
Equation
3
)
where N is the number of channels of the latent code, z s→c is the canonical code, z c→d is the motion code, and ⊙ represents Hadamard product.
17 . The device of claim 11 , wherein
the canonical encoder includes two multilayer perceptrons (MLPs).
18 . The device of claim 11 , wherein
the multimodal motion encoder includes three multilayer perceptrons.
19 . The device of claim 11 , wherein
to generate the talking face image includes
that the talking face image is generated by transferring the multimodal fused latent code refined through a temporal fusion layer to the GAN.
20 . The device of claim 19 , wherein
the temporal fusion layer includes a one-dimensional convolution layer.