Method and device for synthesizing talking head video and computer-readable storage medium
A method for synthesizing a talking head video includes: obtaining speech data to be synthesized and observation data, wherein the observation data is data obtained through observation other than the speech data; performing feature extraction on the speech data to obtain speech features corresponding to the speech data, and performing feature extraction on the observation data to obtain non-speech features corresponding to the observation data; performing temporal modeling on the speech features and first non-speech features to obtain low-dimensional representations, wherein the first non-speech features are non-speech features that are sensitive to temporal changes; and performing video synthesis based on the low-dimensional representations and second non-speech features, wherein the second non-speech features are non-speech features insensitive to temporal changes.
1 . A computer-implemented method for synthesizing a talking head video, the method comprising:
obtaining speech data to be synthesized and observation data, wherein the observation data is data obtained through observation other than the speech data;
performing feature extraction on the speech data to obtain speech features corresponding to the speech data, and performing feature extraction on the observation data to obtain non-speech features corresponding to the observation data;
performing temporal modeling on the speech features and first non-speech features to obtain low-dimensional representations, wherein the first non-speech features are non-speech features with first sensitivity to temporal changes in the non-speech features corresponding to the observation data, and are set as shape-related features; and
performing video synthesis based on the low-dimensional representations and second non-speech features, wherein the second non-speech features are non-speech features with second sensitivity to temporal changes in the non-speech features corresponding to the observation data, and are set as texture-related features, and wherein the first sensitivity is greater than the second sensitivity;
wherein performing the temporal modeling on the speech features and the first non-speech features to obtain the low-dimensional representations comprises:
fusing the speech features and the first non-speech features; and
obtaining the low-dimensional representations by using an autoregressive temporal model to perform the temporal modeling based on the fused speech features and the first non-speech features starting from frame 0, wherein appearance-related features are used as one of the inputs of the autoregressive temporal model; and
wherein performing video synthesis based on the low-dimensional representations and second non-speech features comprises:
obtaining multiple low-dimensional representations through the autoregressive temporal modeling; and
inferring the multiple low-dimensional representations simultaneously through a neural network to obtain multiple talking head video frames at the same time, wherein the neural network comprises a deconvolution network for mapping the low-dimensional representations to a high-dimensional image domain.
2 . The method of claim 1 , wherein the observation data comprises head movement observation data expressed through video, and performing feature extraction on the observation data to obtain non-speech features corresponding to the observation data comprises:
analyzing the video to obtain information on relative positional changes between a human face and a camera angle; and
obtaining third non-speech features expressing a position of the human face based on the information.
3 . The method of claim 1 , wherein the observation data comprises anatomical facial observation data expressed through images, and performing feature extraction on the observation data to obtain non-speech features corresponding to the observation data comprises:
based on a correlation with appearance, separating the anatomical facial observation data to obtain fourth non-speech features that express facial appearance.
4 . The method of claim 1 , wherein the speech features comprise linguistic features and paralinguistic features;
wherein the linguistic features refer to content-related information contained in the speech data; and
wherein the paralinguistic features refer to information contained in the speech data that is independent of the content, and are related to facial expressions and head motion when speaking.
5 . The method of claim 1 , further comprising, before performing temporal modeling on the speech features and first non-speech features to obtain the low-dimensional representations,
aligning the speech features and the non-speech features on a time dimension.
6 . The method of claim 5 , wherein aligning the speech features and the non-speech features on the time dimension comprises:
on the time dimension, aligning the speech features and the non-speech features by downsampling the speech features or upsampling the non-speech features.
7 . The method of claim 1 , wherein the observation data is obtained based on a video of a real head of a desired target with a visual sensor, and the video of the real head comprises: data related to head movement, or, data related to appearance and the data related to head movement.
8 . An electronic device for synthesizing a talking head video, comprising:
one or more processors; and
a memory coupled to the one or more processors, the memory storing programs that, when executed by the one or more processors, cause performance of operations comprising:
obtaining speech data to be synthesized and observation data, wherein the observation data is data obtained through observation other than the speech data;
performing feature extraction on the speech data to obtain speech features corresponding to the speech data, and performing feature extraction on the observation data to obtain non-speech features corresponding to the observation data;
performing temporal modeling on the speech features and first non-speech features to obtain low-dimensional representations, wherein the first non-speech features are non-speech features with first sensitivity to temporal changes in the non-speech features corresponding to the observation data, and are set as shape-related features; and
performing video synthesis based on the low-dimensional representations and second non-speech features, wherein the second non-speech features are non-speech features with second sensitivity to temporal changes in the non-speech features corresponding to the observation data, and are set as texture-related features, and wherein the first sensitivity is greater than the second sensitivity;
wherein performing the temporal modeling on the speech features and the first non-speech features to obtain the low-dimensional representations comprises:
fusing the speech features and the first non-speech features; and
obtaining the low-dimensional representations by using an autoregressive temporal model to perform the temporal modeling based on the fused speech features and the first non-speech features starting from frame 0, wherein appearance-related features are used as one of the inputs of the autoregressive temporal model; and
wherein performing video synthesis based on the low-dimensional representations and second non-speech features comprises:
obtaining multiple low-dimensional representations through the autoregressive temporal modeling; and
inferring the multiple low-dimensional representations simultaneously through a neural network to obtain multiple talking head video frames at the same time, wherein the neural network comprises a deconvolution network for mapping the low-dimensional representations to a high-dimensional image domain.
9 . The electronic device of claim 8 , wherein the observation data comprises head movement observation data expressed through video, and performing feature extraction on the observation data to obtain non-speech features corresponding to the observation data comprises:
analyzing the video to obtain information on relative positional changes between a human face and a camera angle; and
obtaining third non-speech features expressing a position of the human face based on the information.
10 . The electronic device of claim 8 , wherein the observation data comprises anatomical facial observation data expressed through images, and performing feature extraction on the observation data to obtain non-speech features corresponding to the observation data comprises:
based on a correlation with appearance, separating the anatomical facial observation data to obtain fourth non-speech features that express facial appearance.
11 . The electronic device of claim 8 , wherein the speech features comprise linguistic features and paralinguistic features.
12 . The electronic device of claim 8 , wherein the operations further comprise, before performing temporal modeling on the speech features and first non-speech features to obtain the low-dimensional representations,
aligning the speech features and the non-speech features on a time dimension.
13 . The electronic device of claim 12 , wherein aligning the speech features and the non-speech features on the time dimension comprises:
on the time dimension, aligning the speech features and the non-speech features by downsampling the speech features or upsampling the non-speech features.
14 . A non-transitory computer-readable storage medium storing instructions that, when executed by at least one processor of an electronic device, cause the at least one processor to perform a method, the method comprising:
obtaining speech data to be synthesized and observation data, wherein the observation data is data obtained through observation other than the speech data;
performing feature extraction on the speech data to obtain speech features corresponding to the speech data, and performing feature extraction on the observation data to obtain non-speech features corresponding to the observation data;
performing temporal modeling on the speech features and first non-speech features to obtain low-dimensional representations, wherein the first non-speech features are non-speech features that are sensitive to temporal changes in the non-speech features corresponding to the observation data, and are set as shape-related features; and
performing video synthesis based on the low-dimensional representations and second non-speech features in the non-speech features corresponding to the observation data, and are set as texture-related features, and wherein the second non-speech features are non-speech features insensitive to temporal changes;
wherein performing the temporal modeling on the speech features and the first non-speech features to obtain the low-dimensional representations comprises:
fusing the speech features and the first non-speech features; and
obtaining the low-dimensional representations by using an autoregressive temporal model to perform the temporal modeling based on the fused speech features and the first non-speech features starting from frame 0, wherein appearance-related features are used as one of the inputs of the autoregressive temporal model; and
wherein performing video synthesis based on the low-dimensional representations and second non-speech features comprises:
obtaining multiple low-dimensional representations through the autoregressive temporal modeling; and
inferring the multiple low-dimensional representations simultaneously through a neural network to obtain multiple talking head video frames at the same time, wherein the neural network comprises a deconvolution network for mapping the low-dimensional representations to a high-dimensional image domain.
15 . The non-transitory computer-readable storage medium of claim 14 , wherein the observation data comprises head movement observation data expressed through video, and performing feature extraction on the observation data to obtain non-speech features corresponding to the observation data comprises:
analyzing the video to obtain information on relative positional changes between a human face and a camera angle; and
obtaining third non-speech features expressing a position of the human face based on the information.
16 . The non-transitory computer-readable storage medium of claim 14 , wherein the observation data comprises anatomical facial observation data expressed through images, and performing feature extraction on the observation data to obtain non-speech features corresponding to the observation data comprises:
based on a correlation with appearance, separating the anatomical facial observation data to obtain fourth non-speech features that express facial appearance.
17 . The non-transitory computer-readable storage medium of claim 14 , wherein the speech features comprise linguistic features and paralinguistic features.
18 . The non-transitory computer-readable storage medium of claim 14 , wherein the method further comprises, before performing temporal modeling on the speech features and first non-speech features to obtain the low-dimensional representations,
aligning the speech features and the non-speech features on a time dimension.
19 . The non-transitory computer-readable storage medium of claim 18 , wherein aligning the speech features and the non-speech features on the time dimension comprises:
on the time dimension, aligning the speech features and the non-speech features by downsampling the speech features or upsampling the non-speech features.