Identity preserving realistic talking face generation using audio speech of a user
Speech-driven facial animation is useful for a variety of applications such as telepresence, chatbots, etc. The necessary attributes of having a realistic face animation are: 1) audiovisual synchronization, (2) identity preservation of the target individual, (3) plausible mouth movements, and (4) presence of natural eye blinks. Existing methods mostly address audio-visual lip synchronization, and synthesis of natural facial gestures for overall video realism. However, existing approaches are not accurate. Present disclosure provides system and method that learn motion of facial landmarks as an intermediate step before generating texture. Person-independent facial landmarks are generated from audio for invariance to different voices, accents, etc. Eye blinks are imposed on facial landmarks and the person-independent landmarks are retargeted to person-specific landmarks to preserve identity related facial structure. Facial texture is then generated from person-specific facial landmarks that helps to preserve identity-related texture.
1. A processor implemented method, comprising:
obtaining, via one or more hardware processors, an audio speech and an identity image of a target individual;
extracting, via the one or more hardware processors, one or more DeepSpeech features of the target individual from the audio speech;
generating, using the extracted one or more DeepSpeech features, via a trained speech-to-landmark generation network executed by the one or more hardware processors, a speech-induced motion on a sparse representation of a neutral mean face pertaining to the at least one user, wherein the sparse representation of the speech-induced motion comprises a plurality of facial landmark points;
generating, via a video dataset, a plurality of eye blink movements from random noise input, wherein the plurality of eye blink movements are generated for each eye based on a sequence of generated displacements of associated facial landmark points of each eye region, and wherein the plurality of eye blink movements comprise a set of eye landmark points with blink motion;
replacing one or more eye landmark points of the plurality of facial landmark points with the set of eye landmark points with blink motion to obtain a set of final landmark points, the set of final landmark points comprises (i) the set of eye landmark points with blink motion and (ii) one or more landmark points containing the speech-induced motion;
generating one or more target-specific landmark points based on (i) the set of final landmark points and (ii) an identity landmark obtained from the identity image; and
generating, an attention-based texture, using an attention map and a color map that are learnt using the one or more target-specific landmark points and the identity image, wherein the attention-based texture serves as an animated talking face of the target individual.
2. The processor implemented method of claim 1 , wherein the identity image comprises an image containing a neutral facial expression of the target individual.
3. The processor implemented method of claim 1 , wherein the step of generating, via a video dataset, a plurality of eye blink movements from random noise input, comprises:
performing a comparison of the sequence of generated displacements of the associated facial landmark points of each eye region with one or more ground truth displacements, wherein one or more higher order moments of distribution of the sequence of generated displacements are compared with the one or more ground truth displacements; and
generating the plurality of eye blink movements based on the comparison.
4. The processor implemented method of claim 3 , wherein the one or more higher order moments of the sequence of generated displacements comprise one or more statistics of the distribution of the sequence of generated displacements.
5. The processor implemented method of claim 4 , wherein the one or more statistics of the distribution of the sequence of generated displacements comprise at least one of a mean, a variance, and a skewness, wherein a Maximum Mean Discrepancy (MMD) loss is computed using the one or more statistics, and wherein the MMD loss is a measure of similarity or a dissimilarity between a distribution of (i) the plurality of eye blink movements and (ii) one or more ground truth eye blink movements.
6. A system, comprising:
a memory storing instructions;
one or more communication interfaces; and
one or more hardware processors coupled to the memory via the one or more communication interfaces, wherein the one or more hardware processors are configured by the instructions to:
obtain an audio speech and an identity image of a target individual;
extract, via the one or more hardware processors, one or more DeepSpeech features of the target individual from the audio speech;
generate, using the extracted one or more DeepSpeech features, via a trained speech-to-landmark generation network executed by the one or more hardware processors, a speech-induced motion on a sparse representation of a neutral mean face pertaining to the at least one user, wherein the sparse representation of the speech-induced motion comprises a plurality of facial landmark points;
generate, via a video dataset, a plurality of eye blink movements from random noise input, wherein the plurality of eye blink movements are generated for each eye based on a sequence of generated displacements of associated facial landmark points of each eye region, and wherein the plurality of eye blink movements comprise a set of eye landmark points with blink motion;
replace one or more eye landmark points of the plurality of facial landmark points with the set of eye landmark points with blink motion to obtain a set of final landmark points, the set of final landmark points comprises (i) the set of eye landmark points with blink motion and (ii) one or more landmark points containing the speech-induced motion;
generate one or more target-specific landmark points based on (i) the set of final landmark points and (ii) an identity landmark obtained from the identity image; and
generate, an attention-based texture, using an attention map and a color map that are learnt using the one or more target-specific landmark points and the identity image, wherein the attention-based texture serves as an animated talking face of the target individual.
7. The system as claimed in claim 6 , wherein the identity image comprises an image containing a neutral facial expression of the target individual.
8. The system as claimed in claim 6 , wherein the plurality of eye blink movements are generated from random noise input by:
performing a comparison of the sequence of generated displacements of the associated facial landmark points of each eye region with one or more ground truth displacements, wherein one or more higher order moments of distribution of the sequence of generated displacements are compared with the one or more ground truth displacements; and
generating the plurality of eye blink movements based on the comparison.
9. The system as claimed in claim 8 , wherein the one or more higher order moments of the sequence of generated displacements comprise one or more statistics of the distribution of the sequence of generated displacements.
10. The system as claimed in claim 9 , wherein the one or more statistics of the distribution of the sequence of generated displacements comprise at least one of a mean, a variance, and a skewness, wherein a Maximum Mean Discrepancy (MMD) loss is computed using the one or more statistics, and wherein the MMD loss is a measure of a similarity or a dissimilarity between a distribution of (i) the plurality of eye blink movements and (ii) one or more ground truth eye blink movements.
11. A computer program product comprising a non-transitory computer readable medium having a computer readable program embodied therein, wherein the computer readable program, when executed on a computing device, causes the computing device to generate identity preserving realistic talking face using audio speech of a user by:
obtaining an audio speech and an identity image of a target individual;
extracting, via the one or more hardware processors, one or more DeepSpeech features of the target individual from the audio speech;
generating, using the extracted one or more DeepSpeech features, via a trained speech-to-landmark generation network, a speech-induced motion on a sparse representation of a neutral mean face pertaining to the at least one user, wherein the sparse representation of the speech-induced motion comprises a plurality of facial landmark points;
generating, via a video dataset, a plurality of eye blink movements from random noise input, wherein the plurality of eye blink movements are generated for each eye based on a sequence of generated displacements of associated facial landmark points of each eye region, and wherein the plurality of eye blink movements comprise a set of eye landmark points with blink motion;
replacing one or more eye landmark points of the plurality of facial landmark points with the set of eye landmark points with blink motion to obtain a set of final landmark points, the set of final landmark points comprises (i) the set of eye landmark points with blink motion and (ii) one or more landmark points containing the speech-induced motion;
generating one or more target-specific landmark points based on (i) the set of final landmark points and (ii) an identity landmark obtained from the identity image; and
generating, an attention-based texture, using an attention map and a color map that are learnt using the one or more target-specific landmark points and the identity image, wherein the attention-based texture serves as an animated talking face of the target individual.
12. The computer program product as claimed in claim 11 , wherein the identity image comprises an image containing a neutral facial expression of the target individual.
13. The computer program product as claimed in claim 11 , wherein the step of generating, via a video dataset, a plurality of eye blink movements from random noise input, comprises: performing a comparison of the sequence of generated displacements of the associated facial landmark points of each eye region with one or more ground truth displacements, wherein one or more higher order moments of distribution of the sequence of generated displacements of the associated facial landmark points of each eye region are compared with the one or more ground truth displacements; and generating the plurality of eye blink movements based on the comparison.
14. The computer program product as claimed in claim 13 , wherein the one or more higher order moments of the sequence of generated displacements of the associated facial landmark points of each eye region comprise one or more statistics of the distribution of the sequence of generated displacements.
15. The computer program product as claimed in claim 14 , wherein the one or more statistics of the distribution of the sequence of generated displacements comprise at least one of a mean, a variance, and a skewness, wherein a Maximum Mean Discrepancy (MMD) loss is computed using the one or more statistics, and wherein the MMD loss is a measure of a similarity or a dissimilarity between a distribution of (i) the plurality of eye blink movements and (ii) one or more ground truth eye blink movements.