IP Library › Granted Patent US 12,307,567
Granted Patent B2
US 12,307,567 · App. 18/163,704 · Granted May 20, 2025

Methods and systems for emotion-controllable generalized talking face generation

Inventors: Sanjana Sinha (Kolkata, IN); Sandika Biswas (Kolkata, IN); Brojeshwar Bhowmick (Kolkata, IN)
Assignee: Tata Consultancy Services Limited
G06T13/40G06N3/0455G06T13/205G06V10/806G06V40/171G10L25/63
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,307,567
App. No.
18/163,704
Granted
May 20, 2025
Kind
B2
Abstract

This disclosure relates generally to methods and systems for emotion-controllable generalized talking face generation of an arbitrary face image. Most of the conventional techniques for the realistic talking face generation may not be efficient to control the emotion over the face and have limited scope of generalization to an arbitrary unknown target face. The present disclosure proposes a graph convolutional network that uses speech content feature along with an independent emotion input to generate emotion and speech-induced motion on facial geometry-aware landmark representation. The facial geometry-aware landmark representation is further used in by an optical flow-guided texture generation network for producing the texture. A two-branch optical flow-guided texture generation network with motion and texture branches is designed to consider the motion and texture content independently. The optical flow-guided texture generation network then renders emotional talking face animation from a single image of any arbitrary target face.

Claims (92)

1. A processor-implemented method comprising the steps of:

receiving, via one or more hardware processors, a plurality of training samples, wherein each training sample of the plurality of training samples comprises a speech audio input data, an emotion input data comprising an emotion type and an emotion intensity, an input image of a target subject in a neutral emotion, and a ground-truth image corresponding to the emotion input data;

training, via the one or more hardware processors, a geometry-aware landmark generation network, with each training sample at a time, until the plurality of training samples is completed, to obtain a trained speech and emotion driven geometry-aware landmark generation model, wherein the geometry-aware landmark generation network comprises an audio encoder network, a first emotion encoder network, a graph encoder network, and a graph decoder network, and wherein training the geometry-aware landmark generation network with each training sample comprises:

obtaining a set of emotion-invariant speech embedding features, from the speech audio input data present in the training sample, using the audio encoder network;

obtaining a set of first emotion embedding features, from the emotion input data present in the training sample, using the first emotion encoder network;

obtaining a set of graph embedding features, from the input image of the target subject in the neutral emotion present in the training sample, using the graph encoder network;

concatenating (i) the set of emotion-invariant speech embedding features, (ii) the set of first emotion embedding features, and (iii) the set of graph embedding features, to obtain concatenated embedding features of the training sample;

decoding the concatenated embedding features of the training sample, to predict a landmark graph of the training sample, using the graph decoder network, wherein the predicted landmark graph comprises an ordered graph representation of predicted speech and emotion driven geometry-aware facial landmarks of the training sample;

minimizing a loss function of the geometry-aware landmark generation network, wherein the loss function computes a difference between the predicted landmark graph of the training sample, and a ground-truth landmark graph obtained from the ground-truth image corresponding to the training sample; and

updating weights of the geometry-aware landmark generation network, based on the minimization of the loss function of the geometry-aware landmark generation network; and

training, via the one or more hardware processors, a flow-guided texture generation network with each training sample at a time, until the plurality of training samples is completed, to obtain a trained flow-guided texture generation model, using the predicted landmark graph of each training sample, wherein the flow-guided texture generation network comprises an image encoder network, a landmark encoder network, a second emotion encoder network, a feature concatenation encoder-decoder network, and an image decoder network, and wherein training the flow-guided texture generation network with each training sample comprises:

obtaining a set of identity features from the input image of the target subject in the neutral emotion present in the training sample, using the image encoder network;

obtaining a set of differential landmark features, from the predicted landmark graph of the training sample and the neutral landmark graph corresponding to the input image of the target subject in the neutral emotion present in the training sample, using the landmark encoder network;

obtaining a set of second emotion embedding features, from the emotion input data present in the training sample, using the second emotion encoder network;

combining (i) the set of identity features, (ii) the set of differential landmark features, (iii) the set of second emotion embedding features, to obtain a dense flow map and an occlusion map, for the training sample, using the feature concatenation encoder-decoder network;

passing the dense flow map and the occlusion map for the training sample, to the image decoder network, to predict an emotional talking face image for the target subject present in the training sample, wherein the predicted emotional talking face image comprises the speech audio input data and the emotion input data corresponding to the training sample;

minimizing a loss function of the flow-guided texture generation network, wherein the loss function of the flow-guided texture generation network computes the difference between the predicted emotional talking face image of the training sample, and the ground-truth image corresponding to the training sample; and

updating weights of the flow-guided texture generation network, based on the minimization of the loss function of the flow-guided texture generation network.

2. The method of claim 1 , wherein obtaining the set of emotion-invariant speech embedding features, from the speech audio input data present in the training sample, using the audio encoder network, comprises:

extracting a set of DeepSpeech features, from the speech audio input data present in the training sample, using a pre-trained DeepSpeech network; and

encoding the set of DeepSpeech features, using the audio encoder network, to obtain the set of emotion-invariant speech embedding features for the training sample.

3. The method of claim 1 , wherein obtaining the set of first emotion embedding features, from the emotion input data present in the training sample, using the first emotion encoder network, comprises:

obtaining a vector representation of the emotion input data, based on the emotion type and the emotion intensity present in the training sample, using a one-hot encoding technique; and

encoding the vector representation of the emotion input data, using the first emotion encoder network, to obtain the set of first emotion embedding features.

4. The method of claim 1 , wherein obtaining the set of graph embedding features, from the input image of the target subject in the neutral emotion present in the training sample, using the graph encoder network, comprises:

generating a neutral landmark graph, for the input image of the target subject in the neutral emotion present in the training sample, using a facial landmark extraction technique followed by a Delaunay triangulation; and

encoding the neutral landmark graph, using the graph encoder network, to obtain the set of graph embedding features for the training sample, wherein the graph encoder network employs a graph convolution technique.

5. The method of claim 1 , wherein obtaining the set of second emotion embedding features, from the emotion input data present in the training sample, using the second emotion encoder network, comprises:

obtaining a vector representation of the emotion input data, based on the emotion type and the emotion intensity present in the training sample, using a one-hot encoding technique; and

encoding the vector representation of the emotion input data, using the second emotion encoder network, to obtain the set of second emotion embedding features.

6. The method of claim 1 , further comprising:

receiving, via the one or more hardware processors, a speech audio data, an emotion data comprising the emotion type and the emotion intensity, and an image of an arbitrary target subject in a neutral emotion;

predicting, via the one or more hardware processors, the landmark graph of the arbitrary target subject, from the speech audio data, the emotion data, and the image of the arbitrary target subject in the neutral emotion, using the trained speech and emotion driven geometry-aware landmark generation model; and

generating, via the one or more hardware processors, an emotion-controllable generalized talking face image corresponding to the image of the arbitrary target subject in the neutral emotion, based on the predicted landmark graph of the arbitrary target subject in the neutral emotion and the emotion data, using a fine-tuned flow-guided texture generation model, wherein the fine-tuned flow-guided texture generation model is obtained by fine-tuning the trained flow-guided texture generation model with the image of the arbitrary target subject in neutral emotion, using one-shot learning.

7. A system comprising:

a memory storing instructions;

one or more input/output (I/O) interfaces; and

one or more hardware processors coupled to the memory via the one or more I/O interfaces, wherein the one or more hardware processors are configured by the instructions to:

receive a plurality of training samples, wherein each training sample of the plurality of training samples comprises a speech audio input data, an emotion input data comprising an emotion type and an emotion intensity, an input image of a target subject in a neutral emotion, and a ground-truth image corresponding to the emotion input data;

train a geometry-aware landmark generation network, with each training sample at a time, until the plurality of training samples is completed, to obtain a trained speech and emotion driven geometry-aware landmark generation model, wherein the geometry-aware landmark generation network comprises an audio encoder network, a first emotion encoder network, a graph encoder network, and a graph decoder network, and wherein training the geometry-aware landmark generation network with each training sample comprises:

obtaining a set of emotion-invariant speech embedding features, from the speech audio input data present in the training sample, using the audio encoder network;

obtaining a set of first emotion embedding features, from the emotion input data present in the training sample, using the first emotion encoder network;

obtaining a set of graph embedding features, from the input image of the target subject in the neutral emotion present in the training sample, using the graph encoder network;

concatenating (i) the set of emotion-invariant speech embedding features, (ii) the set of first emotion embedding features, and (iii) the set of graph embedding features, to obtain concatenated embedding features of the training sample;

decoding the concatenated embedding features of the training sample, to predict a landmark graph of the training sample, using the graph decoder network, wherein the predicted landmark graph comprises an ordered graph representation of predicted speech and emotion driven geometry-aware facial landmarks of the training sample;

minimizing a loss function of the geometry-aware landmark generation network, wherein the loss function computes a difference between the predicted landmark graph of the training sample, and a ground-truth landmark graph obtained from the ground-truth image corresponding to the training sample; and

updating weights of the geometry-aware landmark generation network, based on the minimization of the loss function of the geometry-aware landmark generation network; and

train a flow-guided texture generation network with each training sample at a time, until the plurality of training samples is completed, to obtain a trained flow-guided texture generation model, using the predicted landmark graph of each training sample, wherein the flow-guided texture generation network comprises an image encoder network, a landmark encoder network, a second emotion encoder network, a feature concatenation encoder-decoder network, and an image decoder network, and wherein training the flow-guided texture generation network with each training sample comprises:

obtaining a set of identity features from the input image of the target subject in the neutral emotion present in the training sample, using the image encoder network;

obtaining a set of differential landmark features, from the predicted landmark graph of the training sample and the neutral landmark graph corresponding to the input image of the target subject in the neutral emotion present in the training sample, using the landmark encoder network;

obtaining a set of second emotion embedding features, from the emotion input data present in the training sample, using the second emotion encoder network;

combining (i) the set of identity features, (ii) the set of differential landmark features, (iii) the set of second emotion embedding features, to obtain a dense flow map and an occlusion map, for the training sample, using the feature concatenation encoder-decoder network;

passing the dense flow map and the occlusion map for the training sample, to the image decoder network, to predict an emotional talking face image for the target subject present in the training sample, wherein the predicted emotional talking face image comprises the speech audio input data and the emotion input data corresponding to the training sample;

minimizing a loss function of the flow-guided texture generation network, wherein the loss function of the flow-guided texture generation network computes the difference between the predicted emotional talking face image of the training sample, and the ground-truth image corresponding to the training sample; and

updating weights of the flow-guided texture generation network, based on the minimization of the loss function of the flow-guided texture generation network.

8. The system of claim 7 , wherein the one or more hardware processors are configured to obtain the set of emotion-invariant speech embedding features, from the speech audio input data present in the training sample, using the audio encoder network, by:

extracting a set of DeepSpeech features, from the speech audio input data present in the training sample, using a pre-trained DeepSpeech network; and

encoding the set of DeepSpeech features, using the audio encoder network, to obtain the set of emotion-invariant speech embedding features for the training sample.

9. The system of claim 7 , wherein the one or more hardware processors are configured to obtain the set of first emotion embedding features, from the emotion input data present in the training sample, using the first emotion encoder network, by:

obtaining a vector representation of the emotion input data, based on the emotion type and the emotion intensity present in the training sample, using a one-hot encoding technique; and

encoding the vector representation of the emotion input data, using the first emotion encoder network, to obtain the set of first emotion embedding features.

10. The system of claim 7 , wherein the one or more hardware processors are configured to obtain the set of graph embedding features, from the input image of the target subject in the neutral emotion present in the training sample, using the graph encoder network, by:

generating a neutral landmark graph, for the input image of the target subject in the neutral emotion present in the training sample, using a facial landmark extraction technique followed by a Delaunay triangulation; and

encoding the neutral landmark graph, using the graph encoder network, to obtain the set of graph embedding features for the training sample, wherein the graph encoder network employs a graph convolution technique.

11. The system of claim 7 , wherein the one or more hardware processors are configured to obtain the set of second emotion embedding features, from the emotion input data present in the training sample, using the second emotion encoder network, by:

obtaining a vector representation of the emotion input data, based on the emotion type and the emotion intensity present in the training sample, using a one-hot encoding technique; and

encoding the vector representation of the emotion input data, using the second emotion encoder network, to obtain the set of second emotion embedding features.

12. The system of claim 7 , wherein the one or more hardware processors are further configured to:

receive a speech audio data, an emotion data comprising the emotion type and the emotion intensity, and an image of an arbitrary target subject in a neutral emotion;

predict the landmark graph of the arbitrary target subject, from the speech audio data, the emotion data, and the image of the arbitrary target subject in the neutral emotion, using the trained speech and emotion driven geometry-aware landmark generation model; and

generate an emotion-controllable generalized talking face image corresponding to the image of the arbitrary target subject in the neutral emotion, based on the predicted landmark graph of the arbitrary target subject in the neutral emotion and the emotion data, using a fine-tuned flow-guided texture generation model, wherein the fine-tuned flow-guided texture generation model is obtained by fine-tuning the trained flow-guided texture generation model with the image of the arbitrary target subject in neutral emotion, using one-shot learning.

13. One or more non-transitory machine-readable information storage mediums comprising one or more instructions which when executed by one or more hardware processors cause:

receiving, a plurality of training samples, wherein each training sample of the plurality of training samples comprises a speech audio input data, an emotion input data further comprising an emotion type and an emotion intensity, an input image of a target subject in a neutral emotion, and a ground-truth image corresponding to the emotion input data;

training, a geometry-aware landmark generation network, with each training sample at a time, until the plurality of training samples is completed, to obtain a trained speech and emotion driven geometry-aware landmark generation model, wherein the geometry-aware landmark generation network comprises an audio encoder network, a first emotion encoder network, a graph encoder network, and a graph decoder network, and wherein training the geometry-aware landmark generation network with each training sample comprises:

obtaining a set of emotion-invariant speech embedding features, from the speech audio input data present in the training sample, using the audio encoder network;

obtaining a set of first emotion embedding features, from the emotion input data present in the training sample, using the first emotion encoder network;

obtaining a set of graph embedding features, from the input image of the target subject in the neutral emotion present in the training sample, using the graph encoder network;

concatenating (i) the set of emotion-invariant speech embedding features, (ii) the set of first emotion embedding features, and (iii) the set of graph embedding features, to obtain concatenated embedding features of the training sample;

decoding the concatenated embedding features of the training sample, to predict a landmark graph of the training sample, using the graph decoder network, wherein the predicted landmark graph comprises an ordered graph representation of predicted speech and emotion driven geometry-aware facial landmarks of the training sample;

minimizing a loss function of the geometry-aware landmark generation network, wherein the loss function computes a difference between the predicted landmark graph of the training sample, and a ground-truth landmark graph obtained from the ground-truth image corresponding to the training sample; and

updating weights of the geometry-aware landmark generation network, based on the minimization of the loss function of the geometry-aware landmark generation network;

training, a flow-guided texture generation network with each training sample at a time, until the plurality of training samples is completed, to obtain a trained flow-guided texture generation model, using the predicted landmark graph of each training sample, wherein the flow-guided texture generation network comprises an image encoder network, a landmark encoder network, a second emotion encoder network, a feature concatenation encoder-decoder network, and an image decoder network, and wherein training the flow-guided texture generation network with each training sample comprises:

obtaining a set of identity features from the input image of the target subject in the neutral emotion present in the training sample, using the image encoder network;

obtaining a set of differential landmark features, from the predicted landmark graph of the training sample and the neutral landmark graph corresponding to the input image of the target subject in the neutral emotion present in the training sample, using the landmark encoder network;

obtaining a set of second emotion embedding features, from the emotion input data present in the training sample, using the second emotion encoder network;

combining (i) the set of identity features, (ii) the set of differential landmark features, (iii) the set of second emotion embedding features, to obtain a dense flow map and an occlusion map, for the training sample, using the feature concatenation encoder-decoder network;

passing the dense flow map and the occlusion map for the training sample, to the image decoder network, to predict an emotional talking face image for the target subject present in the training sample, wherein the predicted emotional talking face image comprises the speech audio input data and the emotion input data corresponding to the training sample;

minimizing a loss function of the flow-guided texture generation network, wherein the loss function of the flow-guided texture generation network computes the difference between the predicted emotional talking face image of the training sample, and the ground-truth image corresponding to the training sample; and

updating weights of the flow-guided texture generation network, based on the minimization of the loss function of the flow-guided texture generation network;

receiving, a speech audio data, an emotion data further comprising the emotion type and the emotion intensity, and an image of an arbitrary target subject in a neutral emotion;

predicting, the landmark graph of the arbitrary target subject, from the speech audio data, the emotion data, and the image of the arbitrary target subject in the neutral emotion, using the trained speech and emotion driven geometry-aware landmark generation model; and

generating, an emotion-controllable generalized talking face image corresponding to the image of the arbitrary target subject in the neutral emotion, based on the predicted landmark graph of the arbitrary target subject in the neutral emotion and the emotion data, using a fine-tuned flow-guided texture generation model, wherein the fine-tuned flow-guided texture generation model is obtained by fine-tuning the trained flow-guided texture generation model with the image of the arbitrary target subject in neutral emotion, using one-shot learning.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 3, 2023
From: SINHA, SANJANA; BISWAS, SANDIKA; BHOWMICK, BROJESHWAR
To: TATA CONSULTANCY SERVICES LIMITED
Reel/Frame 062589/0057 →
Priority Claims (1)
IN 202221025055 · Apr 28, 2022 · national
Continuity (1)
Related Publication 20230351662A1 · Nov 2, 2023
References Cited (6)
US 7340393B2 · Mitsuyoshi · 2008 [cited by applicant]
CN 113299255A · 2021 [cited by applicant]
Eskimez, Sefik Emre et al., “Speech Driven Talking Face Generation from a Single Image and an Emotion Condition”, Transactions on Multimedia, Date: Jul. 2021, vol. 24, pp. 3480-3490, Publisher: IEEE, https://arxiv.org/p… [cited by applicant]
Wang, Suzhen et al., “One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning”, Computer Vision and Pattern Recognition, Date: Dec. 2021, Publisher: Arxiv, https://arxiv.org/pdf/2112.02749… [cited by applicant]
Zhang, Zhimeng et al., “Flow-guided One-shot Talking Face Generation with a High-resolution Audio-visual Dataset”, Conference on Computer Vision and Pattern Recognition (CVPR), Date: 2021, Publisher: IEEE, https://opena… [cited by applicant]
Wang, Kaisiyuan et al., “MEAD: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation”, European Conference on Computer Vision, Date: Nov. 2020, pp. 700-717, Publisher: Springer, https://wywu.github.io… [cited by applicant]