Granted Patent B2

US 12,731,319 · App. 18/348,856 · Granted Sep 8, 2026

Inventors: Gaurav Bharaj (Santa Monica, CA); Qingju Liu (London, GB)

Assignee: Flawless Holdings Limited

G06T13/40G10L15/063

Abstract

Claims (50)

1 . A computer-implemented method of generating a facial representation, the method comprising:

receiving, at a machine learned model, an input text segment, an input audio signal corresponding to the input text segment, and target style data, wherein the target style data represents a target audio style;

generating, by a text encoder of the machine learned model, a representation of the input text segment;

determining, by an aligner of the machine learned model, a time alignment between the input text segment and the input audio signal; and

generating, by a decoder of the machine learned model, the facial representation based at least in part on the representation of the input text segment, the time alignment, and the target style data, wherein the facial representation comprises a sequence of facial expressions corresponding to the input text segment;

wherein the machine learned model is trained by:

performing a first training operation comprising training the model based at least in part on:

generating, by a first configuration of the decoder, an output audio representation based on a first training text segment and a corresponding first training audio signal; and

updating the model so as to reduce a deviation between the output audio representation and an audio representation of the first training audio signal; and

performing a second training operation comprising training the model based at least in part on:

generating, by a second configuration of the decoder, an output facial representation based at least in part on a second training text segment and a corresponding second training audio signal; and

updating the model so as to reduce a deviation between the output facial representation and a training facial representation corresponding to the second training text segment and second training audio signal.

2 . The method of claim 1 , comprising:

receiving, at the machine learned model, a reference audio signal exhibiting the target audio style; and

generating, by a style encoder of the machine learned model, the target style data based at least in part on the reference audio signal.

3 . The method of claim 1 , comprising:

receiving input video data comprising target footage of a target human face; and

generating output video data based at least in part on the input video data and on the facial representation, wherein in the output video data the target human face exhibits the sequence of facial expressions corresponding to the input text segment.

4 . The method of claim 1 , wherein a weight associated with the text encoder, energy, or pitch is held fixed during the second training operation.

5 . The method of claim 1 , wherein during the first training operation the model is trained with a first duration of training audio signals, and during the second training operation the model is trained with a second duration of training audio signals, wherein the first duration is at least five times the second duration, or at least ten times the second duration.

6 . The method of claim 1 , wherein generating the time alignment comprises determining, from the input audio signal, a ground truth duration of phonemes represented in the input text segment, and wherein the facial representation is generated based at least in part on the ground truth duration of the phonemes.

7 . The method of claim 1 , wherein the target style data is an embedded representation of the target audio style, and wherein the method further comprises predicting, from the target style data, an energy parameter, a pitch parameter, and a residual parameter.

8 . The method of claim 1 , wherein the target style data specifies an energy parameter, a pitch parameter, and a residual parameter.

9 . The method of claim 7 , wherein the method further comprises generating, by a variance adaptor of the machine learned model, a combined signal representing a combination of the representation of the input text segment, the energy parameter, the pitch parameter, and the residual parameter; and

wherein the facial representation is generated by the decoder based at least in part on the combined signal.

10 . The method of claim 1 , wherein the facial representation comprises a blendshape.

11 . A computer-implemented method of training a machine learning model for generating a facial representation, the method comprising:

initializing the machine learning model, wherein the machine learning model comprises:

a text encoder configured to generate, from a received text segment, a representation of the received text segment;

an aligner configured to determine a time alignment between the received text segment and a received audio signal corresponding to the received text segment; and

a decoder configured to generate an output based at least in part on the representation of the received text segment, the time alignment, and received target style data representing an audio style;

performing a first training operation comprising training the machine learning model based at least in part on:

generating, by a first configuration of the decoder, an output audio representation based on a first training text segment and a corresponding first training audio signal; and

updating the machine learning model so as to reduce a deviation between the output audio representation and an audio representation of the first training audio signal; and

performing a second training operation comprising training the model based at least in part on:

generating, by a second configuration of the decoder, an output facial representation based at least in part on a second training text segment and a corresponding second training audio signal; and

updating the machine learning model so as to reduce a deviation between the output facial representation and a training facial representation corresponding to the second training text segment and second training audio signal; and

outputting, based at least in part on the first training operation and the second training operation, a machine learned model, wherein the machine learned model comprises the decoder in the second configuration.

12 . A system comprising one or more processors and one or more non-transient storage media storing machine readable instructions which, when executed by the one or more processors, cause the one or more processors to carry out a method comprising:

receiving, at a machine learned model, an input text segment, an input audio signal corresponding to the input text segment, and target style data, wherein the target style data represents a target audio style;

generating, by a text encoder of the machine learned model, a representation of the input text segment;

determining, by an aligner of the machine learned model, a time alignment between the input text segment and the input audio signal; and

generating, by a decoder of the machine learned model, the facial representation based at least in part on the representation of the input text segment, the time alignment, and the target style data, wherein the facial representation comprises a sequence of facial expressions corresponding to the input text segment;

wherein the machine learned model is trained by:

performing a first training operation comprising training the model based at least in part on:

generating, by a first configuration of the decoder, an output audio representation based on a first training text segment and a corresponding first training audio signal; and

updating the model so as to reduce a deviation between the output audio representation and an audio representation of the first training audio signal; and

performing a second training operation comprising training the model based at least in part on:

generating, by a second configuration of the decoder, an output facial representation based at least in part on a second training text segment and a corresponding second training audio signal; and

updating the model so as to reduce a deviation between the output facial representation and a training facial representation corresponding to the second training text segment and second training audio signal.

Assignments (1)

Continuity (1)

References Cited (43)