Granted Patent B2

US 12,236,943 · App. 17/764,651 · Granted Feb 25, 2025

Inventors: Guem Buel Hwang (Seoul, KR); Gyeong Su Chae (Seoul, KR)

Assignee: DEEPBRAIN AI INC.

G10L15/16G10L21/10G10L15/25G10L2021/105

Abstract

Claims (40)

1. An apparatus for generating a lip sync image having one or more processors and a memory which stores one or more programs executed by the one or more processors, the apparatus comprising:

a first artificial neural network model configured to generate an utterance match synthesis image by using a person background image and an utterance match audio signal as an input, and generate an utterance mismatch synthesis image by using the person background image and an utterance mismatch audio signal as an input; and

a second artificial neural network model configured to output classification values for an input pair in which an image and a voice match and an input pair in which an image and a voice do not match by using the input pairs as an input,

wherein the utterance match audio signal is a voice signal which matches a figure in which the corresponding person utters in the person background image,

wherein the utterance mismatch audio signal is a voice signal which does not match the figure in which the corresponding person utters in the person background image,

wherein the second artificial neural network model is trained to classify the input pair in which an image and a voice match as True, and to classify the input pair in which an image and a voice do not match as False,

wherein the first artificial neural network model is configured to receive the utterance mismatch synthesis image generated by the first artificial neural network model and the utterance mismatch audio signal used as the input when generating the utterance mismatch synthesis image and classify the utterance mismatch synthesis image and the utterance mismatch audio signal as True, and propagate a generative adversarial error to the first artificial neural network model through an adversarial learning method.

2. The apparatus of claim 1 , wherein the person background image is an image in which a portion associated with an utterance of a person is masked.

3. The apparatus of claim 1 , wherein the first artificial neural network model comprises:

a first encoder configured to use the person background image as an input, and extract an image feature vector from the input person background image;

a second encoder configured to use the utterance match audio signal corresponding to the person background image as an input, and extract a voice feature vector from the input utterance match audio signal;

a combiner configured to generate a combined vector by combining the image feature vector and the voice feature vector; and

a decoder configured to use the combined vector as an input, and generate the utterance match synthesis image based on the combined vector.

4. The apparatus of claim 3 , wherein an objective function L reconstruction for the generation of the utterance match synthesis image of the first artificial neural network model is represented by the following equation:

L reconstruction =∥I i −Î ii ∥

where I i is Original utterance image;

Î ii is Utterance match synthesis image; and

∥A−B∥ is Function for obtaining difference between A and B.

5. The apparatus of claim 4 , wherein an objective function L discriminator of the second artificial neural network model is represented by the following equation:

L discriminator =log(1− D ( I i ,A i ))+log( D ( I i ,A i ))

where D is Neural network of the second artificial neural network model;

(I i , A i ) is Input pair in which an image and a voice match (i-th image and i-th voice); and

(I i , A j ) is Input pair in which an image and a voice do not match (i-th image and j-th voice).

6. The apparatus of claim 5 , wherein an adversarial objective function L adversarial for the generation of the utterance mismatch synthesis image of the first artificial neural network model is represented by the following equation:

L adversarial =−log( D ( G ( M i * I i ,A i ), A j ))

where G is Neural network constituting the first artificial neural network model;

M i *I i is Person background image in which portion associated with utterance is masked (M i : mask);

G (M i *I i , A j ) is Utterance mismatch synthesis image generated by the first artificial neural network model; and

A j is Utterance mismatch audio signal not corresponding to person background image.

7. The apparatus of claim 6 , wherein a final objective function L T for the generation of the utterance match synthesis image and the utterance mismatch synthesis image of first artificial neural network model is represented by the following equation:

L T =L reconstruction +λL adversarial

where λ is Weight.

8. A method for generating a lip sync image performed by a computing device having one or more processors and a memory which stores one or more programs executed by the one or more processors, the method comprising:

generating, in a first artificial neural network model, an utterance match synthesis image by using a person background image and an utterance match audio signal as an input;

generating, in a first artificial neural network model, an utterance mismatch synthesis image by using the person background image and an utterance mismatch audio signal as an input; and

outputting, in a second artificial neural network model, classification values for an input pair in which an image and a voice match and an input pair in which an image and a voice do not match by using the input pairs as an input,

wherein the utterance match audio signal is a voice signal which matches a figure in which the corresponding person utters in the person background image,

wherein the utterance mismatch audio signal is a voice signal which does not match the figure in which the corresponding person utters in the person background image,

wherein the second artificial neural network model is trained to classify the input pair in which an image and a voice match as True, and to classify the input pair in which an image and a voice do not match as False,

wherein the first artificial neural network model is configured to receive the utterance mismatch synthesis image generated by the first artificial neural network model and the utterance mismatch audio signal used as the input when generating the utterance mismatch synthesis image and classify the utterance mismatch synthesis image and the utterance mismatch audio signal as True, and propagate a generative adversarial error to the first artificial neural network model through an adversarial learning method.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (13)