Granted Patent B2

US 12,536,739 · App. 18/450,428 · Granted Jan 27, 2026

Inventors: Zhuoxuan Li (Hong Kong, HK); Jingyi Xu (Hong Kong, HK); Yi Li (Hong Kong, HK); Hon Wah Wong (Hong Kong, HK); Yanchen Wang (Hong Kong, HK); Man Choi Chan (Hong Kong, HK)

Assignee: Hong Kong Applied Science and Technology Research Institute Company Limited

G06T17/00G06F3/04855G06T5/50G06T13/40G06V40/171G10L15/02G06T2207/10004G06T2207/20221G06T2207/30201

Abstract

Claims (53)

1 . An apparatus for virtual avatar generation, comprising:

a face recorder configured to receive and store a first facial image and a second facial image;

a face encoder configured to:

identify faces of the first facial image and the second facial image;

extract a plurality of first facial features from the first facial image and then encode them into M first facial feature vectors; and

extract a plurality of second facial features from the second facial image and then encode them into M second facial feature vectors;

a face fusion engine configured to synthesize a facial fused feature vector from the first facial feature vectors in combination with the second facial feature vectors, thereby generating a fused image from the facial fused feature vector;

a voice recorder configured to receive and store a first audio clip and a second audio clip;

a voice encoder configured to:

identify voices of the first audio clip and the second audio clip;

extract a plurality of first voice features from the first audio clip and then encode them into N first voice feature vectors; and

extract a plurality of second voice features from the second audio clip and then encode them into N second voice feature vectors;

a voice fusion engine configured to synthesize a voice fused feature vector from the first voice feature vectors in combination with the second voice feature vectors, thereby generating a fused voice from the voice fused feature vector, wherein the facial fused feature vector is generated from the first facial feature vectors and the second facial feature vectors according to a first weight distribution, and the voice fused feature vector is generated from the first voice feature vectors and the second voice feature vectors according to a second weight distribution;

wherein the facial fused feature vector is generated from the first facial feature vectors and the second facial feature vectors according to a first weight distribution, and the voice fused feature vector is generated from the first voice feature vectors and the second voice feature vectors according to a second weight distribution;

a database storing a plurality of face image references and audio clip references; and

an automatic weight calculator configured to calculate the first weight distribution and the second weight distribution by matching the fused image with the fused voice based on pairs of the face image references and audio clip references, wherein the automatic weight calculator is further configured to:

determine a pair of intermediate weights for a set of the first facial feature vectors, the second facial feature vectors, the first voice feature vectors, the second voice feature vectors by using a pair of the face image references and the audio clip references;

calculate a loss function using the set of the first facial feature vectors, the second facial feature vectors, the first voice feature vectors, the second voice feature vectors, the intermediate weights, and the pair of the face image references and the audio clip references;

calculate the loss function for all sets of the first facial feature vectors, the second facial feature vectors, the first voice feature vectors, the second voice feature vectors, the face image references, and the audio clip references, aiming to identify the minimum loss function among them; and

output determined weights, which result in the minimum loss function, for determining the first weight distribution and the second weight distribution to the face fusion engine and the voice fusion engine.

2 . The apparatus of claim 1 , further comprising a user interface that has a virtual dashboard with:

M controllable bars representing facial weights to be assigned to the first facial feature vectors and the second facial feature vectors, wherein the first weight distribution is produced by a series of the facial weights; and

N controllable bars representing voice weights to be assigned to the first voice feature vectors and the second voice feature vectors, wherein the second weight distribution is produced by a series of the voice weights.

3 . The apparatus of claim 2 , wherein the M first facial feature vectors are paired one-to-one with the M second facial feature vectors, and the N first voice feature vectors are paired one-to-one with the N second voice feature vectors.

4 . The apparatus of claim 3 , wherein a first vector of the first facial feature vectors is paired with a first vector of the second facial feature vectors by a first facial weight set, wherein a second vector of the first facial feature vectors is paired with a second vector of the second facial feature vectors by a second facial weight set permitted to be different than the first facial weight set.

5 . The apparatus of claim 1 , further comprising a user interface that has an electronic display configured to dynamically show the fused image and play the fused voice.

6 . The apparatus of claim 1 , further comprising an avatar generator configured to generate a virtual avatar incorporating the fused image and the fused voice.

7 . The apparatus of claim 6 , further comprising a cache configured to store the fused image and the fused voice to be accessed by the avatar generator.

8 . A method for virtual avatar generation, comprising:

receiving and storing a first facial image and a second facial image by a face recorder;

identifying faces of the first facial image and the second facial image by a face encoder;

extracting a plurality of first facial features from the first facial image and then encode them into M first facial feature vectors by the face encoder;

extracting a plurality of second facial features from the second facial image and then encode them into M second facial feature vectors by the face encoder;

synthesizing a facial fused feature vector from the first facial feature vectors in combination with the second facial feature vectors by a face fusion engine, thereby generating a fused image from the facial fused feature vector;

receiving and storing a first audio clip and a second audio clip by a voice recorder;

identifying voices of the first audio clip and the second audio clip by a voice encoder;

extracting a plurality of first voice features from the first audio clip and then encode them into N first voice feature vectors by the voice encoder;

extracting a plurality of second voice features from the second audio clip and then encode them into N second voice feature vectors by the voice encoder;

synthesizing a voice fused feature vector from the first voice feature vectors in combination with the second voice feature vectors by a voice fusion engine, thereby generating a fused voice from the voice fused feature vector, wherein the facial fused feature vector is generated from the first facial feature vectors and the second facial feature vectors according to a first weight distribution, and the voice fused feature vector is generated from the first voice feature vectors and the second voice feature vectors according to a second weight distribution;

storing a plurality of face image references and audio clip references in a database;

calculating, by an automatic weight calculator, the first weight distribution and the second weight distribution by matching the fused image with the fused voice based on pairs of the face image references and audio clip references;

determining, by the automatic weight calculator, a pair of intermediate weights for a set of the first facial feature vectors, the second facial feature vectors, the first voice feature vectors, the second voice feature vectors by using a pair of the face image references and the audio clip references;

calculating, by the automatic weight calculator, a loss function value using the set of the first facial feature vectors, the second facial feature vectors, the first voice feature vectors, the second voice feature vectors, the intermediate weights, and the pair of the face image references and the audio clip references;

calculating, by the automatic weight calculator, the loss function for all sets of the first facial feature vectors, the second facial feature vectors, the first voice feature vectors, the second voice feature vectors, the face image references, and the audio clip references, aiming to identify the minimum loss function among them; and

outputting, by the automatic weight calculator, determined weights, which result in the minimum loss function, for determining the first weight distribution and the second weight distribution to the face fusion engine and the voice fusion engine.

9 . The method of claim 8 , further comprising:

showing M controllable bars, by a virtual dashboard, representing facial weights to be assigned to the first facial feature vectors and the second facial feature vectors, wherein the first weight distribution is produced by a series of the facial weights; and

showing N controllable bars, by the virtual dashboard, representing voice weights to be assigned to the first voice feature vectors and the second voice feature vectors, wherein the second weight distribution is produced by a series of the voice weights.

10 . The method of claim 9 , wherein the M first facial feature vectors are paired one-to-one with the M second facial feature vectors, and the N first voice feature vectors are paired one-to-one with the N second voice feature vectors.

11 . The method of claim 10 , wherein a first vector of the first facial feature vectors is paired with a first vector of the second facial feature vectors by a first facial weight set, wherein a second vector of the first facial feature vectors is paired with a second vector of the second facial feature vectors by a second facial weight set permitted to be different than the first facial weight set.

12 . The method of claim 8 , further comprising dynamically showing the fused image and play the fused voice by an electronic display.

13 . The method of claim 8 , further comprising generating a virtual avatar incorporating the fused image and the fused voice by an avatar generator.

14 . The method of claim 13 , further comprising storing the fused image and the fused voice in a cache for accessing by the avatar generator.

Assignments (1)

Continuity (1)

References Cited (37)