Granted Patent B2

US 12,749,337 · App. 18/260,126 · Granted Sep 29, 2026

Inventors: Wenyu Chen (Singapore, CN); Gengdai Liu (Guangzhou, CN)

Assignee: BIGO TECHNOLOGY PTE. LTD.

G06V40/172G06T7/248G06T7/60G06T17/20G06V10/7715G06V20/41G06V20/46G06V40/165G06V40/171G06V40/174G06T2207/10016G06T2207/30201

Abstract

Claims (105)

1 . A face tracking method, applicable to an optimization thread, the method comprising:

determining a current facial identity vector used by a tracking thread as an initial facial identity vector upon invoking the optimization thread;

acquiring a first keyframe data set, wherein the first keyframe data set is a data set updated by the tracking thread upon performing face tracking, and the first keyframe data set comprises face tracking data;

acquiring optimized face tracking data by optimizing the face tracking data in the first keyframe data set based on the initial facial identity vector;

acquiring an optimized facial identity vector by performing iterative optimization on the initial facial identity vector based on the optimized face tracking data;

upon each iteration, determining, based on the optimized facial identity vector and the initial facial identity vector, whether an iteration stop condition is satisfied;

in response to the iteration stop condition being satisfied, making the tracking thread determine, in response to receiving the optimized facial identity vector, a received facial identity vector as the current facial identity vector by sending the optimized facial identity vector to the tracking thread;

in response to the iteration stop condition being not satisfied, making the tracking thread update, in response to a second keyframe set of a second keyframe data set being a non-empty set, the second keyframe data set to the first keyframe data set upon receiving a clear instruction by sending the clear instruction for clearing video frame in the second keyframe data set to the tracking thread; and

determining the optimized facial identity vector as the initial facial identity vector, and returning to the process of acquiring the first keyframe data set.

2 . The method according to claim 1 , wherein the face tracking data comprises a face keypoint, posture data, and expression data; and acquiring the optimized face tracking data by optimizing the face tracking data in the first keyframe data set based on the initial facial identity vector comprises:

constructing a three-dimensional face model based on the initial facial identity vector and the expression data;

acquiring a face keypoint of the three-dimensional face model; and

acquiring optimized posture data and optimized expression data as the optimized face tracking data by solving optimal posture data and optimal expression data based on the face keypoint of the three-dimensional face model and the face keypoint in the face tracking data.

3 . The method according to claim 2 , wherein acquiring the optimized posture data and the optimized expression data as the optimized face tracking data by solving the optimal posture data and the optimal expression data based on the face keypoint of the three-dimensional face model and the face keypoint in the face tracking data comprises:

solving the optimized face tracking data according to the following formula:

(

P

i

k

,

δ

i

k

)

=

arg

⁢

min

⁡

(

∑

j



∏

P

i

(

C

0

k

-

1

+

C

exp

k

-

1

δ

i

)

j

-

Q

i

j



+

γ

⁢



P

i

,

δ

i



)

wherein k represents a k th iteration;

C

0

k

-

1

 represents a neutral face used in the k th iteration; C_exp{circumflex over ( )}(k−1) represents an expression shape fusion deformer used in the k th iteration; ΠP i (⋅) represents j face keypoints acquired by projecting the three-dimensional face model (C 0 +C exp δ i ) j ; Q i represents a face keypoint of the face tracking data in the first keyframe data set; γ represents a parameter; δ i represents the expression data; P i represents the posture data; and i represents an i th keyframe.

4 . The method according to claim 1 , wherein the face tracking data comprises a face keypoint, posture data, and expression data; and acquiring the optimized facial identity vector by performing the iterative optimization on the initial facial identity vector based on the optimized face tracking data comprises:

calculating a face size of a tracked face based on the face keypoint;

calculating an expression weight of each keyframe based on the expression data of each keyframe; and

acquiring the optimized facial identity vector by performing iterative solving based on the face tracking data, the face size, the expression weight of each keyframe, the current facial identity vector, and the initial facial identity vector.

5 . The method according to claim 4 , wherein calculating the expression weight of each keyframe based on the expression data of each keyframe comprises:

determining minimum expression data from the expression data of all keyframes; and

calculating the expression weight of the keyframe based on a predetermined constant term, the minimum expression data, and the expression data of the keyframe, wherein the expression weight of the keyframe is negatively related to the expression data of the keyframe.

6 . The method according to claim 4 , wherein acquiring the optimized facial identity vector by performing the iterative solving based on the face tracking data, the face size, the expression weight of each keyframe, the current facial identity vector, and the initial facial identity vector comprises:

constructing a three-dimensional face model based on the current facial identity vector and the expression data of each keyframe;

acquiring a plurality of projected face keypoints by projecting the three-dimensional face model to a two-dimensional plane;

calculating a distance sum between the plurality of projected face keypoints and the face keypoints; and

acquiring the optimized facial identity vector by performing the iterative solving based on the expression data, the distance sum, the expression weight of each keyframe, the face size, the current facial identity vector, and the initial facial identity vector.

7 . The method according to claim 1 , wherein determining, based on the optimized facial identity vector and the initial facial identity vector, whether the iteration stop condition is satisfied comprises:

calculating a face change rate based on the optimized facial identity vector and the initial facial identity vector;

determining whether the face change rate is less than a predetermined change rate threshold;

determining that the iteration stop condition is satisfied in response to the face change rate being less than the predetermined change rate threshold; and

determining that the iteration stop condition is not satisfied in response to the face change rate being not less than the predetermined change rate threshold.

8 . The method according to claim 7 , wherein calculating the face change rate based on the optimized facial identity vector and the initial facial identity vector comprises:

acquiring a face size of an average face;

calculating a distance between a face mesh corresponding to the optimized facial identity vector and a face mesh corresponding to the initial facial identity vector; and

calculating a ratio of the distance to the face size of the average face as the face change rate.

9 . The method according to claim 1 , wherein upon sending the optimized facial identity vector to the tracking thread, the method further comprises:

updating a frame vector of each keyframe based on expression data and posture data of optimized keyframes in the first keyframe data set; and

updating a first PCA subspace based on the frame vector of each keyframe.

10 . The method according to claim 1 , wherein prior to determining the current facial identity vector used by the tracking thread as the initial facial identity vector, the method further comprises:

assigning a first PCA subspace to a second PCA subspace.

11 . An electronic device for tracking a face, comprising:

at least one processor; and

a storage apparatus configured to store at least one program;

wherein the at least one program, when run by the at least one processor, causes the at least one processor to perform the face tracking method as defined in claim 1 .

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (12)