Granted Patent B2

US 12,412,339 · App. 18/258,493 · Granted Sep 9, 2025

Inventors: Xiaowei Zhang (Guangzhou, CN); Zhongyuan Hu (Guangzhou, CN); Gengdai Liu (Guangzhou, CN)

Assignee: BIGO TECHNOLOGY PTE. LTD.

G06T17/20

Abstract

Claims (64)

1. A method for reconstructing three-dimensional faces, comprising:

estimating a dynamic reconstruction parameter of a current video frame for three-dimensional face reconstruction by inputting, in response to a steady-state reconstruction parameter of the current video frame for the three-dimensional face reconstruction having been estimated by a pre-constructed teacher network model, the current video frame into a student network model distilled from the teacher network model; and

reconstructing a three-dimensional face corresponding to the current video frame by inputting the steady-state reconstruction parameter and the dynamic reconstruction parameter into a pre-constructed three-dimensional deformation model.

2. The method according to claim 1 , wherein prior to estimating the dynamic reconstruction parameter of the current video frame for the three-dimensional face reconstruction by inputting the current video frame into the student network model distilled from the teacher network model, the method further comprises:

estimating a comprehensive reconstruction parameter of an associated video frame with a same face as the current video frame for the three-dimensional face reconstruction by inputting the associated video frame into the teacher network model, and determining a steady-state reconstruction parameter in the comprehensive reconstruction parameter as the steady-state reconstruction parameter of the current video frame for the three-dimensional face reconstruction.

3. The method according to claim 2 , wherein the associated video frame is a first video frame with a face within the current video frame in a video to which the current video frame belongs, or the associated video frames are first m video frames with the face within the current video frame in the video to which the current video frame belongs, wherein m represents a target number.

4. The method according to claim 2 , wherein

the associated video frames are first m video frames with a face within the current video frame in a video to which the current video frame belongs, wherein m represents a target number; and

determining the steady-state reconstruction parameter in the comprehensive reconstruction parameter as the steady-state reconstruction parameter of the current video frame for the three-dimensional face reconstruction comprises:

acquiring the steady-state reconstruction parameter of the current video frame by fusing the steady-state reconstruction parameters of a same type from the comprehensive reconstruction parameters of the associated video frames.

5. The method according to claim 1 , wherein the student network model is acquired by distilling the teacher network model by following processes:

estimating a first comprehensive reconstruction parameter of a first video frame for the three-dimensional face reconstruction and a second comprehensive reconstruction parameter of a second video frame for the three-dimensional face reconstruction by inputting the first video frame and the second video frame into the teacher network model, the first video frame and the second video frame being from a video frame training pair with a same face in a video training set;

estimating a target dynamic reconstruction parameter of the second video frame for the three-dimensional face reconstruction by inputting the second video frame from the video frame training pair into the student network model, and reconstructing a three-dimensional face corresponding to the second video frame by inputting a target steady-state reconstruction parameter in the first comprehensive reconstruction parameter and the target dynamic reconstruction parameter into the three-dimensional deformation model;

calculating a first loss function between the second comprehensive reconstruction parameter and the target dynamic reconstruction parameter and a second loss function between the three-dimensional face corresponding to the second video frame and the second video frame under a predetermined two-dimensional supervision term, and adjusting a weight corresponding to each of loss functions in the first loss function and the second loss function; and

generating a corresponding fitting loss function based on each of the loss functions and the corresponding weight, and training the student network model by performing inverse correction on the student network model using the fitting loss function.

6. The method according to claim 5 , wherein the second loss function under the predetermined two-dimensional supervision term comprises: an image pixel loss function, a keypoint loss function, an identity feature loss function, and an albedo penalty function.

7. The method according to claim 6 , wherein

in a case that the image pixel loss function between the three-dimensional face corresponding to the second video frame and the second video frame is to be calculated, the method further comprises:

segmenting a corresponding skin mask from the second video frame; and

calculating the image pixel loss function between the three-dimensional face corresponding to the second video frame and the second video frame comprises:

acquiring the corresponding image pixel loss function by calculating, based on the skin mask, a pixel error between a pixel point in the three-dimensional face corresponding to the second video frame and the same pixel point in a face skin region in the second video frame.

8. The method according to claim 6 , wherein

in a case that the keypoint loss function between the three-dimensional face corresponding to the second video frame and the second video frame is to be calculated, the method further comprises:

extracting key feature points at predetermined positions from the second video frame, and determining visibility of each of the key feature points; and

calculating the keypoint loss function between the three-dimensional face corresponding to the second video frame and the second video frame comprises:

acquiring the corresponding keypoint loss function by calculating a position reconstruction error, between the three-dimensional face corresponding to the second video frame and the second video frame, of each of visible key feature points.

9. The method according to claim 6 , wherein

in a case that the identity feature loss function between the three-dimensional face corresponding to the second video frame and the second video frame is to be calculated, the method further comprises:

acquiring a first identity feature of the second video frame and a second identity feature of the three-dimensional face corresponding to the second video frame by inputting the three-dimensional face corresponding to the second video frame and the second video frame into a pre-constructed face recognition model; and

calculating the identity feature loss function between the three-dimensional face corresponding to the second video frame and the second video frame comprises:

calculating the corresponding identity feature loss function based on a similarity between the first identity feature and the second identity feature.

10. The method according to claim 6 , wherein

in a case that the albedo penalty function of the three-dimensional face corresponding to the second video frame is to be calculated, the method further comprises:

calculating an albedo of each of mesh vertices in the three-dimensional face corresponding to the second video frame; and

calculating the albedo penalty function of the three-dimensional face corresponding to the second video frame comprises:

calculating the corresponding albedo penalty function based on the albedo of each of the mesh vertices in the three-dimensional face and a predetermined albedo range.

11. The method according to claim 1 , wherein

the three-dimensional deformation model is composed of a bilinear face model and a principal component analysis (PCA) model, wherein a core tensor of the three-dimensional face is defined in the bilinear face model, the core tensor being composed of three dimensions of a mesh vertex, a face shape, and a face expression of the three-dimensional face; and the PCA model defines an average face albedo and a principal component base representing a change of a face albedo; and

reconstructing the three-dimensional face corresponding to the current video frame by inputting the steady-state reconstruction parameter and the dynamic reconstruction parameter into the pre-constructed three-dimensional deformation model comprises:

acquiring a corresponding three-dimensional deformation face by adjusting the core tensor by inputting a reconstruction parameter representing a change of the face shape in the steady-state reconstruction parameters and a reconstruction parameter representing a change of the face expression in the dynamic reconstruction parameters into the bilinear face model; and

acquiring a reconstructed three-dimensional face by correcting an albedo of the three-dimensional deformation face based on the average face albedo by inputting the three-dimensional deformation face and a reconstruction parameter matched with the principal component base under the PCA model in the steady-state reconstruction parameters and the dynamic reconstruction parameters into the PCA model.

12. The method according to claim 11 , wherein prior to reconstructing the three-dimensional face corresponding to the current video frame by inputting the steady-state reconstruction parameter and the dynamic reconstruction parameter into the pre-constructed three-dimensional deformation model, the method further comprises:

collecting three-dimensional face scanning data from a multi-dimensional data source, and acquiring a corresponding core tensor by performing high-order singular value decomposition, under a face shape dimension and a face expression dimension, on a three-dimensional face reconstructed based on the three-dimensional face scanning data.

13. The method according to claim 11 , wherein the three-dimensional deformation model further defines an illumination parameter representing a change of face illumination, a position parameter representing a face translation, and a rotation parameter representing a head posture.

14. The method according to claim 1 , wherein upon reconstructing the three-dimensional face corresponding to the current video frame by inputting the steady-state reconstruction parameter and the dynamic reconstruction parameter into the pre-constructed three-dimensional deformation model, the method further comprises:

rendering the three-dimensional face corresponding to the current video frame using a differentiable renderer.

15. The method according to claim 1 , further comprising:

estimating a comprehensive reconstruction parameter of the current video frame for the three-dimensional face reconstruction by inputting, in response to the steady-state reconstruction parameter of the current video frame for the three-dimensional face reconstruction not being estimated by the pre-constructed teacher network model, the current video frame into the teacher network model, and reconstructing the three-dimensional face corresponding to the current video frame by inputting the comprehensive reconstruction parameter of the current video frame for the three-dimensional face reconstruction into the three-dimensional deformation model.

16. A computer device for reconstructing three-dimensional faces, comprising:

one or more processors; and

a storage apparatus, configured to store one or more programs; wherein

the one or more processors, when loading and running the one or more programs, are caused to perform:

estimating a dynamic reconstruction parameter of a current video frame for three-dimensional face reconstruction by inputting, in response to a steady-state reconstruction parameter of the current video frame for the three-dimensional face reconstruction having been estimated by a pre-constructed teacher network model, the current video frame into a student network model distilled from the teacher network model; and

reconstructing a three-dimensional face corresponding to the current video frame by inputting the steady-state reconstruction parameter and the dynamic reconstruction parameter into a pre-constructed three-dimensional deformation model.

17. The computer device according to claim 16 , wherein the one or more processors, when loading and running the one or more programs, are caused to perform:

prior to estimating the dynamic reconstruction parameter of the current video frame for the three-dimensional face reconstruction by inputting the current video frame into the student network model distilled from the teacher network model, estimating a comprehensive reconstruction parameter of an associated video frame with a same face as the current video frame for the three-dimensional face reconstruction by inputting the associated video frame into the teacher network model, and determining a steady-state reconstruction parameter in the comprehensive reconstruction parameter as the steady-state reconstruction parameter of the current video frame for the three-dimensional face reconstruction.

18. The computer device according to claim 17 , wherein the associated video frame is a first video frame with a face within the current video frame in a video to which the current video frame belongs, or the associated video frames are first m video frames with the face within the current video frame in the video to which the current video frame belongs, wherein m represents a target number.

19. The computer device according to claim 17 , wherein

the associated video frames are first m video frames with a face within the current video frame in a video to which the current video frame belongs, wherein m represents a target number; and

the one or more processors, when loading and running the one or more programs, are caused to perform:

acquiring the steady-state reconstruction parameter of the current video frame by fusing the steady-state reconstruction parameters of a same type from the comprehensive reconstruction parameters of the associated video frames.

20. A non-transitory computer-readable storage medium, storing a computer program, wherein the computer program, when loaded and run by a processor, causes the processor to perform:

estimating a dynamic reconstruction parameter of a current video frame for three-dimensional face reconstruction by inputting, in response to a steady-state reconstruction parameter of the current video frame for the three-dimensional face reconstruction having been estimated by a pre-constructed teacher network model, the current video frame into a student network model distilled from the teacher network model; and

reconstructing a three-dimensional face corresponding to the current video frame by inputting the steady-state reconstruction parameter and the dynamic reconstruction parameter into a pre-constructed three-dimensional deformation model.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (40)