Reinforced differentiable attribute for 3D face reconstruction
Techniques performed by a data processing system for reconstructing a three-dimensional (3D) model of the face of a human subject herein include obtaining source data comprising a two-dimensional (2D) image, three-dimensional (3D) image, or depth information representing a face of a human subject. Reconstructing the 3D model of the face also includes generating a 3D model of the face of the human subject based on the source data by analyzing the source data to produce a coarse 3D model of the face of the human subject, and refining the coarse 3D model through free form deformation to produce a fitted 3D model. The coarse 3D model may be a 3D Morphable Model (3DMM), and the coarse 3D model may be refined through free-form deformation in which the deformation of the mesh is limited by applying an as-rigid-as-possible (ARAP) deformation constraint.
1. A data processing system comprising:
a processor; and
a computer-readable medium storing executable instructions for causing the processor to perform operations of:
obtaining source data comprising a two-dimensional (2D) image, three-dimensional (3D) image, or depth information representing a face of a human subject;
analyzing the source data to produce a coarse 3D model of the face of the human subject;
providing the source data to a neural network trained to analyze the source data and output a corrective shape residual, the corrective shape residual modeling a deformation of a mesh of the coarse 3D model for generating a fitted 3D model;
obtaining the corrective shape residual from the neural network; and
applying free-form deformation to the mesh of the coarse 3D model to refine a shape of the mesh according to the corrective shape residual.
2. The data processing system of claim 1 , wherein to analyze the source data to produce the coarse 3D model the computer-readable medium includes instructions configured to cause the processor to perform the operation of:
producing the coarse 3D model of the face using a 3D Morphable Model (3DMM).
3. The data processing system of claim 1 , wherein to deform the mesh according to the corrective shape residual the computer-readable medium includes instructions configured to cause the processor to perform the operation of:
limiting the deformation of the mesh by applying an as-rigid-as-possible (ARAP) deformation constraint.
4. The data processing system of claim 1 , wherein the computer-readable medium includes executable instructions for causing the processor to perform operations of:
rendering the 2D image from the coarse 3D model using a rendering pipeline that utilizes one or more differentiable attributes that can be used to further refine the coarse 3D model; and
comparing the 2D image to a reference ground-truth image to determine a photometric loss function for further refining the coarse 3D model.
5. The data processing system of claim 4 , wherein the one or more differentiable attributes include depth, color, and mask attributes.
6. The data processing system of claim 4 , wherein the computer-readable medium includes executable instructions for causing the processor to perform operations of:
rendering the 2D image using a soft rasterization process that applies a convolutional kernel to blur the rendered 2D image to propagate attributes across vertices of the mesh.
7. A method performed by a data processing system for generating a model, the method comprising:
obtaining source data comprising a two-dimensional (2D) image, three-dimensional (3D) image, or depth information representing a face of a human subject;
analyzing the source data to produce a coarse 3D model of the face of the human subject;
providing the source data to a neural network trained to analyze the source data and output a corrective shape residual, the corrective shape residual modeling a deformation of a mesh of the coarse 3D model for generating a fitted 3D model;
obtaining the corrective shape residual from the neural network; and
applying free-form deformation to the mesh of the coarse 3D model to refine a shape of the mesh according to the corrective shape residual.
8. The method of claim 7 , wherein analyzing the 2D image of the face to produce the coarse 3D model of the face of the human subject includes producing the coarse 3D model of the face using a 3D Morphable Model (3DMM).
9. The method of claim 7 , wherein deforming the mesh according to the corrective shape residual includes limiting the deformation of the mesh by applying an as-rigid-as-possible (ARAP) deformation constraint.
10. The method of claim 7 , further comprising:
rendering the 2D image from the coarse 3D model using a rendering pipeline that utilizes one or more differentiable attributes that can be used to further refine the coarse 3D model; and
comparing the 2D image to a reference ground-truth image to determine a photometric loss function for further refining the coarse 3D model.
11. The method of claim 10 , wherein the one or more differentiable attributes include depth, color, and mask attributes.
12. The method of claim 10 , further comprising:
rendering the 2D image using a soft rasterization process that applies a convolutional kernel to blur the rendered 2D image to propagate attributes across vertices of the mesh.
13. A machine-readable medium storing instructions that, when executed on a processor of a data processing system, cause the data processing system to generate a model, by:
obtaining source data comprising a two-dimensional (2D) image, three-dimensional (3D) image, or depth information representing a face of a human subject;
analyzing the source data of the face to produce a coarse 3D model of the face of the human subject;
providing the source data to a neural network trained to analyze the source data and output a corrective shape residual, the corrective shape residual modeling a deformation of a mesh of the coarse 3D model for generating a fitted 3D model;
obtaining the corrective shape residual from the neural network; and
applying free-form deformation to the mesh of the coarse 3D model to refine a shape of the mesh according to the corrective shape residual.
14. The machine-readable medium of claim 13 , wherein to analyze the 2D image of the face to produce the coarse 3D model, the machine-readable medium includes instructions configured to cause the processor to perform an operation of producing the coarse 3D model of the face using a 3D Morphable Model (3DMM).
15. The machine-readable medium of claim 13 , wherein to deform the mesh according to the corrective shape residual the machine-readable medium includes instructions configured to cause the processor to perform an operation of limiting the deformation of the mesh by applying an as-rigid-as-possible (ARAP) deformation constraint.
16. The machine-readable medium of claim 13 , wherein the machine-readable medium includes executable instructions for causing the processor to perform operations of:
rendering the 2D image from the coarse 3D model using a rendering pipeline that utilizes one or more differentiable attributes that can be used to further refine the coarse 3D model; and
comparing the 2D image to a reference ground-truth image to determine a photometric loss function for further refining the coarse 3D model.
17. The machine-readable medium of claim 16 , wherein the machine-readable medium includes executable instructions for causing the processor to perform an operation of rendering the 2D image using a soft rasterization process that applies a convolutional kernel to blur the rendered 2D image to propagate attributes across vertices of the mesh.