Granted Patent B2

US 12,657,824 · App. 18/472,826 · Granted Jun 16, 2026

Inventors: Shichen Liu (Los Angeles, CA); Yunxuan Cai (Los Angeles, CA); Yajie Zhao (Los Angeles, CA); Bipin Kishore (Los Angeles, CA)

Assignee: UNIVERSITY OF SOUTHERN CALIFORNIA

G06T17/20G06T7/40G06T7/60G06T7/70G06T2207/10024G06T2207/20084

Abstract

Claims (97)

1 . A method for constructing a 3D face model, the method comprising:

providing a reference as a template, the reference containing a 3D mesh that represents an item and a UV texture mapping that maps each point on the 3D mesh to a 2D UV space, the item comprising a median face, vertex positions or surface points;

receiving one or more input images in an image space of a face;

extracting geometry feature and texture feature in the image space;

extracting features in a UV space;

iteratively computing or producing a feature map via visual semantic correlation between the UV space and the image space and regress geometry updates, including projecting or providing the extracted texture feature to the UV space to predict texture maps and comparing said features to the reference, and inputting the computed or produced feature map to an RNN-based neural optimizer composed of Gated Recurrent Units (GRU) to determine a hidden state;

outputting a pose in a canonical space of the 3D mesh and a UV-space position map for the one or more input images, wherein each pixel in the UV-space position map stores a coordinate of a corresponding point in the canonical space of the 3D mesh, wherein the geometry feature is utilized to fit a 6-DoF rigid transformation that represents the output pose in the canonical space of the 3D mesh and the UV-space position map, and

optimizing a head pose and/or facial geometry into the 6-DoF rigid transformation and the UV-space position map, said optimizing comprising:

fixing the position map to be a mean face geometry of the 3D mesh of the reference and updating the 6-DoF rigid transformation using a first number iterations;

updating both the 6-DoF rigid transformation and the UV-space position map for a second number of iterations; and

fixing the 6-DoF rigid transformation and updating the UV-space position map using a third number of iterations.

2 . The method according to claim 1 , wherein the reference is a reference face model or a position map.

3 . The method according to claim 1 , wherein the received one or more input images comprises multi-view input images.

4 . The method according to claim 1 , wherein the outputted pose is a camera pose or a head pose.

5 . The method according to claim 1 , wherein the feature map is a correlation feature map.

6 . The method according to claim 1 , wherein the method further comprises utilizing the texture feature for producing an albedo map, a specular map, and a displacement map.

7 . The method according to claim 6 , wherein the projecting the extracted texture feature comprises warping and said feature warping is used to capture textures and high frequency details of the face of the input image.

8 . The method according to claim 1 , wherein the iteratively computing or producing the feature map comprises using a geometry decoding network to process a hidden vector and compute a geometry update tuple comprising a quaternion for correcting the output pose, and an offset for correcting the UV-space map.

9 . The method according to claim 8 , wherein the quaternion is a predicted quaternion.

10 . The method according to claim 8 , wherein the computing of the geometry update tuple comprises, for each pixel in the UV space, enumerating a multi-scale grid around the 3D position of each pixel, projecting grid points to the image space using pose or camera parameters, and comparing corresponding points between the geometry feature and the UV feature to calculate visual semantic similarities, wherein the camera parameters comprise camera rotation and a predicted vector for correcting the camera translation.

11 . The method according to claim 1 , wherein the reference comprises a dataset acquired by a Light Stage scan system that employs photometric stereo in combination with polarization promotion to facilitate pore level accuracy in both geometry reconstruction and reflectance maps.

12 . The method according to claim 11 , wherein a camera setup of the light stage scan system comprises 25 machine vision cameras, including 17 monochrome cameras to achieve more efficient capture and higher-resolution results, allowing for sub-millimeter geometry details, albedo, and specular reflectance reconstruction, and eight color cameras, which aid in stereo base mesh reconstruction.

13 . The method according to claim 11 , wherein the dataset includes around 30,000 4K×4K multiview images and processed geometry in sub-millimeter accuracy and consistency, and skin reflectance (albedo and specularity) corresponding to each image.

14 . A non-transitory machine-readable medium storing instructions which, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:

providing a reference as a template, the reference containing a 3D mesh that represents an item and a UV texture mapping that maps each point on the 3D mesh to a 2D UV space, the item comprising a median face, vertex positions or surface points;

receiving one or more input images in an image space of a face;

extracting geometry feature and texture feature in the image space;

extracting features in a UV space;

iteratively computing or producing a feature map via visual semantic correlation between the UV space and the image space and regress geometry updates, including projecting or providing the extracted texture feature to the UV space to predict texture maps and comparing said features to the reference, and inputting the computed or produced feature map to an RNN-based neural optimizer composed of Gated Recurrent Units (GRU) to determine a hidden state;

outputting a pose in a canonical space of the 3D mesh and a UV-space position map for the one or more input images, wherein each pixel in the UV-space position map stores a coordinate of a corresponding point in the canonical space of the 3D mesh, wherein the geometry feature is utilized to fit a 6-DoF rigid transformation that represents the output pose in the canonical space of the 3D mesh and the UV-space position map, and

optimizing a head pose and/or facial geometry into the 6-DoF rigid transformation and the UV-space position map, said optimizing comprising:

fixing the position map to be a mean face geometry of the 3D mesh of the reference and updating the 6-DoF rigid transformation using a first number iterations;

updating both the 6-DoF rigid transformation and the UV-space position map for a second number of iterations; and

fixing the 6-DoF rigid transformation and updating the UV-space position map using a third number of iterations.

15 . A system comprising:

at least one programmable processor;

a non-transitory machine-readable medium storing instructions which, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising:

providing a reference as a template, the reference containing a 3D mesh that represents an item and a UV texture mapping that maps each point on the 3D mesh to a 2D UV space, the item comprising a median face, vertex positions or surface points;

receiving one or more input images in an image space of a face;

extracting geometry feature and texture feature in the image space;

extracting features in a UV space;

iteratively computing or producing a feature map via visual semantic correlation between the UV space and the image space and regress geometry updates, including projecting or providing the extracted texture feature to the UV space to predict texture maps and comparing said features to the reference, and inputting the computed or produced feature map to an RNN-based neural optimizer composed of Gated Recurrent Units (GRU) to determine a hidden state;

outputting a pose in a canonical space of the 3D mesh and a UV-space position map for the one or more input images, wherein each pixel in the UV-space position map stores a coordinate of a corresponding point in the canonical space of the 3D mesh, wherein the geometry feature is utilized to fit a 6-DoF rigid transformation that represents the output pose in the canonical space of the 3D mesh and the UV-space position map, and

optimizing a head pose and/or facial geometry into the 6-DoF rigid transformation and the UV-space position map, said optimizing comprising:

fixing the position map to be a mean face geometry of the 3D mesh of the reference and updating the 6-DoF rigid transformation using a first number iterations;

updating both the 6-DoF rigid transformation and the UV-space position map for a second number of iterations; and

fixing the 6-DoF rigid transformation and updating the UV-space position map using a third number of iterations.

16 . A system for obtaining images of a subject for constructing a 3D face model, the system comprising:

a modular frame assembly for enclosing a 180 degree area, the modular frame assembly comprising at least two vertical support members and at least two horizontal support members, a first horizonal support member being provided at a first horizontal position and at least a second horizontal support member being provided at a second horizontal position;

a plurality of cameras provided on the modular frame assembly, said plurality of cameras being configured to receive power from a power source;

at least one interval timer connected to said plurality of cameras;

a lighting system comprising a plurality of light boxes, each light box spaced relative to the modular frame assembly about the 180 degree area for lighting the subject from multiple directions, and

at least one programmable processor connected to said plurality of cameras,

wherein, upon activation of said at least one interval timer, said plurality of cameras are directed to obtain one or more images of the subject and communicate with the at least one programmable processor, and

wherein said at least one programmable processor is configured to:

provide a reference as a template, the reference containing a 3D mesh that represents an item and a UV texture mapping that maps each point on the 3D mesh to a 2D UV space, the item comprising a median face, vertex positions or surface points;

receive one or more input images in an image space of a face;

extract geometry feature and texture feature in the image space;

extract features in a UV space;

iteratively compute or produce a feature map via visual semantic correlation between the UV space and the image space and regress geometry updates, including projecting or providing the extracted texture feature to the UV space to predict texture maps and comparing said features to the reference, and inputting the computed or produced feature map to an RNN-based neural optimizer composed of Gated Recurrent Units (GRU) to determine a hidden state;

output a pose in a canonical space of the 3D mesh and a UV-space position map for the one or more input images, wherein each pixel in the UV-space position map stores a coordinate of a corresponding point in the canonical space of the 3D mesh, wherein the geometry feature is utilized to fit a 6-DoF rigid transformation that represents the output pose in the canonical space of the 3D mesh and the UV-space position map, and

optimizing a head pose and/or facial geometry into the 6-DoF rigid transformation and the UV-space position map, said optimizing comprising:

fixing the position map to be a mean face geometry of the 3D mesh of the reference and updating the 6-DoF rigid transformation using a first number iterations;

updating both the 6-DoF rigid transformation and the UV-space position map for a second number of iterations; and

fixing the 6-DoF rigid transformation and updating the UV-space position map using a third number of iterations.

17 . The system according to claim 16 , wherein two or more cameras are provided on the first horizonal support member and a single camera is provided on the at least a second horizontal support member.

18 . The system according to claim 16 , wherein the modular frame assembly comprises three frame structures hingedly connected together for adjustment around the 180 degree area.

19 . The system according to claim 16 , further comprising a blackout curtain surrounding the modular frame assembly for limiting and/or preventing interference from environmental and/or external light sources.

20 . A method for constructing a 3D face model, the method comprising:

providing a reference as a template, the reference containing a 3D mesh that represents an item and a UV texture mapping that maps each point on the 3D mesh to a 2D UV space, the item comprising a median face, vertex positions or surface points;

receiving one or more input images in an image space of a face;

extracting geometry feature and texture feature in the image space;

extracting features in a UV space;

iteratively computing or producing a feature map via visual semantic correlation between the UV space and the image space and regress geometry updates, including projecting or providing the extracted texture feature to the UV space to predict texture maps and comparing said features to the reference, and inputting the computed or produced feature map to an RNN-based neural optimizer composed of Gated Recurrent Units (GRU) to determine a hidden state; and

outputting a pose in a canonical space of the 3D mesh and a UV-space map for the one or more input images, wherein each pixel in the UV-space map stores a coordinate of a corresponding point in the canonical space of the 3D mesh,

wherein the iteratively computing or producing the feature map comprises using a geometry decoding network to process a hidden vector and compute a geometry update tuple comprising a quaternion for correcting the output pose, and an offset for correcting the UV-space map,

wherein the computing of the geometry update tuple comprises, for each pixel in the UV space, enumerating a multi-scale grid around the 3D position of each pixel, projecting grid points to the image space using pose or camera parameters, and comparing corresponding points between the geometry feature and the UV feature to calculate visual semantic similarities, wherein the camera parameters comprise camera rotation and a predicted vector for correcting the camera translation.

21 . A non-transitory machine-readable medium storing instructions which, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:

providing a reference as a template, the reference containing a 3D mesh that represents an item and a UV texture mapping that maps each point on the 3D mesh to a 2D UV space, the item comprising a median face, vertex positions or surface points;

receiving one or more input images in an image space of a face;

extracting geometry feature and texture feature in the image space;

extracting features in a UV space;

iteratively computing or producing a feature map via visual semantic correlation between the UV space and the image space and regress geometry updates, including projecting or providing the extracted texture feature to the UV space to predict texture maps and comparing said features to the reference, and inputting the computed or produced feature map to an RNN-based neural optimizer composed of Gated Recurrent Units (GRU) to determine a hidden state; and

outputting a pose in a canonical space of the 3D mesh and a UV-space map for the one or more input images, wherein each pixel in the UV-space map stores a coordinate of a corresponding point in the canonical space of the 3D mesh,

wherein the iteratively computing or producing the feature map comprises using a geometry decoding network to process a hidden vector and compute a geometry update tuple comprising a quaternion for correcting the output pose, and an offset for correcting the UV-space map,

wherein the computing of the geometry update tuple comprises, for each pixel in the UV space, enumerating a multi-scale grid around the 3D position of each pixel, projecting grid points to the image space using pose or camera parameters, and comparing corresponding points between the geometry feature and the UV feature to calculate visual semantic similarities, wherein the camera parameters comprise camera rotation and a predicted vector for correcting the camera translation.

22 . A system comprising:

at least one programmable processor;

a non-transitory machine-readable medium storing instructions which, when executed by the at least one programmable processor, cause the at least one programmable processor to perform operations comprising:

providing a reference as a template, the reference containing a 3D mesh that represents an item and a UV texture mapping that maps each point on the 3D mesh to a 2D UV space, the item comprising a median face, vertex positions or surface points;

receiving one or more input images in an image space of a face;

extracting geometry feature and texture feature in the image space;

extracting features in a UV space;

iteratively computing or producing a feature map via visual semantic correlation between the UV space and the image space and regress geometry updates, including projecting or providing the extracted texture feature to the UV space to predict texture maps and comparing said features to the reference, and inputting the computed or produced feature map to an RNN-based neural optimizer composed of Gated Recurrent Units (GRU) to determine a hidden state; and

outputting a pose in a canonical space of the 3D mesh and a UV-space map for the one or more input images, wherein each pixel in the UV-space map stores a coordinate of a corresponding point in the canonical space of the 3D mesh,

wherein the iteratively computing or producing the feature map comprises using a geometry decoding network to process a hidden vector and compute a geometry update tuple comprising a quaternion for correcting the output pose, and an offset for correcting the UV-space map,

wherein the computing of the geometry update tuple comprises, for each pixel in the UV space, enumerating a multi-scale grid around the 3D position of each pixel, projecting grid points to the image space using pose or camera parameters, and comparing corresponding points between the geometry feature and the UV feature to calculate visual semantic similarities, wherein the camera parameters comprise camera rotation and a predicted vector for correcting the camera translation.

Assignments (3)

Continuity (3)

References Cited (91)