Granted Patent B2

US 12,524,965 · App. 18/473,768 · Granted Jan 13, 2026

Inventor: Kohei Miyamoto (San Jose, CA)

Assignees: SONY GROUP CORPORATION; SONY CORPORATION OF AMERICA

G06T17/205G06T5/70G06T7/11G06T2207/10028G06T2207/20084

Abstract

Claims (69)

1 . An electronic device, comprising:

circuitry configured to:

acquire a set of three-dimensional (3D) scans including a body portion of an object;

determine a set of segments of the body portion from each 3D scan of the acquired set of 3D scans;

apply, based on the determined set of segments, a neural network model on the acquired set of 3D scans, wherein the neural network model includes an encoder model and a decoder model;

determine, by the encoder model, based on the acquired set of 3D scans, a set of weights associated with the determined set of segments;

determine, by the decoder model, a set of vertex difference vectors associated with the determined set of segments, wherein

each vector of the determined set of vertex difference vectors corresponds to a 3D blend-shape associated with the determined set of segments, and

each segment of the determined set of segments is configured to move independently in the 3D blend-shape;

determine, based on the determined set of weights and the determined set of vertex difference vectors, a regularization function associated with the determined set of segments;

reconstruct a 3D mesh sequence based on the determined set of vertex difference vectors; and

re-train the neural network model based on the acquired set of 3D scans, the determined regularization function, and the reconstructed 3D mesh sequence, wherein

the re-trained neural network model is configured to determine a set of 3D blend-shapes based on a set of input 3D scans, and

the set of 3D blend-shapes includes the 3D blend-shape.

2 . The electronic device according to claim 1 , wherein the body portion of the object corresponds to a face of a person.

3 . The electronic device according to claim 1 , wherein the determination of the set of segments of the body portion is based on at least one of a clustering technique or a specific input.

4 . The electronic device according to claim 1 , wherein the determined regularization function is configured to reduce a number of the set of 3D blend-shapes.

5 . The electronic device according to claim 1 , wherein the determined regularization function corresponds to a Lasso (L1) regression function.

6 . The electronic device according to claim 1 , wherein the circuitry is further configured to:

determine a smoothening function associated with the determined set of segments, wherein

the determined smoothening function is configured to smoothen boundaries of the determined set of segments, and

the re-trained neural network model is further based on the determined smoothening function.

7 . The electronic device according to claim 6 , wherein the determined smoothening function corresponds to a Laplacian boundary-smoothening function.

8 . The electronic device according to claim 1 , wherein the each vector of the determined set of vertex difference vectors corresponds to at least one of:

a region-based blend-shape of a segment of the determined set of segments, or

mask information associated with the segment.

9 . The electronic device according to claim 1 , wherein

the determined set of vertex difference vectors includes a first vector and a second vector,

the first vector has a first valid area,

the second vector has a second valid area,

the first valid area overlaps with the second valid area, and

the overlap between the first valid area and the second valid area hide boundaries between segments associated with the first vector and the second vector.

10 . The electronic device according to claim 9 , wherein

the overlap between the first valid area and the second valid area is smoothened with a smoothening function, and

the re-trained neural network model is based on the smoothening function.

11 . A method, comprising:

in an electronic device:

acquiring a set of three-dimensional (3D) scans including a body portion of an object;

determining a set of segments of the body portion from each 3D scan of the acquired set of 3D scans;

applying, based on the determined set of segments, a neural network model on the acquired set of 3D scans, wherein the neural network model includes an encoder model and a decoder model;

determining, by the encoder model, based on the acquired set of 3D scans, a set of weights associated with the determined set of segments;

determining, by the decoder model, a set of vertex difference vectors associated with the determined set of segments, wherein

each vector of the determined set of vertex difference vectors corresponds to a 3D blend-shape associated with the determined set of segments, and

each segment of the determined set of segments is configured to move independently in the 3D blend-shape;

determining, based on the determined set of weights and the determined set of vertex difference vectors, a regularization function associated with the determined set of segments;

reconstructing a 3D mesh sequence based on the determined set of vertex difference vectors; and

re-training the neural network model based on the acquired set of 3D scans, the determined regularization function, and the reconstructed 3D mesh sequence, wherein

the re-trained neural network model is configured to determine a set of 3D blend-shapes based on a set of input 3D scans, and

the set of 3D blend-shapes includes the 3D blend-shape.

12 . The method according to claim 11 , wherein the determined regularization function is configured to reduce a number of the set of 3D blend-shapes.

13 . The method according to claim 11 , wherein the determined regularization function corresponds to a Lasso (L1) regression function.

14 . The method according to claim 11 , further comprising:

determining a smoothening function associated with the determined set of segments, wherein

the determined smoothening function is configured to smoothen boundaries of the determined set of segments, and

the re-training of the neural network model is further based on the determined smoothening function.

15 . The method according to claim 14 , wherein the determined smoothening function corresponds to a Laplacian boundary-smoothening function.

16 . A non-transitory computer-readable medium having stored thereon, computer-executable instructions that when executed by an electronic device, cause the electronic device to execute operations, the operations comprising:

acquiring a set of three-dimensional (3D) scans including a body portion of an object;

determining a set of segments of the body portion from each 3D scan of the acquired set of 3D scans;

applying, based on the determined set of segments, a neural network model on the acquired set of 3D scans, wherein the neural network model includes an encoder model and a decoder model;

determining, by the encoder model, based on the acquired set of 3D scans, a set of weights associated with the determined set of segments;

determining, by the decoder model, a set of vertex difference vectors associated with the determined set of segments, wherein

each vector of the determined set of vertex difference vectors corresponds to a 3D blend-shape associated with the determined set of segments, and

each segment of the determined set of segments is configured to move independently in the 3D blend-shape;

determining, based on the determined set of weights and the determined set of vertex difference vectors, a regularization function associated with the determined set of segments;

reconstructing a 3D mesh sequence based on the determined set of vertex difference vectors; and

re-training the neural network model based on the acquired set of 3D scans, the determined regularization function, and the reconstructed 3D mesh sequence, wherein

the re-trained neural network model is configured to determine a set of 3D blend-shapes based on a set of input 3D scans, and

the set of 3D blend-shapes includes the 3D blend-shape.

Assignments (1)

Continuity (1)

References Cited (34)