IP Library Granted Patent US 12664745
Granted Patent B2
US 12664745 · App. 18/559,591 · Granted Jun 23, 2026

Information processing apparatus and information processing method

Inventors: Ankit Sharma (Tokyo, JP); Natsuki Fukuda (Tokyo, JP)
Assignee: SONY GROUP CORPORATION
G06T19/20G06T7/246G06T7/73G06T19/006G06T2207/10016G06T2207/30201G06T2207/30204G06T2219/2004
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12664745
App. No.
18/559,591
Granted
Jun 23, 2026
Kind
B2
Abstract

Provided is an information processing apparatus including a display control unit and a fitting unit. The display control unit displays a distribution image indicating distribution of a plurality of marker points extracted from a face model of a performer in a superimposed manner on a face image of the performer. The fitting unit fits the face model to the face image by using position information of a placed marker each time the marker is placed on a face of the performer after the distribution image is superimposed, and adjusts a superimposed position of the distribution image based on a fitting result.

Claims (96)

1 . An information processing apparatus, comprising:

a processor configured to:

extract distribution of a plurality of marker points from a face model of a performer;

superimpose a first distribution image on a face image of the performer, wherein the first distribution image indicates the extracted distribution of the plurality of marker points;

display the superimposed first distribution image;

fit the face model to the face image of the performer, based on first position information of a first marker placed on a face of the performer, wherein

the face model is fitted to provide a first fitted face model, and

the first marker is placed on the face of the performer subsequent to the display of the superimposed first distribution image; and

adjust a position of the superimposed first distribution image based on the first fitted face model.

2 . The information processing apparatus according to claim 1 , wherein

the face model includes second position information of each of a plurality of marker point groups,

one marker point group of the plurality of marker point groups include the plurality of marker points,

each of the plurality of marker point groups includes a priority to fit the face model,

the priority of each of the plurality of marker point groups is different,

the processor is further configured to display a plurality of distribution images including the first distribution image,

each of the plurality of distribution images indicates the second position information of a respective marker point group of the plurality of marker point groups,

each of the plurality of distribution images is associated with the priority of the respective marker point group,

a plurality of priorities includes the priority of each of the plurality of marker point groups, and

the plurality of distribution images is displayed in a descending order of the plurality of priorities associated with the plurality of distribution images.

3 . The information processing apparatus according to claim 2 , wherein

a first change associated with the second position information of a first marker point group of the plurality of marker point groups is least among a plurality of changes associated with the second position information of the plurality of marker point groups,

the plurality of changes associated with the second position information of the plurality of marker point groups are via a facial expression of the performer, and

the priority of the first marker point group is highest among the plurality of priorities.

4 . The information processing apparatus according to claim 2 , wherein the processor is further configured to:

extract position information of each of a plurality of landmarks from the face image; and

fit the face model based on the extracted third position information of each of the plurality of landmarks, wherein

the first distribution image indicates distribution of a first marker point group of the plurality of marker point groups,

the first marker point group includes the plurality of marker points, and

the priority of the first marker point group is higher among the plurality of priorities.

5 . The information processing apparatus according to claim 4 , wherein

the first marker point group includes at least one or more anatomical feature point,

the at least one anatomical feature point is in a portion of the face, and

the portion of the face exclude a jaw of the performer.

6 . The information processing apparatus according to claim 4 , wherein the processor is further configured to:

fit the face model, via the first position information and the extracted third position information, to provide a second fitted face model; and

adjust the position of the superimposed first distribution image based on the second fitted face model.

7 . The information processing apparatus according to claim 6 , wherein

the processor is further configured to increase contribution of the first position information of the first marker based on a plurality of markers,

the plurality of markers includes the first marker, and

the plurality of markers is placed on the face of the performer.

8 . The information processing apparatus according to claim 4 , wherein the processor is further configured to:

fit the face model, via a plurality of first position information of a plurality of first markers, to provide a second fitted face model, wherein

the plurality of first position information includes the first position information,

the plurality of first markers is placed on the face of the performer,

the plurality of first markers includes the first marker, and

the plurality of first markers is associated with the plurality of marker points;

superimpose a second distribution image of the plurality of distribution images on the face image of the performer, wherein

the second distribution image is superimposed based on the second fitted face model,

the second distribution image indicates distribution of a second marker point group, and

the priority of the second marker point group is highest among the plurality of priorities except the priority of the first marker point group; and

display the superimposed second distribution image.

9 . The information processing apparatus according to claim 8 , wherein the second marker point group includes at least one anatomical feature point on a jaw of the performer.

10 . The information processing apparatus according to claim 9 , wherein the processor is further configured to:

superimpose the second fitted face model on the face image; and

display the superimposed second fitted face model along with the superimposed second distribution image.

11 . The information processing apparatus according to claim 8 , wherein the processor is further configured to:

fit the face model, via fourth position information of a second marker, to provide a third fitted face model, wherein

the second marker is on the face of the performer, and

the third fitted face model is provided subsequent to the display of the superimposed second distribution image; and

adjust a position of the superimposed second distribution image based on the third fitted face model.

12 . The information processing apparatus according to claim 8 , wherein the processor is further configured to:

fit the face model, via a plurality of fifth position information of a plurality of second markers, to provide a third fitted face model, wherein

the second marker point group includes a plurality of second marker points,

the plurality of second markers is associated with the plurality of second marker points, and

the plurality of second markers is placed on the face of the performer,

superimpose a third distribution image of the plurality of distribution images on the face image of the performer, wherein

the third distribution image is superimposed based on the third fitted face model,

the third distribution image indicates distribution of a third marker point group, and

the priority of the third marker point group is highest among the plurality of priorities except the priority of the first marker point group and the priority of the second marker point group; and

display the superimposed third distribution image.

13 . The information processing apparatus according to claim 12 , wherein the processor is further configured to:

fit the face model, via sixth position information of a third marker, to provide a fourth fitted face model, wherein

the third marker is placed on the face of the performer, and

the fourth fitted face model is provided subsequent to the display of the superimposed third distribution image; and

adjust a seventh position of the superimposed third distribution image based on the fourth fitted face model.

14 . The information processing apparatus according to claim 1 , wherein the processor is further configured to:

determine that a distance between a position of a second marker and a position of a marker point of the plurality of marker points is greater than a threshold value, wherein

the position of the second marker is on the face of the performer, and

the second marker is associated with the marker point; and

notify of the second marker based on the determination.

15 . An information processing method, comprising:

extracting distribution of a plurality of marker points from a face model of a performer;

superimposing a distribution image on a face image of the performer, wherein the distribution image indicates the extracted distribution of the plurality of marker points;

displaying the superimposed distribution image;

fitting the face model to the face image of the performer, based on position information of a marker placed on a face of the performer, wherein

the face model is fitted to provide a fitted face model, and

the marker is placed on the face of the performer subsequent to the display of the superimposed distribution image; and

adjusting a position of the superimposed distribution image based on the fitted face model.

16 . A non-transitory computer-readable medium having stored thereon computer-executable instructions, that when executed by a computer, causes the computer to execute operations, the operations comprising:

extracting distribution of a plurality of marker points from a face model of a performer;

superimposing a distribution image on a face image of the performer, wherein the distribution image indicates the extracted distribution of the plurality of marker points;

displaying the superimposed distribution image; and

fitting the face model to the face image of the performer, based on position information of a marker placed on a face of the performer, wherein

the face model is fitted to provide a fitted face model, and

the marker is placed on the face of the performer subsequent to the display of the superimposed distribution image; and

adjusting a position of the superimposed distribution image based on the fitted face model.