Granted Patent B2

US 12,748,490 · App. 19/016,442 · Granted Sep 29, 2026

Inventors: Kwanghyun Won (Suwon-si, KR); Cheolseong Park (Suwon-si, KR); Youngho Oh (Suwon-si, KR); Woongil Choi (Suwon-si, KR)

Assignee: Samsung Electronics Co., Ltd.

G06F3/013G06T7/20G06T7/70G06T2207/10004G06T2207/20024G06T2207/30201H04N13/122H04N13/383

Abstract

Claims (89)

1 . A method performed by an electronic device, the method comprising:

obtaining, by the electronic device, position information of a target part corresponding to a past time point, and position information of the target part corresponding to a reference time point, from an image that includes a facial region of a viewer and is input through a camera;

obtaining, by the electronic device, position change information of the target part corresponding to the reference time point, based on the position information of the target part corresponding to the past time point, and the position information of the target part corresponding to the reference time point;

predicting, by the electronic device, a future velocity of the target part based on the position change information of the target part corresponding to the reference time point, and position change information of the target part corresponding to the past time point;

obtaining, by the electronic device, velocity change information of the target part corresponding to the reference time point, based on the position change information of the target part corresponding to the reference time point, and the position change information of the target part corresponding to the past time point;

predicting, by the electronic device, a future acceleration of the target part based on the velocity change information of the target part corresponding to the reference time point, velocity change information of the target part corresponding to the past time point, and a movement state of the viewer, the movement state of the viewer being one among the viewer being stationary and the viewer moving;

predicting, by the electronic device, future positions of eyes corresponding to a target time point based on the future velocity and the future acceleration; and

outputting, by the electronic device, an image based on the future positions of the eyes corresponding to the target time point.

2 . The method of claim 1 , wherein the predicting of the future positions of the eyes corresponding to the target time point based on the future velocity and the future acceleration comprises:

predicting a first future position of the target part corresponding to the target time point based on the future velocity and the future acceleration;

obtaining a second future position of the target part corresponding to the target time point, based on the first future position of the target part corresponding to the target time point, and a future position of the target part corresponding to a first time point that is prior to the target time point; and

predicting the future positions of the eyes corresponding to the target time point based on the second future position of the target part.

3 . The method of claim 1 , wherein the predicting of the future positions of the eyes corresponding to the target time point based on the future velocity and the future acceleration comprises:

predicting a first future position of a center between the eyes corresponding to the target time point based on the future velocity and the future acceleration; and

predicting the future positions of the eyes corresponding to the target time point based on interpupillary distance (IPD) information and the first future position of the center between the eyes corresponding to the target time point.

4 . The method of claim 1 , wherein the obtaining of the position information of the target part corresponding to the past time point, and the position information of the target part corresponding to the reference time point, from the image that includes the facial region of the viewer and is input through the camera comprises:

obtaining raw position information of the target part corresponding to the reference time point, from the image including the facial region of the viewer; and

obtaining the position information of the target part corresponding to the reference time point, based on the position information of the target part corresponding to the past time point, and the raw position information of the target part.

5 . The method of claim 1 , wherein the obtaining of the position change information of the target part corresponding to the reference time point based on the position information of the target part corresponding to the past time point, and the position information of the target part corresponding to the reference time point comprises:

obtaining raw information about a change in a position of the target part corresponding to the reference time point, based on the position information of the target part corresponding to the past time point, and the position information of the target part corresponding to the reference time point; and

obtaining the position change information of the target part corresponding to the reference time point, based on a filtering strength determined based on a degree of movement of the target part, the position change information of the target part corresponding to the past time point, and the raw information about the change in the position of the target part.

6 . The method of claim 1 , wherein the predicting of the future acceleration of the target part based on the velocity change information of the target part corresponding to the reference time point, and the velocity change information of the target part corresponding to the past time point comprises:

obtaining nonlinearly predicted acceleration information of the target part corresponding to the target time point, based on the velocity change information of the target part corresponding to the reference time point, the velocity change information of the target part corresponding to the past time point, and a parameter value determined based on the movement state of the viewer; and

predicting the future acceleration of the target part based on the nonlinearly predicted acceleration information and an acceleration of the target part that is predicted with respect to a first time point prior to the target time point,

wherein the future acceleration of the target part corresponds to the target time point.

7 . The method of claim 1 , further comprising:

obtaining a scaling value for the future velocity based on a direction of the future velocity,

wherein the predicting of the future positions of the eyes corresponding to the target time point based on the future velocity and the future acceleration comprises predicting the future positions of the eyes corresponding to the target time point, based on the future velocity, the future acceleration and the scaling value for the future velocity.

8 . The method of claim 1 , further comprising:

obtaining a future position of the eyes corresponding to a second time point that is subsequent to the target time point,

wherein the outputting of the image based on the future positions of the eyes corresponding to the target time point comprises:

predicting future positions of the eyes with respect to the target time point, based on the future positions of the eyes corresponding to the target time point, and the future positions of the eyes corresponding to the second time point; and

outputting the image based on the future positions of the eyes with respect to the target time point.

9 . The method of claim 8 , further comprising:

obtaining future positions of the eyes corresponding to a third time point that is between the target time point and the second time point,

wherein the predicting of the future positions of the eyes with respect to the target time point, based on the future positions of the eyes corresponding to the target time point, and the future positions of the eyes corresponding to the second time point comprises predicting the future positions of the eyes with respect to the target time point, based on the future positions of the eyes corresponding to the target time point, the future positions of the eyes corresponding to the second time point, and the future positions of the eyes corresponding to the third time point.

10 . The method of claim 1 , further comprising:

predicting a future position of a center between the eyes corresponding to the target time point, based on a future acceleration of the center between the eyes, and a future velocity of the center between the eyes,

wherein the target part comprises at least one of a left eye or a right eye, and

wherein the outputting of the image based on the future positions of the eyes corresponding to the target time point comprises:

predicting the future positions of the eyes with respect to the target time point, based on the future positions of the eyes corresponding to the target time point, and the future position of the center between the eyes corresponding to the target time point, and

outputting the image based on the future positions of the eyes with respect to the target time point.

11 . The method of claim 10 , wherein the predicting of the future positions of the eyes with respect to the target time point, based on the future positions of the eyes corresponding to the target time point, and the future position of the center between the eyes corresponding to the target time point comprises:

predicting a future position of the center between the eyes with respect to the target time point, based on the future positions of the eyes corresponding to the target time point, and the future position of the center between the eyes corresponding to the target time point;

filtering out noise from the future position of the center between the eyes with respect to the target time point; and

predicting the future positions of the eyes with respect to the target time point, based on the future position of the center between the eyes from which the noise is filtered out.

12 . One or more non-transitory computer-readable storage media storing one or more computer programs including computer-executable instructions that, when executed by one or more processors of an electronic device individually or collectively, cause the electronic device to perform operations, the operations comprising:

obtaining, by the electronic device, position information of a target part corresponding to a past time point, and position information of the target part corresponding to a reference time point, from an image that includes a facial region of a viewer and is input through a camera;

obtaining, by the electronic device, position change information of the target part corresponding to the reference time point, based on the position information of the target part corresponding to the past time point, and the position information of the target part corresponding to the reference time point;

predicting, by the electronic device, a future velocity of the target part based on the position change information of the target part corresponding to the reference time point, and position change information of the target part corresponding to the past time point;

obtaining, by the electronic device, velocity change information of the target part corresponding to the reference time point, based on the position change information of the target part corresponding to the reference time point, and the position change information of the target part corresponding to the past time point;

predicting, by the electronic device, a future acceleration of the target part based on the velocity change information of the target part corresponding to the reference time point, velocity change information of the target part corresponding to the past time point, and a movement state of the viewer, the movement state of the viewer being one among the viewer being stationary and the viewer moving;

predicting, by the electronic device, future positions of eyes corresponding to a target time point based on the future velocity and the future acceleration; and

outputting, by the electronic device, an image based on the future positions of the eyes corresponding to the target time point.

13 . An electronic device comprising:

memory storing one or more computer programs; and

one or more processors communicatively coupled to the memory,

wherein the one or more computer programs include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to:

obtain position information of a target part corresponding to a past time point, and position information of the target part corresponding to a reference time point, from an image that includes a facial region of a viewer and is input through a camera,

obtain position change information of the target part corresponding to the reference time point, based on the position information of the target part corresponding to the past time point, and the position information of the target part corresponding to the reference time point,

predict a future velocity of the target part based on the position change information of the target part corresponding to the reference time point, and position change information of the target part corresponding to the past time point,

obtain velocity change information of the target part corresponding to the reference time point, based on the position change information of the target part corresponding to the reference time point, and the position change information of the target part corresponding to the past time point,

predict a future acceleration of the target part based on the velocity change information of the target part corresponding to the reference time point, velocity change information of the target part corresponding to the past time point, and a movement state of the viewer, the movement state of the viewer being one among the viewer being stationary and the viewer moving,

predict future positions of eyes corresponding to a target time point based on the future velocity and the future acceleration, and

output an image based on the future positions of the eyes corresponding to the target time point.

14 . The electronic device of claim 13 , wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to:

predict a first future position of the target part corresponding to the target time point based on the future velocity and the future acceleration,

obtain a second future position of the target part corresponding to the target time point, based on the first future position of the target part corresponding to the target time point, and a future position of the target part corresponding to a first time point that is prior to the target time point, and

predict the future positions of the eyes corresponding to the target time point based on the second future position of the target part.

15 . The electronic device of claim 13 , wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to:

predict a first future position of a center between the eyes corresponding to the target time point based on the future velocity and the future acceleration, and

predict the future positions of the eyes corresponding to the target time point based on interpupillary distance (IPD) information and the first future position of the center between the eyes corresponding to the target time point.

16 . The electronic device of claim 13 , wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to:

obtain raw position information of the target part corresponding to the reference time point, from the image including the facial region of the viewer, and

obtain the position information of the target part corresponding to the reference time point, based on the position information of the target part corresponding to the past time point, and the raw position information of the target part.

17 . The electronic device of claim 13 , wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to:

obtain raw information about a change in a position of the target part corresponding to the reference time point, based on the position information of the target part corresponding to the past time point, and the position information of the target part corresponding to the reference time point, and

obtain the position change information of the target part corresponding to the reference time point, based on a filtering strength determined based on a degree of movement of the target part, the position change information of the target part corresponding to the past time point, and the raw information about the change in the position of the target part.

18 . The electronic device of claim 13 , wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to:

obtain nonlinearly predicted acceleration information of the target part corresponding to the target time point, based on the velocity change information of the target part corresponding to the reference time point, the velocity change information of the target part corresponding to the past time point, and a parameter value determined based on the movement state of the viewer, and

predict the future acceleration of the target part based on the nonlinearly predicted acceleration information and an acceleration of the target part that is predicted with respect to a first time point prior to the target time point,

wherein the future acceleration of the target part corresponds to the target time point.

19 . The electronic device of claim 13 , wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to:

obtain a scaling value for the future velocity based on a direction of the future velocity, and

predict the future positions of the eyes corresponding to the target time point, based on the future velocity, the future acceleration and the scaling value for the future velocity.

20 . The electronic device of claim 13 , wherein the one or more computer programs further include computer-executable instructions that, when executed by the one or more processors individually or collectively, cause the electronic device to:

obtain a future position of the eyes corresponding to a second time point that is subsequent to the target time point,

predict future positions of the eyes with respect to the target time point, based on the future positions of the eyes corresponding to the target time point, and the future positions of the eyes corresponding to the second time point, and

output the image based on the future positions of the eyes with respect to the target time point.

Assignments (1)

Priority Claims (1)

Continuity (2)

References Cited (21)