Granted Patent B2

US 12,639,877 · App. 18/057,117 · Granted May 26, 2026

Inventors: Wenbo Li (Santa Clara, CA); Yi Wei (Sunnyvale, CA); Jing Zhu (Jackson Heights, NY); Hongxia Jin (San Jose, CA)

Assignee: Samsung Electronics Co., Ltd.

G06T13/80G06T7/246G06T7/70G06V10/82G06V40/171G06V40/176G06T2207/10016G06T2207/30201G06V2201/10

Abstract

Claims (86)

1 . A method comprising:

obtaining a source image of a user;

obtaining a driving video in which a face or a head of the user is moving;

generating metadata identifying animations to be applied to the source image so that the source image mimics at least some movements of the user's face or head in the driving video, wherein the metadata comprises raw metadata determined from the driving video and refined metadata determined from the raw metadata, and wherein the raw metadata describes expression deformations of the user's face captured by the driving video; and

transmitting the source image and the metadata to an end user device configured to animate the source image based on the metadata;

wherein generating the metadata comprises:

generating expression deformations for facial keypoints in each image frame of the driving video using an expression deformation function;

generating refined expression deformations for the facial keypoints using a refinement model based on the expression deformations;

identifying primary facial keypoints from the facial keypoints;

replacing the refined expression deformations related to the primary facial keypoints with the expression deformations generated by the expression deformation function related to the primary facial keypoints using a merge function to produce merged metadata; and

suppressing one or more artifacts associated with one or more objects that temporarily occlude at least a portion of the user's head or body in the driving video or that temporarily appear in the driving video, the one or more artifacts suppressed by adjusting one or more subsequent sets of raw metadata associated with the driving video based on one or more previous sets of raw metadata associated with the driving video that do not contain noisy metadata immediately preceding the one or more subsequent sets of raw metadata; and

wherein suppressing the one or more artifacts associated with the one or more objects comprises replacing a first portion of the refined metadata describing at least one of the expression deformations of the user's face with a second portion of the raw metadata describing the at least one of the expression deformations of the user's face.

2 . The method of claim 1 , wherein generating the metadata further comprises:

for each of multiple image frames in the driving video, determining the raw metadata, wherein, for each image frame, the raw metadata describes a pose of the user's head and the expression deformation of the user's face in the image frame, wherein the raw metadata associated with at least one image frame in the driving video includes noise from the one or more objects that temporarily occlude at least the portion of the user's head or body or that temporarily appear in the driving video; and

providing the raw metadata to a trained machine learning model to generate the refined metadata, wherein the trained machine learning model is trained to suppress effects of the one or more objects that temporarily occlude at least the portion of the user's head or body or that temporarily appear in the driving video.

3 . The method of claim 2 , wherein generating the metadata further comprises:

identifying facial keypoints associated with the user's face in the source image; and

for each of the multiple image frames in the driving video, transforming the facial keypoints associated with the user's face based on the corresponding refined metadata to produce streaming facial keypoints, the streaming facial keypoints forming part of the metadata.

4 . The method of claim 3 , wherein transforming the facial keypoints associated with the user's face for each of the multiple image frames in the driving video is based on the merged metadata associated with the image frame.

5 . The method of claim 3 , wherein generating the metadata further comprises:

determining additional metadata describing a pose of the user's head and an expression deformation of the user's face in the source image; and

transforming the facial keypoints associated with the user's face based on the additional metadata to produce source facial keypoints, the source facial keypoints forming part of the metadata.

6 . The method of claim 2 , wherein the trained machine learning model comprises one or more recurrent neural networks.

7 . The method of claim 6 , wherein the one or more recurrent neural networks comprise at least one of:

a first gated recurrent unit (GRU)-based recurrent neural network trained to receive raw pitch, yaw, and roll angles and output refined pitch, yaw, and roll angles;

a second GRU-based recurrent neural network trained to receive raw translation deformations and output refined translation deformations; and

a third GRU-based recurrent neural network trained to receive raw expression deformations and output refined expression deformations.

8 . An apparatus comprising:

at least one processing device configured to:

obtain a source image of a user;

obtain a driving video in which a face or a head of the user is moving;

generate metadata identifying animations to be applied to the source image so that the source image mimics at least some movements of the user's face or head in the driving video, wherein the metadata comprises raw metadata determined from the driving video and refined metadata determined from the raw metadata, and wherein the raw metadata describes expression deformations of the user's face captured by the driving video; and

initiate transmission of the source image and the metadata to an end user device configured to animate the source image based on the metadata;

wherein, to generate the metadata, the at least one processing device is configured to:

generate expression deformations for facial keypoints in each image frame of the driving video using an expression deformation function;

generate refined expression deformations for the facial keypoints using a refinement model based on the expression deformations;

identify primary facial keypoints from the facial keypoints;

replace the refined expression deformations related to the primary facial keypoints with the expression deformations generated by the expression deformation function related to the primary facial keypoints using a merge function to produce merged metadata; and

suppress one or more artifacts associated with one or more objects that temporarily occlude at least a portion of the user's head or body in the driving video or that temporarily appear in the driving video, the at least one processing device configured to suppress the one or more artifacts by adjusting one or more subsequent sets of raw metadata associated with the driving video based on one or more previous sets of raw metadata associated with the driving video that do not contain noisy metadata immediately preceding the one or more subsequent sets of raw metadata; and

wherein, to suppress the one or more artifacts associated with the one or more objects, the at least one processing device is configured to replace a first portion of the refined metadata describing at least one of the expression deformations of the user's face with a second portion of the raw metadata describing the at least one of the expression deformations of the user's face.

9 . The apparatus of claim 8 , wherein:

to generate the metadata, the at least one processing device is further configured, for each of multiple image frames in the driving video, to determine the raw metadata, wherein, for each image frame, the raw metadata describes a pose of the user's head and the expression deformation of the user's face in the image frame, wherein the raw metadata associated with at least one image frame in the driving video includes noise from the one or more objects that temporarily occlude at least the portion of the user's head or body or that temporarily appear in the driving video; and

the at least one processing device is configured to provide the raw metadata to a trained machine learning model to generate the refined metadata, wherein the trained machine learning model is trained to suppress effects of the one or more objects that temporarily occlude at least the portion of the user's head or body or that temporarily appear in the driving video.

10 . The apparatus of claim 9 , wherein, to generate the metadata, the at least one processing device is further configured to:

identify facial keypoints associated with the user's face in the source image; and

for each of the multiple image frames in the driving video, transform the facial keypoints associated with the user's face based on the corresponding refined metadata to produce streaming facial keypoints, the streaming facial keypoints forming part of the metadata.

11 . The apparatus of claim 10 , wherein: the at least one processing device is configured to transform the facial keypoints associated with the user's face for each of the multiple image frames in the driving video based on the merged metadata associated with the image frame.

12 . The apparatus of claim 10 , wherein, to generate the metadata, the at least one processing device is further configured to:

determine additional metadata describing a pose of the user's head and an expression deformation of the user's face in the source image; and

transform the facial keypoints associated with the user's face based on the additional metadata to produce source facial keypoints, the source facial keypoints forming part of the metadata.

13 . The apparatus of claim 9 , wherein the trained machine learning model comprises one or more recurrent neural networks.

14 . The apparatus of claim 13 , wherein the one or more recurrent neural networks comprise at least one of:

a first gated recurrent unit (GRU)-based recurrent neural network trained to receive raw pitch, yaw, and roll angles and output refined pitch, yaw, and roll angles;

a second GRU-based recurrent neural network trained to receive raw translation deformations and output refined translation deformations; and

a third GRU-based recurrent neural network trained to receive raw expression deformations and output refined expression deformations.

15 . A non-transitory computer readable medium containing instructions that when executed cause at least one processor to:

obtain a source image of a user;

obtain a driving video in which a face or a head of the user is moving;

generate metadata identifying animations to be applied to the source image so that the source image mimics at least some movements of the user's face or head in the driving video, wherein the metadata comprises raw metadata determined from the driving video and refined metadata determined from the raw metadata, and wherein the raw metadata describes expression deformations of the user's face captured by the driving video; and

initiate transmission of the source image and the metadata to an end user device configured to animate the source image based on the metadata;

wherein the instructions that when executed cause the at least one processor to generate the metadata comprise:

instructions that when executed cause the at least one processor to:

generate expression deformations for facial keypoints in each image frame of the driving video using an expression deformation function;

generate refined expression deformations for the facial keypoints using a refinement model based on the expression deformations;

identify primary facial keypoints from the facial keypoints:

replace the refined expression deformations related to the primary facial keypoints with the expression deformations generated by the expression deformation function related to the primary facial keypoints using a merge function to produce merged metadata; and

suppress one or more artifacts associated with one or more objects that temporarily occlude at least a portion of the user's head or body in the driving video or that temporarily appear in the driving video by adjusting one or more subsequent sets of raw metadata associated with the driving video based on one or more previous sets of raw metadata associated with the driving video that do not contain noisy metadata immediately preceding the one or more subsequent sets of raw metadata; and

wherein the instructions that when executed cause the at least one processor to suppress the artifacts associated with the one or more objects comprise:

instructions that when executed cause the at least one processor to replace a first portion of the refined metadata describing at least one of the expression deformations of the user's face with a second portion of the raw metadata describing the at least one of the expression deformations of the user's face.

16 . The non-transitory computer readable medium of claim 15 , wherein:

the instructions that when executed cause the at least one processor to generate the metadata further comprise:

instructions that when executed cause the at least one processor, for each of multiple image frames in the driving video, to determine the raw metadata, wherein, for each image frame, the raw metadata describes a pose of the user's head and the expression deformation of the user's face in the image frame, wherein the raw metadata associated with at least one image frame in the driving video includes noise from the one or more objects that temporarily occlude at least the portion of the user's head or body or that temporarily appear in the driving video; and

the instructions when executed cause the at least one processor to provide the raw metadata to a trained machine learning model to generate the refined metadata, wherein the trained machine learning model is trained to suppress effects of the one or more objects that temporarily occlude at least the portion of the user's head or body or that temporarily appear in the driving video.

17 . The non-transitory computer readable medium of claim 16 , wherein the instructions that when executed cause the at least one processor to generate the metadata further comprise:

instructions that when executed cause the at least one processor to:

identify facial keypoints associated with the user's face in the source image; and

for each of the multiple image frames in the driving video, transform the facial keypoints associated with the user's face based on the corresponding refined metadata to produce streaming facial keypoints, the streaming facial keypoints forming part of the metadata.

18 . The non-transitory computer readable medium of claim 17 , wherein: the instructions when executed cause the at least one processor to transform the facial keypoints associated with the user's face for each of the multiple image frames in the driving video based on the merged metadata associated with the image frame.

19 . The non-transitory computer readable medium of claim 17 , wherein the instructions that when executed cause the at least one processor to generate the metadata further comprise:

instructions that when executed cause the at least one processor to:

determine additional metadata describing a pose of the user's head and an expression deformation of the user's face in the source image; and

transform the facial keypoints associated with the user's face based on the additional metadata to produce source facial keypoints, the source facial keypoints forming part of the metadata.

20 . The non-transitory computer readable medium of claim 16 , wherein the trained machine learning model comprises:

a first gated recurrent unit (GRU)-based recurrent neural network trained to receive raw pitch, yaw, and roll angles and output refined pitch, yaw, and roll angles;

a second GRU-based recurrent neural network trained to receive raw translation deformations and output refined translation deformations; and

a third GRU-based recurrent neural network trained to receive raw expression deformations and output refined expression deformations.

Assignments (1)

Continuity (2)

References Cited (30)