Granted Patent B2

US 12,450,705 · App. 17/127,506 · Granted Oct 21, 2025

Inventors: Oran Gafni (Ramat Gan, IL); Lior Wolf (Herzliya, IL)

Assignee: Meta Platforms, Inc.

G06T5/75G06N20/00G06T5/70G06V40/172

Abstract

Claims (59)

1. A method comprising:

generating, based on an image of a person, a first identity encoding representing a first facial identity of the person;

generating a second identity encoding representing a second facial identity different from the first facial identity of the person by processing the first identity encoding with a first facial-identity altering machine-learning model comprising one or more fully connected layers and training data to facilitate generating the second identity encoding, wherein the training data comprises (i) pairs of original face images and corresponding altered face images, and (ii) pairs of images of a different first person and a second person;

generating a source encoding by using an encoder to process a source image of the person comprising an expression;

generating an intermediate image by using a decoder to process the source encoding and the second identity encoding, the intermediate image comprising a face comprising the second facial identity and the expression of the person in the source image; and

generating an output image by blending the source image with facial features of the face in the intermediate image.

2. The method of claim 1 , wherein a first machine-learning model is associated with the encoder and the decoder.

3. The method of claim 2 , further comprising:

utilizing the pairs of images as training data at an iteration of a training procedure associated with the first machine-learning model, wherein the pairs of images comprise a first image of the first person comprising a second expression and a second image of the second person.

4. The method of claim 3 , wherein the iteration of the training procedure comprises:

generating a training source encoding by using the encoder of the first machine-learning model to process a distorted first image of the pair;

generating a training identity encoding by using a pre-trained face recognition machine-learning model to process the second image of the pair;

generating a training intermediate image by using the decoder of the first machine-learning model to process the training source encoding and the training identity encoding;

generating a training output image by blending the first image with facial features of a face in the training intermediate image;

determining losses based on the training intermediate image or the training output image, wherein the losses comprise perceptual losses; and

updating trainable variables of the first machine-learning model based on the computed losses.

5. The method of claim 4 , wherein the distorted first image prevents the first machine-learning model from generating the training intermediate image identical to the first image.

6. The method of claim 4 , further comprising:

determining the perceptual losses at k different layers, and wherein the perceptual losses at i lowest layers out of the k layers are determined based on comparisons between the first image and the training output image, and wherein the perceptual losses at the remaining k−i layers are determined based on comparisons between the second image and the training output image.

7. The method of claim 6 , further comprising:

updating the trainable variables of the first machine-learning model to minimize the perceptual losses.

8. The method of claim 1 , further comprising:

generating the first identity encoding by processing the image with a pre-trained face recognition machine-learning model.

9. The method of claim 1 , further comprising:

training the first facial-identity altering machine-learning model with the pairs of original face images and the corresponding altered face images as the training data.

10. The method of claim 9 , further comprising:

generating an altered face image corresponding to an original face image by processing the original face image with the first facial-identity altering machine-learning model.

11. The method of claim 10 , wherein the first facial-identity altering machine-learning model comprises a second encoder and a second decoder.

12. The method of claim 1 , wherein the source image corresponds to a frame of a video stream.

13. The method of claim 1 , further comprising:

generating a blending mask in an instance in which the source encoding and the second identity encoding are processed by the decoder, and wherein the blending mask represents a blending weight to be applied to the intermediate image at one or more pixels of the output image.

14. The method of claim 13 , wherein blending the source image with facial features of the face in the intermediate image comprises:

creating one or more output images by applying an inverse of the blending mask to the source image; and

projecting the blending mask applied to the intermediate image to the one or more output images.

15. A non-transitory computer-readable medium storing instructions that, when executed, cause:

generating, based on an image of a person, a first identity encoding representing a first facial identity of the person;

generating a second identity encoding representing a second facial identity different from the first facial identity of the person by processing the first identity encoding with a first facial-identity altering machine-learning model comprising one or more fully connected layers and comprising training data to facilitate generating the second identity encoding, wherein the training data comprises (i) pairs of original face images and corresponding altered face images, and (ii) pairs of images of a different first person and a second person;

generating a source encoding by using an encoder to process a source image of the person comprising an expression;

generating an intermediate image by using a decoder to process the source encoding and the second identity encoding, the intermediate image comprising a face comprising the second facial identity and the expression of the person in the source image; and

generating an output image by blending the source image with facial features of the face in the intermediate image.

16. The computer-readable medium of claim 15 , wherein a first machine-learning model is associated with the encoder and the decoder.

17. The computer-readable medium of claim 16 , wherein the instructions, when executed, further cause:

utilizing the pairs of images as training data at an iteration of a training procedure associated with the first machine-learning model, wherein the pairs of images comprise a first image of the first person comprising a second expression and a second image of the second person.

18. The computer-readable medium of claim 17 , wherein the iteration of the training procedure comprises:

generating a training source encoding by using the encoder of the first machine-learning model to process a distorted first image of the pair;

generating a training identity encoding by using a pre-trained face recognition machine-learning model to process the second image of the pair;

generating a training intermediate image by using the decoder of the first machine-learning model to process the training source encoding and the training identity encoding;

generating a training output image by blending the first image with facial features of a face in the training intermediate image;

determining losses based on the training intermediate image or the training output image, wherein the losses comprise perceptual losses; and

updating trainable variables of the first machine-learning model based on the computed losses.

19. A system comprising:

one or more processors; and

at least one non-transitory memory comprising instructions executable by the one or more processors, the one or more processors operable when executing the instructions to:

generate, based on an image of a person, a first identity encoding representing a first facial identity of the person;

generate a second identity encoding representing a second facial identity different from the first facial identity of the person by processing the first identity encoding with a first facial-identity altering machine-learning model comprising one or more fully connected layers and comprising training data to facilitate generating the second identity encoding, wherein the training data comprises (i) pairs of original face images and corresponding altered face images, and (ii) pairs of images of a different first person and a second person;

generate a source encoding by using an encoder to process a source image of the person comprising an expression;

generate an intermediate image by using a decoder to process the source encoding and the second identity encoding, the intermediate image comprising a face comprising the second facial identity and the expression of the person in the source image; and

generate an output image by blending the source image with facial features of the face in the intermediate image.

20. The system of claim 19 , wherein the encoder and the decoder belong to a first machine-learning model.

Assignments (2)

Continuity (1)

References Cited (68)