IP Library › Granted Patent US 11,756,249
Granted Patent B2
US 11,756,249 · App. 17/489,339 · Granted Sep 12, 2023

Layering of post-capture processing in a messaging system

Inventors: Jean Luo (Seattle, WA); Oleksandr Grytsiuk (Marina del Rey, CA); Chenguang Liu (Los Angeles, CA); Oleksii Gordiienko (Toronto, CA)
Assignee: Snap Inc.
G06T11/60G06T1/20G06T5/50H04L51/063H04L51/10H04L51/52
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,756,249
App. No.
17/489,339
Granted
Sep 12, 2023
Kind
B2
Abstract

Systems and methods described herein provide for retrieving, from a storage device, first image data previously captured by a client device. The systems and methods further detect a selection of a first image processing operation and perform the first image processing operation on the first image data to generate second image data. The systems and methods further detect a selection of a second image processing operation and perform the second image processing operation on the second image data to generate third image data. The systems and methods generate a message comprising the third image data.

Claims (65)

1. A method, comprising:

accessing, by one or more hardware processors, first image data;

identifying, by the one or more hardware processors, based on the first image data, a target facial feature;

detecting, by the one or more hardware processors, a user selection of an image processing operation to modify the target facial feature with a media overlay;

performing, by the one or more hardware processors, the image processing operation on the first image data using the media overlay to generate second image data, the second image data including a modified target facial feature of a first face;

causing, by the one or more hardware processors, display of the second image data on a user interface of a device;

performing, by the one or more hardware processors, different image processing operation on the second image data using a particular media overlay to generate particular image data, the particular image data including graphical elements that occlude multiple faces different from the first face, the graphical elements corresponding to the different image processing operation being different than the modified target facial features of the first face corresponding to the image processing operation; and

causing, by the one or more hardware processors, display of the particular image data on the user interface of the device, the particular image data comprising a composite image including the modified target facial features of the first face and the graphical elements that occlude the multiple faces.

2. The method of claim 1 , wherein the image processing operation on the first image data using the media overlay is performed using a machine learning model.

3. The method of claim 1 , further comprising:

using a machine learning model to generate a reference facial feature based on training data.

4. The method of claim 3 , wherein identifying the target facial feature comprises:

identifying, based on the training data, a facial landmark of a face included in an image represented by the first image data; and

determining, based on the facial landmark, that the reference facial feature aligns with the target facial feature.

5. The method of claim 1 , further comprising:

determining that the first image data represents the first face and a second face, the first face including the target facial feature;

detecting a first user selection of the first face to apply the image processing operation; and

based on the first user selection of the first face and the user selection of the image processing operation, performing the image processing operation on the first image data using the media overlay to generate the second image data.

6. The method of claim 5 , further comprising:

detecting a second user selection of the second face to apply a second image processing operation that corresponds to a second media overlay; and

performing the second image processing operation on the second image data using the second media overlay to generate third image data.

7. The method of claim 6 , further comprising:

causing display of the third image data on the user interface of the device.

8. The method of claim 1 , wherein the first image data corresponds to an image that is captured via a camera coupled to the device.

9. The method of claim 1 , further comprising:

generating a message comprising the second image data; and

sending, to a server, the message for sharing with one or more other devices.

10. A system comprising:

a processor;

a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:

accessing first image data;

identifying, based on the first image data, a target facial feature;

detecting a user selection of an image processing operation to modify the target facial feature with a media overlay;

performing the image processing operation on the first image data using the media overlay to generate second image data, the second image data including a modified target facial feature of a first face;

causing display of the second image data on a user interface of a device;

performing a different image processing operation on the second image data using a particular media overlay to generate particular image data, the particular image data including graphical elements that occlude multiple faces different from the first face, the graphical elements corresponding to the different image processing operation being different than the modified target facial features of the first face corresponding to the image processing operation; and

causing display of the particular image data on the user interface of the device, the particular image data comprising a composite image including the modified target facial features of the first face and the graphical elements that occlude the multiple faces.

11. The system of claim 10 , wherein the image processing operation on the first image data using the media overlay is performed using a machine learning model.

12. The system of claim 10 , wherein the operations further comprise:

using a machine learning model to generate a reference facial feature based on training data.

13. The system of claim 12 , wherein the operations of identifying the target facial feature further comprise:

identifying, based on the training data, a facial landmark of a face included in an image represented by the first image data; and

determining, based on the facial landmark, that the reference facial feature aligns with the target facial feature.

14. The system of claim 10 , wherein the operations further comprise:

determining that the first image data represents the first face and a second face, the first face including the target facial feature;

detecting a first user selection of the first face to apply the image processing operation; and

based on the first user selection of the first face and the user selection of the image processing operation, performing the image processing operation on the first image data using the media overlay to generate the second image data.

15. The system of claim 14 , wherein the operations further comprise:

detecting a second user selection of the second face to apply a second image processing operation that corresponds to a second media overlay; and

performing the second image processing operation on the second image data using the second media overlay to generate third image data.

16. The system of claim 15 , wherein the operations further comprise:

causing display of the third image data on the user interface of the device.

17. The system of claim 10 , wherein the first image data corresponds to an image that is captured via a camera coupled to the device.

18. The system of claim 10 , wherein the operations further comprise:

generating a message comprising the second image data; and

sending, to a server, the message for sharing with one or more other devices.

19. A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:

accessing first image data;

identifying, based on the first image data, a target facial feature;

detecting a user selection of an image processing operation to modify the target facial feature with a media overlay;

performing the image processing operation on the first image data using the media overlay to generate second image data, the second image data including a modified target facial feature of a first face;

causing display of the second image data on a user interface of a device;

performing a different image processing operation on the second image data using a particular media overlay to generate particular image data, the particular image data including graphical elements that occlude multiple faces different from the first face, the graphical elements corresponding to the different image processing operation being different than the modified target facial features of the first face corresponding to the image processing operation; and

causing display of the particular image data on the user interface of the device, the particular image data comprising a composite image including the modified target facial features of the first face and the graphical elements that occlude the multiple faces.

20. The non-transitory computer-readable medium of claim 19 , wherein the image processing operation on the first image data using the media overlay is performed using a machine learning model.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 15, 2021
From: LUO, JEAN; GRYTSIUK, OLEKSANDR; LIU, CHENGUANG; GORDIIENKO, OLEKSII
To: SNAP INC.
Reel/Frame 057801/0240 →
Continuity (2)
Continuation 16732051 · Dec 31, 2019
Related Publication 20220020194A1 · Jan 20, 2022
Cited By (3)
US 12,267,287 US 12,340,450 US 12,506,699