IP Library › Granted Patent US 12,267,287
Granted Patent B2
US 12,267,287 · App. 18/318,439 · Granted Apr 1, 2025

Post-capture processing in a messaging system

Inventors: Jean Luo (Seattle, WA); Oleksandr Grytsiuk (Marina del Rey, CA); Ivan Golub (Playa Vista, CA); Mykhailo Sokolov (Marina del Rey, CA)
Assignee: Snap Inc.
H04L51/10G06F3/04883G06T3/60G06T7/70G06T11/60H04L51/52
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,267,287
App. No.
18/318,439
Granted
Apr 1, 2025
Kind
B2
Abstract

The subject technology retrieves, by a client device from a storage device, first image data captured by the client device at a previous time. The subject technology receiving first metadata corresponding to a selected image processing operation. The subject technology generates second image data based on the first metadata and the image processing operation performed on the first image data. The subject technology generates second metadata comprising information related to the image processing operation, the second metadata including a first identifier associated with the first image data, and second identifier associated with the second image data. The subject technology generates a message comprising the second metadata, the second image data, and the first image data.

Claims (56)

1. A method, comprising:

retrieving, by a client device from a storage device, first image data captured by the client device;

generating second image data based on first metadata indicating a selected image processing operation and the selected image processing operation being performed on the first image data;

generating second metadata comprising information corresponding to the image processing operation, a first identifier associated with the first image data, and a second identifier associated with the second image data, the second metadata further comprising an identifier of at least one media overlay utilized with the image processing operation and media overlay metadata, the media overlay metadata including information regarding the at least one media overlay that is applied to the first image data, wherein the second metadata further comprises gesture information indicating a tap count indicating a number of touch inputs received; and

generating a message comprising the second metadata, the second image data, and the first image data, wherein the second image data comprises modified image data after the at least one media overlay is utilized with the image processing operation on the first image data, the first image data comprises unmodified image data prior to the modified image data from the second image data; and

sending the message to a second client device, the message including the second metadata, the second image data comprising the modified image data, and including the first image data, wherein the second metadata further comprises a media overlay name, a media overlay type, a media overlay index, a number of media overlays, a media overlay score, and a media overlay group.

2. The method of claim 1 , further comprising:

receiving second metadata corresponding to a second selected image processing operation, the second selected image processing operation being different than the selected image processing operation;

generating third image data based on the second metadata and the selected second image processing operation performed on the first image data, the third image data comprising second modified image data after the selected second image processing operation is performed on the first image data;

generating third metadata comprising information corresponding to the second selected image processing operation, the first identifier associated with the first image data, and a third identifier associated with the third image data, the third metadata further comprising a second identifier of a second media overlay utilized with the second selected image processing operation and second media overlay metadata, the second media overlay metadata including information regarding the second media overlay that is applied to the first image data;

generating composite image data based on a combination of the second image data and the third image data; and

generating a second message comprising the third metadata, the second metadata, the composite image data, and the first image data, and the first image data being stored in the storage device at a previous time and retrieved from the storage device at a subsequent time when generating the message.

3. The method of claim 1 , further comprising:

receiving sensor data indicating an orientation of the client device corresponding to the first image data, wherein generating second image data is further based on the orientation of the client device; and

storing, in the message, the second image data separately from the first image data, wherein the first image data includes at least one representation of an original facial feature of a person, the second image data is different than the first image data, the second image data comprises a modified facial feature of the original facial feature based at least in part on the selected image processing operation.

4. The method of claim 1 , wherein the selected image processing operation is selected via a display of the client device using a touch input-swipe gesture.

5. The method of claim 1 , wherein the selected image processing operation corresponds to a media overlay operation that belongs to a particular group of media overlay operations based on a category associated with the media overlay operations.

6. The method of claim 1 , further comprising:

rendering, for display by the client device, the second image data comprising a first image corresponding to the first image data modified by the selected image processing operation.

7. The method of claim 1 , wherein the first image data includes a representation of an environment.

8. The method of claim 7 , wherein generating second image data further includes:

performing a tracking process based on the first image data, wherein the tracking process comprises:

tracking a surface recognized in the representation of the environment,

tracking a face recognized in the representation of the environment, or

tracking an object using a machine learning model.

9. The method of claim 1 , wherein the gesture information further indicates a swipe direction, or a gesture sequence count indicating a number of swipes.

10. A system comprising:

a processor;

a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:

retrieving, by a client device from a storage device, first image data captured by the client device;

generating second image data based on first metadata indicating a selected image processing operation and the selected image processing operation being performed on the first image data;

generating second metadata comprising information corresponding to the image processing operation, a first identifier associated with the first image data, and a second identifier associated with the second image data, the second metadata further comprising an identifier of at least one media overlay utilized with the image processing operation and media overlay metadata, the media overlay metadata including information regarding the at least one media overlay that is applied to the first image data, wherein the second metadata further comprises gesture information indicating a tap count indicating a number of touch inputs received; and

generating a message comprising the second metadata, the second image data, and the first image data, wherein the second image data comprises modified image data after the at least one media overlay is utilized with the image processing operation on the first image data, the first image data comprises unmodified image data prior to the modified image data from the second image data; and

sending the message to a second client device, the message including the second metadata, the second image data comprising the modified image data, and including the first image data, wherein the second metadata further comprises a media overlay name, a media overlay type, a media overlay index, a number of media overlays, a media overlay score, and a media overlay group.

11. The system of claim 10 , wherein the memory includes further instructions, which further cause the processor to perform further operations comprising:

sending, to a server, the message for sharing with one or more other client devices.

12. The system of claim 10 , wherein the memory includes further instructions, which further cause the processor to perform further operations comprising:

receiving sensor data including an orientation of the client device corresponding to the first image data, the sensor data including metadata associated with the first image data, wherein generating second image data is further based on the sensor data, the received sensor data indicating an orientation of the client device.

13. The system of claim 10 , wherein the memory includes further instructions, which further cause the processor to perform further operations comprising:

storing the second image data separately from the first image data.

14. The system of claim 10 , wherein the selected image processing operation is selected via a display of the client device using a touch input.

15. The system of claim 10 , wherein the selected image processing operation corresponds to a media overlay operation that belongs to a particular group of media overlay operations based on a category associated with the media overlay operations.

16. The system of claim 10 , wherein the memory includes further instructions, which further cause the processor to perform further operations comprising:

rendering, for display by the client device, the second image data comprising a first image corresponding to the first image data modified by the selected image processing operation.

17. The system of claim 10 , wherein the first image data includes a representation of an environment.

18. The system of claim 17 , wherein to generate second image data further causes the processor to perform further operations comprising:

performing a tracking process based on the first image data, wherein the tracking process comprises:

tracking a surface recognized in the representation of the environment,

tracking a face recognized in the representation of the environment, or

tracking an object using a machine learning model.

19. A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:

retrieving, by a client device from a storage device, first image data captured by the client device;

generating second image data based on first metadata indicating a selected image processing operation and the selected image processing operation being performed on the first image data;

generating second metadata comprising information corresponding to the image processing operation, a first identifier associated with the first image data, and a second identifier associated with the second image data, the second metadata further comprising an identifier of at least one media overlay utilized with the image processing operation and media overlay metadata, the media overlay metadata including information regarding the at least one media overlay that is applied to the first image data, wherein the second metadata further comprises gesture information indicating a tap count indicating a number of touch inputs received; and

generating a message comprising the second metadata, the second image data, and the first image data, wherein the second image data comprises modified image data after the at least one media overlay is utilized with the image processing operation on the first image data, the first image data comprises unmodified image data prior to the modified image data from the second image data; and

sending the message to a second client device, the message including the second metadata, the second image data comprising the modified image data, and including the first image data, wherein the second metadata further comprises a media overlay name, a media overlay type, a media overlay index, a number of media overlays, a media overlay score, and a media overlay group.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 16, 2023
From: LUO, JEAN; GRYTSIUK, OLEKSANDR; GOLUB, IVAN; SOKOLOV, MYKHAILO
To: SNAP INC.
Reel/Frame 063658/0397 →
Continuity (2)
Continuation 16732025 · Dec 31, 2019
Related Publication 20230291702A1 · Sep 14, 2023
References Cited (125)
US 8752113B1 · Good et al. · 2014 [cited by applicant]
US 10755036B1 · Sheth et al. · 2020 [cited by applicant]
US 11164353B2 · Luo et al. · 2021 [cited by applicant]
US 11237702B2 · Luo et al. · 2022 [cited by applicant]
US 11695718B2 · Luo et al. · 2023 [cited by applicant]
US 11750546B2 · Luo et al. · 2023 [cited by applicant]
US 11756249B2 · Luo et al. · 2023 [cited by applicant]
US 12034687B2 · Luo et al. · 2024 [cited by applicant]
US 20040125423A1 · Nishi et al. · 2004 [cited by applicant]
US 20100036967A1 · Caine et al. · 2010 [cited by applicant]
US 20130083215A1 · Wisniewski · 2013 [cited by applicant]
US 20130111337A1 · Deng et al. · 2013 [cited by applicant]
US 20130235069A1 · Ubillos et al. · 2013 [cited by applicant]
US 20130236093A1 · Gatt et al. · 2013 [cited by applicant]
US 20130239056A1 · Ubillos et al. · 2013 [cited by applicant]
US 20160061623A1 · Pahwa et al. · 2016 [cited by applicant]
US 20160188131A1 · Ubillos et al. · 2016 [cited by applicant]
US 20170032577A1 · Smith et al. · 2017 [cited by applicant]
US 20170294038A1 · Moriya · 2017 [cited by applicant]
US 20170352092A1 · Mitchell et al. · 2017 [cited by applicant]
US 20180060263A1 · Radhakrishnan et al. · 2018 [cited by applicant]
US 20180143748A1 · Ahmed et al. · 2018 [cited by applicant]
US 20180164986A1 · Al Majid et al. · 2018 [cited by applicant]
US 20180207885A1 · Liu et al. · 2018 [cited by applicant]
US 20180211444A1 · Shaviv et al. · 2018 [cited by applicant]
US 20180225024A1 · Tsai · 2018 [cited by examiner]
US 20180241704A1 · Al Majid et al. · 2018 [cited by applicant]
US 20180335922A1 · Nilo · 2018 [cited by examiner]
US 20190066365A1 · Schmalstieg et al. · 2019 [cited by applicant]
US 20190073115A1 · Moll et al. · 2019 [cited by applicant]
US 20190190865A1 · Jeon et al. · 2019 [cited by applicant]
US 20190207884A1 · Kozhemiak et al. · 2019 [cited by applicant]
US 20190207885A1 · Kozhemiak et al. · 2019 [cited by applicant]
US 20190250934A1 · Kim et al. · 2019 [cited by applicant]
US 20190332522A1 · Leydon et al. · 2019 [cited by applicant]
US 20190379617A1 · Luo et al. · 2019 [cited by applicant]
US 20200090246A1 · Goyal et al. · 2020 [cited by applicant]
US 20210065454A1 · Goodrich et al. · 2021 [cited by applicant]
US 20210200390A1 · Luo et al. · 2021 [cited by applicant]
US 20210201545A1 · Luo et al. · 2021 [cited by applicant]
US 20210203627A1 · Luo et al. · 2021 [cited by applicant]
US 20210203628A1 · Luo et al. · 2021 [cited by applicant]
US 20220020194A1 · Luo et al. · 2022 [cited by applicant]
US 20230353520A1 · Luo et al. · 2023 [cited by applicant]
US 20230377233A1 · Luo et al. · 2023 [cited by applicant]
US 20240323154A1 · Luo et al. · 2024 [cited by applicant]
CN 104247392 · 2014 [cited by applicant]
CN 107077608 · 2017 [cited by applicant]
CN 108351973 · 2018 [cited by applicant]
CN 110189248A · 2019 [cited by applicant]
CN 114902171A · 2022 [cited by applicant]
WO WO2013133895A1 · 2013 [cited by applicant]
WO WO2018147741A1 · 2018 [cited by applicant]
WO 2018213322 · 2018 [cited by applicant]
WO WO2021138630A1 · 2021 [cited by applicant]
“U.S. Appl. No. 16/732,001, Final Office Action mailed Jan. 7, 2021”, 9 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,001, Non Final Office Action mailed Jun. 26, 2020”, 7 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,001, Notice of Allowance mailed May 28, 2021”, 6 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,001, Notice of Allowance mailed Sep. 22, 2021”, 5 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,001, PTO Response to Rule 312 Communication mailed Aug. 19, 2021”, 2 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,001, Response filed May 7, 2021 to Final Office Action mailed Jan. 7, 2021”, 12 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,001, Response filed Sep. 28, 2020 to Non Final Office Action mailed Jun. 26, 2020”, 10 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,001, Supplemental Notice of Allowability mailed Jun. 16, 2021”, 2 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,025, Examiner Interview Summary mailed Jun. 24, 2021”, 2 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,025, Final Office Action mailed May 3, 2022”, 23 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,025, Final Office Action mailed Jul. 12, 2021”, 17 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,025, Non Final Office Action mailed Jan. 7, 2022”, 18 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,025, Non Final Office Action mailed Mar. 22, 2021”, 12 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,025, Non Final Office Action mailed Oct. 11, 2022”, 22 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,025, Notice of Allowance mailed Feb. 9, 2023”, 9 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,025, Response filed Jan. 11, 2023 to Non Final Office Action mailed Oct. 11, 2022”, 14 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,025, Response filed Apr. 7, 2022 to Non Final Office Action mailed Jan. 7, 2022”, 15 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,025, Response filed Jun. 22, 2021 to Non Final Office Action mailed Mar. 22, 2021”, 11 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,025, Response filed Aug. 3, 2022 to Final Office Action mailed May 3, 2022”, 15 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,025, Response filed Oct. 12, 2021 to Final Office Action mailed Jul. 12, 2021”, 12 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,040, Examiner Interview Summary mailed Jul. 1, 2021”, 2 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,040, Final Office Action mailed Sep. 8, 2022”, 13 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,040, Final Office Action mailed Oct. 26, 2021”, 12 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,040, Non Final Office Action mailed Mar. 1, 2022”, 13 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,040, Non Final Office Action mailed Apr. 20, 2021”, 10 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,040, Notice of Allowance mailed Apr. 3, 2023”, 10 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,040, Response filed Jan. 9, 2023 to Final Office Action mailed Sep. 8, 2022”, 14 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,040, Response filed Jan. 26, 2022 to Final Office Action mailed Oct. 26, 2021”, 11 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,040, Response filed Jun. 1, 2022 to Non Final Office Action mailed Mar. 1, 2022”, 12 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,040, Response filed Jul. 20, 2021 to Non Final Office Action mailed Apr. 20, 2021”, 12 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,051, Corrected Notice of Allowability mailed Aug. 10, 2021”, 2 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,051, Examiner Interview Summary mailed Feb. 26, 2021”, 3 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,051, Examiner Interview Summary mailed Jun. 7, 2021”, 2 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,051, Final Office Action mailed Mar. 22, 2021”, 15 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,051, Non Final Office Action mailed Oct. 29, 2020”, 12 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,051, Notice of Allowance mailed Jun. 30, 2021”, 8 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,051, Response filed Mar. 1, 2021 to Non Final Office Action mailed Oct. 29, 2020”, 9 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,051, Response filed Jun. 14, 2021 to Final Office Action mailed Mar. 22, 2021”, 9 pgs. [cited by applicant]
“U.S. Appl. No. 17/489,339, Final Office Action mailed Jan. 17, 2023”, 8 pgs. [cited by applicant]
“U.S. Appl. No. 17/489,339, Non Final Office Action mailed Sep. 14, 2022”, 20 pgs. [cited by applicant]
“U.S. Appl. No. 17/489,339, Notice of Allowance mailed Apr. 27, 2023”, 9 pgs. [cited by applicant]
“U.S. Appl. No. 17/489,339, Response filed Apr. 17, 2023 to Final Office Action mailed Jan. 17, 2023”, 10 pgs. [cited by applicant]
“U.S. Appl. No. 17/489,339, Response filed Dec. 14, 2022 to Non Final Office Action mailed Sep. 14, 2022”, 12 pgs. [cited by applicant]
“International Application Serial No. PCT/US2020/070923, International Preliminary Report on Patentability mailed Jul. 14, 2022”, 12 pgs. [cited by applicant]
“International Application Serial No. PCT/US2020/070923, International Search Report mailed May 28, 2021”, 5 pgs. [cited by applicant]
“International Application Serial No. PCT/US2020/070923, International Search Report mailed Jun. 22, 2021”, 6 pgs. [cited by applicant]
“International Application Serial No. PCT/US2020/070923, Invitation to Pay Additional Fees mailed Apr. 6, 2021”, 11 pgs. [cited by applicant]
“International Application Serial No. PCT/US2020/070923, Written Opinion mailed May 28, 2021”, 10 pgs. [cited by applicant]
“International Application Serial No. PCT/US2020/070923, Written Opinion mailed Jun. 22, 2021”, 10 pgs. [cited by applicant]
U.S. Appl. No. 16/732,025, filed Dec. 31, 2019, Post-Capture Processing in a Messaging System. [cited by applicant]
U.S. Appl. No. 16/732,040, filed Dec. 31, 2019, Providing Post-Capture Media Overlays for Post-Capture Processing in a Messaging System. [cited by applicant]
U.S. Appl. No. 16/732,051 U.S. Pat. No. 11,164,353 filed Dec. 31, 2019, Layering of Post-Capture Processing in a Messaging System. [cited by applicant]
U.S. Appl. No. 17/489,339, filed Sep. 29, 2021, Layering of Post-Capture Processing in a Messaging System. [cited by applicant]
U.S. Appl. No. 16/732,001 U.S. Pat. No. 11,237,702, filed Dec. 31, 2019, Carousel Interface for Post-Capture Processing in a Messaging System. [cited by applicant]
U.S. Appl. No. 17/568,423, filed Jan. 4, 2022, Carousel Interface for Post Capture Processing in a Messaging System. [cited by applicant]
“U.S. Appl. No. 18/363,271, Response filed Sep. 19, 2024 to Non Final Office Action mailed Jun. 21, 2024”, 9 pgs. [cited by applicant]
“Chinese Application Serial No. 202080090849.6, Response to Examiner Telephone Interview Filed Aug. 26, 2024”, w/ English Claims, 16 pgs. [cited by applicant]
“Chinese Application Serial No. 202080090849.6, Office Action mailed Sep. 11, 2024”, w/ English translation, 20 pgs. [cited by applicant]
“Chinese Application Serial No. 202080090849.6, Response Filed Jul. 29, 2024 to Office Action mailed May 29, 2024”, w/ English Claims, 18 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,025, Corrected Notice of Allowability mailed Jun. 1, 2023”, 2 pgs. [cited by applicant]
“European Application Serial No. 20842914.2, Response filed Feb. 9, 2023 to Communication pursuant to Rules 161(1) and 162 EPC mailed Aug. 9, 2022”, 45 pgs. [cited by applicant]
“U.S. Appl. No. 16/732,040, Corrected Notice of Allowability mailed Aug. 9, 2023”, 2 pgs. [cited by applicant]
“Chinese Application Serial No. 202080090849.6, Office Action mailed Oct. 10, 2023”, w/ English Translation, 21 pgs. [cited by applicant]
“U.S. Appl. No. 18/350,121, Notice of Allowance mailed Mar. 8, 2024”, 9 pgs. [cited by applicant]
“U.S. Appl. No. 18/350,121, Corrected Notice of Allowability mailed Apr. 15, 2024”, 2 pgs. [cited by applicant]
“Chinese Application Serial No. 202080090849.6, Office Action mailed May 29, 2024”, w/ English Translation, 22 pgs. [cited by applicant]
“U.S. Appl. No. 18/363,271, Non Final Office Action mailed Jun. 21, 2024”, 18 pgs. [cited by applicant]
“U.S. Appl. No. 18/363,271, Final Office Action mailed Oct. 8, 2024”, 18 pgs. [cited by applicant]
“Korean Application Serial No. 10-2022-7026228, Notice of Preliminary Rejection mailed Sep. 27, 2024”, w/ English translation, 12 pgs. [cited by applicant]
“Chinese Application Serial No. 202080090849.6, Response filed Nov. 11, 2024 to Office Action mailed Sep. 11, 2024”, w/ current English claims, 17 pgs. [cited by applicant]
Cited By (1)
US 12,506,699