IP Library › Granted Patent US 12,456,224
Granted Patent B2
US 12,456,224 · App. 18/439,350 · Granted Oct 28, 2025

Augmented reality experiences of color palettes in a messaging system

Inventors: Jean Luo (Seattle, WA); Celia Nicole Mourkogiannis (Los Angeles, CA)
Assignee: Snap Inc.
G06T7/80G06T7/40G06T11/001G06T11/60G06V10/56G06V20/20G06V40/165G06V40/28H04L51/10
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,456,224
App. No.
18/439,350
Granted
Oct 28, 2025
Kind
B2
Abstract

The subject technology receives image data including a representation of a physical item. The subject technology analyzes the image data to determine an object corresponding to the physical item. The subject technology identifies a set of colors corresponding to a set of regions of the determined object. The subject technology analyzes second image data to detect a second object corresponding to a representation of a particular body part of a user. The subject technology generates augmented reality content based at least in part on the identified set of colors and the detected second object. The subject technology causes display, at a client device, the augmented reality content applied to the detected second object.

Claims (58)

1. A method, comprising:

analyzing first image data to determine an object corresponding to a beauty product, the analyzing based at least in part on a recognition process performed on the first image data wherein the recognition process further comprises:

classifying, using a first image classifier, the first image data to determine that the object includes a physical identification indicator, and

classifying, using a second image classifier, the first image data to perform object classification of the object based on a set of 3D models or images;

identifying a set of colors corresponding to the beauty product, and the identified set of colors corresponds to a second set of colors included in a portion of the first image data including a set of regions of the object, and wherein each region of the set of regions is associated with a particular color;

identifying a body part of a user based on second image data; and

generating augmented reality content based at least in part on the identified set of colors and the identified body part of the user.

2. The method of claim 1 , further comprising:

prior to analyzing the first image data, receiving the first image data, the first image data being captured by a camera of a client device.

3. The method of claim 1 , wherein the physical identification indicator comprises text, a barcode pattern, or a QR code pattern.

4. The method of claim 1 , further comprising:

prior identifying the body part of the user, receiving the second image data captured by a camera of a client device.

5. The method of claim 1 , wherein the beauty product corresponds to a makeup palette and the identified body part of the user comprises a forearm.

6. The method of claim 1 , wherein identifying the set of colors further comprises:

differentiating a set of colors values of pixels in a particular region from the set of regions of the beauty product; and

determining a representative color value based on an average of the set of colors.

7. The method of claim 6 , wherein generating the augmented reality content further comprises:

generating a shape based on the representative color value; and

mapping the shape to a surface of the body part.

8. The method of claim 1 , further comprising:

determining a set of textures based on pixels from the set of regions of the beauty product.

9. The method of claim 8 , wherein generating the augmented reality content further comprises:

performing a texture transfer process based on the set of textures to a surface of the body part.

10. The method of claim 1 , further comprising:

performing a body feature detection process to detect a first body feature; and

identifying a particular body part corresponding to the detected first body feature.

11. A system comprising:

a processor; and

a memory including instructions that, when executed by the processor, cause the processor to perform operations comprising:

analyzing first image data to determine an object corresponding to a beauty product, the analyzing based at least in part on a recognition process performed on the first image data wherein the recognition process further comprises:

classifying, using a first image classifier, the first image data to determine that the object includes a physical identification indicator, and

classifying, using a second image classifier, the first image data to perform object classification of the object based on a set of 3D models or images;

identifying a set of colors corresponding to the beauty product, and the identified set of colors corresponds to a second set of colors included in a portion of the first image data including a set of regions of the object, and wherein each region of the set of regions is associated with a particular color;

identifying a body part of a user based on second image data; and

generating augmented reality content based at least in part on the identified set of colors and the identified body part of the user.

12. The system of claim 11 , wherein the operations further comprise:

prior to analyzing the first image data, receiving the first image data, the first image data being captured by a camera of a client device.

13. The system of claim 11 , wherein the physical identification indicator comprises text, a barcode pattern, or a QR code pattern.

14. The system of claim 11 , wherein the operations further comprise:

prior identifying the body part of the user, receiving the second image data captured by a camera of a client device.

15. The system of claim 11 , wherein the beauty product corresponds to a makeup palette and the identified body part of the user comprises a forearm.

16. The system of claim 11 , wherein identifying the set of colors further comprises:

differentiating a set of colors values of pixels in a particular region from the set of regions of the beauty product; and

determining a representative color value based on an average of the set of colors.

17. The system of claim 16 , wherein generating the augmented reality content further comprises:

generating a shape based on the representative color value; and

mapping the shape to a surface of the body part.

18. The system of claim 11 , wherein the operations further comprise:

determining a set of textures based on pixels from the set of regions of the beauty product.

19. The system of claim 18 , wherein generating the augmented reality content further comprises:

performing a texture transfer process based on the set of textures to a surface of the body part.

20. A non-transitory computer-readable medium comprising instructions, which when executed by a computing device, cause the computing device to perform operations comprising:

analyzing first image data to determine an object corresponding to a beauty product, the analyzing based at least in part on a recognition process performed on the first image data wherein the recognition process further comprises:

classifying, using a first image classifier, the first image data to determine that the object includes a physical identification indicator, and

classifying, using a second image classifier, the first image data to perform object classification of the object based on a set of 3D models or images;

identifying a set of colors corresponding to the beauty product, and the identified set of colors corresponds to a second set of colors included in a portion of the first image data including a set of regions of the object, and wherein each region of the set of regions is associated with a particular color;

identifying a body part of a user based on second image data; and

generating augmented reality content based at least in part on the identified set of colors and the identified body part of the user.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 12, 2024
From: LUO, JEAN; MOURKOGIANNIS, CELIA NICOLE
To: SNAP INC.
Reel/Frame 066444/0777 →
Continuity (4)
Continuation 18056060 · Nov 16, 2022
Continuation 17205891 · Mar 18, 2021
Provisional Application 63003464 · Apr 1, 2020
Related Publication 20240185460A1 · Jun 6, 2024
References Cited (129)
US 7962634B2 · Cortos et al. · 2011 [cited by applicant]
US 10552477B2 · Ghatare et al. · 2020 [cited by applicant]
US 10698706B1 · Rabe et al. · 2020 [cited by applicant]
US 10755061B1 · Monroy-Hernández · 2020 [cited by applicant]
US 10832305B1 · Woodbeck et al. · 2020 [cited by applicant]
US 11055759B1 · Natesh et al. · 2021 [cited by applicant]
US 11069094B1 · Evangelista et al. · 2021 [cited by applicant]
US 11521334B2 · Luo et al. · 2022 [cited by applicant]
US 11915305B2 · Luo et al. · 2024 [cited by applicant]
US 11922661B2 · Luo et al. · 2024 [cited by applicant]
US 12118601B2 · Luo et al. · 2024 [cited by applicant]
US 12165242B2 · Luo et al. · 2024 [cited by applicant]
US 20030217892A1 · Persky · 2003 [cited by applicant]
US 20080005746A1 · Jianzhong et al. · 2008 [cited by applicant]
US 20120099756A1 · Sherman et al. · 2012 [cited by applicant]
US 20130325779A1 · Shahshahani et al. · 2013 [cited by applicant]
US 20140379729A1 · Savage et al. · 2014 [cited by applicant]
US 20150178705A1 · Vodopia · 2015 [cited by applicant]
US 20160283998A1 · Pathak · 2016 [cited by applicant]
US 20160342863A1 · Kwon et al. · 2016 [cited by applicant]
US 20170068986A1 · Chan et al. · 2017 [cited by applicant]
US 20180055736A1 · Hasegawa · 2018 [cited by applicant]
US 20180075523A1 · Sartori Odizzio · 2018 [cited by examiner]
US 20180084306A1 · Hunter · 2018 [cited by applicant]
US 20180268458A1 · Popa et al. · 2018 [cited by applicant]
US 20190014884A1 · Fu et al. · 2019 [cited by applicant]
US 20190016257A1 · Sakata et al. · 2019 [cited by applicant]
US 20190023281A1 · Fukutaka et al. · 2019 [cited by applicant]
US 20190114061A1 · Daniels et al. · 2019 [cited by applicant]
US 20190122063A1 · Kim et al. · 2019 [cited by applicant]
US 20190179405A1 · Sun et al. · 2019 [cited by applicant]
US 20190182369A1 · Gauvin et al. · 2019 [cited by applicant]
US 20190215574A1 · Saxena et al. · 2019 [cited by applicant]
US 20190266661A1 · De Sousa Moura et al. · 2019 [cited by applicant]
US 20190325498A1 · Clark · 2019 [cited by applicant]
US 20200250714A1 · Wang · 2020 [cited by applicant]
US 20200302693A1 · Singh et al. · 2020 [cited by applicant]
US 20200305579A1 · Yang et al. · 2020 [cited by applicant]
US 20200357178A1 · Lin · 2020 [cited by examiner]
US 20210015241A1 · Kuang · 2021 [cited by applicant]
US 20210224889A1 · Colby · 2021 [cited by examiner]
US 20210312523A1 · Luo et al. · 2021 [cited by applicant]
US 20210312533A1 · Luo et al. · 2021 [cited by applicant]
US 20210312672A1 · Luo et al. · 2021 [cited by applicant]
US 20210312678A1 · Luo et al. · 2021 [cited by applicant]
US 20210383460A1 · Sartori Odizzio · 2021 [cited by examiner]
US 20220101404A1 · Boscolo · 2022 [cited by examiner]
US 20220148161A1 · Guo et al. · 2022 [cited by applicant]
US 20220351416A1 · Zhang et al. · 2022 [cited by applicant]
US 20230071930A1 · Luo et al. · 2023 [cited by applicant]
US 20240161179A1 · Luo et al. · 2024 [cited by applicant]
US 20250022037A1 · Luo et al. · 2025 [cited by applicant]
CN 115668263A · 2023 [cited by applicant]
EP 3098771A1 · 2016 [cited by applicant]
JP 2009294290A · 2009 [cited by applicant]
JP 4816207B2 · 2011 [cited by applicant]
KR 20150040245A · 2015 [cited by applicant]
KR 20160142742A · 2016 [cited by applicant]
KR 20170074826A · 2017 [cited by applicant]
KR 20170090057A · 2017 [cited by applicant]
KR 20180025754A · 2018 [cited by applicant]
KR 101866859B1 · 2018 [cited by applicant]
KR 20180104051A · 2018 [cited by applicant]
KR 101993802B1 · 2019 [cited by applicant]
KR 102668172B1 · 2024 [cited by applicant]
WO WO2009120921A1 · 2009 [cited by applicant]
WO WO2010120901A1 · 2010 [cited by applicant]
WO WO2019140091A1 · 2019 [cited by applicant]
WO WO2021203118A1 · 2021 [cited by applicant]
Global Cosmetic Industry164.3: 8(1). Allured Publishing Corp., [Online] Retrieved from the internet: <Drugstore.com>, (Mar. 1999), 2 pgs. [cited by applicant]
“U.S. Appl. No. 17/205,837, Examiner Interview Summary mailed Jan. 29, 2024”, 2 pgs. [cited by applicant]
“U.S. Appl. No. 17/205,837, Final Office Action mailed Aug. 3, 2023”, 13 pgs. [cited by applicant]
“U.S. Appl. No. 17/205,837, Final Office Action mailed Dec. 8, 2022”, 33 pgs. [cited by applicant]
“U.S. Appl. No. 17/205,837, Non Final Office Action mailed Mar. 24, 2023”, 20 pgs. [cited by applicant]
“U.S. Appl. No. 17/205,837, Non Final Office Action mailed May 25, 2022”, 25 pgs. [cited by applicant]
“U.S. Appl. No. 17/205,837, Response Filed Jan. 22, 2024 to Final Office Action Mailed Aug. 3, 2023”, 18 pgs. [cited by applicant]
“U.S. Appl. No. 17/205,837, Response filed Mar. 8, 2023 to Final Office Action mailed Dec. 8, 2022”, 21 pgs. [cited by applicant]
“U.S. Appl. No. 17/205,837, Response filed Jul. 24, 2023 to Non Final Office Action mailed Mar. 24, 2023”, 13 pgs. [cited by applicant]
“U.S. Appl. No. 17/205,837, Response filed Aug. 23, 2022 to Non Final Office Action mailed May 25, 2022”, 15 pgs. [cited by applicant]
“U.S. Appl. No. 17/205,891, Non Final Office Action mailed Feb. 17, 2022”, 11 pgs. [cited by applicant]
“U.S. Appl. No. 17/205,891, Notice of Allowability mailed Oct. 31, 2022”, 2 pgs. [cited by applicant]
“U.S. Appl. No. 17/205,891, Notice of Allowance mailed Jul. 19, 2022”, 10 pgs. [cited by applicant]
“U.S. Appl. No. 17/205,891, Response filed May 17, 2022 to Non Final Office Action mailed Feb. 17, 2022”, 11 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,923, Final Office Action mailed Mar. 2, 2023”, 24 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,923, Non Final Office Action mailed Jun. 15, 2023”, 27 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,923, Non Final Office Action mailed Nov. 3, 2022”, 23 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,923, Notice of Allowance mailed Oct. 17, 2023”, 11 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,923, Response filed Feb. 3, 2023 to Non Final Office Action mailed Nov. 3, 2022”, 15 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,923, Response filed Jun. 2, 2023 to Final Office Action mailed Mar. 2, 2023”, 17 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,923, Response filed Sep. 15, 2023 to Non Final Office Action mailed Jun. 15, 2023”, 22 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,927, Final Office Action mailed Aug. 10, 2023”, 24 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,927, Final Office Action mailed Sep. 29, 2022”, 20 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,927, Non Final Office Action mailed Feb. 23, 2023”, 21 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,927, Non Final Office Action mailed Apr. 12, 2022”, 15 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,927, Response filed Feb. 9, 2024 to Final Office Action mailed Aug. 10, 2023”, 13 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,927, Response filed May 23, 2023 to Non Final Office Action mailed Feb. 23, 2023”, 14 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,927, Response filed Jul. 11, 2022 to Non Final Office Action mailed Apr. 12, 2022”, 13 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,927, Response filed Dec. 29, 2022 to Final Office Action mailed Sep. 29, 2022”, 14 pgs. [cited by applicant]
“U.S. Appl. No. 18/056,060, Corrected Notice of Allowability mailed Nov. 6, 2023”, 2 pgs. [cited by applicant]
“U.S. Appl. No. 18/056,060, Non Final Office Action mailed May 25, 2023”, 18 pgs. [cited by applicant]
“U.S. Appl. No. 18/056,060, Notice of Allowance mailed Oct. 20, 2023”, 11 pgs. [cited by applicant]
“U.S. Appl. No. 18/056,060, Response filed Aug. 25, 2023 to Non Final Office Action mailed May 25, 2023”, 10 pgs. [cited by applicant]
“eBay (EBAY) issued patent titled Identifying product metadata from an item image”, Publication: News Bites—Computing & Information [Melbourne], (Feb. 7, 2020), 7 pgs. [cited by applicant]
“International Application Serial No. PCT/US2021/070284, International Preliminary Report on Patentability mailed Oct. 13, 2022”, 11 pgs. [cited by applicant]
“International Application Serial No. PCT/US2021/070284, International Search Report mailed Aug. 23, 2021”, 5 pgs. [cited by applicant]
“International Application Serial No. PCT/US2021/070284, Invitation to Pay Additional Fees mailed Jun. 30, 2021”, 11 pgs. [cited by applicant]
“International Application Serial No. PCT/US2021/070284, Written Opinion mailed Aug. 23, 2021”, 5 pgs. [cited by applicant]
“Total Immersion Launches 1st Augmented Reality M-Commerce App”, AsiaPulse News, [Online]. Retrieved from the Internet: <URL: https://dialog.proquest.com/professional/docview/854484301?accountid=131444>, (Mar. 1, 2011),… [cited by applicant]
“U.S. Appl. No. 17/205,837, Non Final Office Action mailed Feb. 14, 2024”, 16 pgs. [cited by applicant]
“U.S. Appl. No. 17/205,837, Notice of Allowance mailed Jun. 12, 2024”, 11 pgs. [cited by applicant]
“U.S. Appl. No. 17/205,837, Response filed May 14, 2024 to Non Final Office Action mailed Feb. 14, 2024”, 21 pgs. [cited by applicant]
“U.S. Appl. No. 17/205,837, Supplemental Notice of Allowability mailed Jun. 27, 2024”, 2 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,927, Examiner Interview Summary mailed Jun. 26, 2024”, 2 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,927, Non Final Office Action mailed Mar. 20, 2024”, 24 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,927, Notice of Allowability mailed Oct. 22, 2024”, 3 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,927, Notice of Allowance mailed Jul. 31, 2024”, 8 pgs. [cited by applicant]
“U.S. Appl. No. 17/249,927, Response filed Jun. 20, 2024 to Non Final Office Action mailed Mar. 20, 2024”, 15 pgs. [cited by applicant]
“U.S. Appl. No. 18/421,653, Final Office Action mailed Feb. 20, 2025”, 22 pgs. [cited by applicant]
“U.S. Appl. No. 18/421,653, Non Final Office Action mailed Oct. 10, 2024”, 29 pgs. [cited by applicant]
“U.S. Appl. No. 18/421,653, Response filed Jan. 10, 2025 to Non Final Office Action mailed Oct. 10, 2024”, 21 pgs. [cited by applicant]
“European Application Serial No. 21718474.6, Communication Pursuant to Article 94(3) EPC mailed Jan. 7, 2025”, 5 pgs. [cited by applicant]
“European Application Serial No. 21718474.6, Response filed Feb. 5, 2025 to Communication Pursuant to Article 94(3) EPC mailed Jan. 7, 2025”, 17 pgs. [cited by applicant]
“U.S. Appl. No. 18/421,653, Response filed May 20, 2025 to Final Office Action mailed Feb. 20, 2025”, 16 pgs. [cited by applicant]
U.S. Appl. No. 17/249,923 U.S. Pat. No. 11,915,305, filed Mar. 18, 2021, Identification of Physical Products for Augmented Reality Experiences in a Messaging System. [cited by applicant]
U.S. Appl. No. 18/421,653, filed Jan. 24, 2024, Identification of Physical Products for Augmented Reality Experiences in a Messaging System. [cited by applicant]
U.S. Appl. No. 17/249,927, filed Mar. 18, 2021, Generating Augmented Reality Experiences With Physical Products Using Profile Information. [cited by applicant]
U.S. Appl. No. 17/205,891 U.S. Pat. No. 11,521,334, filed Mar. 18, 2021, Augmented Reality Experiences of Color Palettes in a Messaging System. [cited by applicant]
U.S. Appl. No. 17/205,837, filed Mar. 18, 2021, Analyzing Facial Features for Augmented Reality Experiences of Physical Products in a Messaging System. [cited by applicant]
U.S. Appl. No. 18/056,060, filed Nov. 16, 2022, Augmented Reality Experiences of Color Palettes in a Messaging System. [cited by applicant]