IP Library › Granted Patent US 12,446,822
Granted Patent B2
US 12,446,822 · App. 17/931,109 · Granted Oct 21, 2025

Diverse cosmetic and skin care product matching system

Inventors: Prasad Tendulkar (Foster City, CA); Nelly Mensah (San Francisco, CA); Anna Teresa Nicholas (Hazelton, PA); Jillian Bertone (Lafayette, CA); Lisa Zhu (Belmont, CA); Jennifer Stokes (Seattle, WA); Witono Liujanto (San Ramon, CA); Aaron Oelckers (Pleasant Hill, CA); Marc McCotter (Hana, HI); Harsha Huddar (Livermore, CA); Fiona Kavanagh (San Francisco, CA); Amit Adiraju Narasimha (Santa Clara, CA); Naroju Janardhan (San Ramon, CA)
Assignee: Sephora USA, Inc.
A61B5/441A45D44/005A61B5/0064A61B5/442G06Q30/0621G06Q30/0631G06Q30/0633G06T7/0012G06T7/90G06V10/26G06V10/56G06V10/764G06V10/774G06V10/776G06V10/82G06V40/10A45D2044/007G06Q30/01G06Q30/0641G06T2200/24G06T2207/20081G06T2207/30088G06T2207/30201
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,446,822
App. No.
17/931,109
Granted
Oct 21, 2025
Kind
B2
Abstract

Various techniques pertain to a system having multiple mobile computing devices that includes a unified front end computing device and a mobile computing device of a client, a first artificial intelligence model operatively coupled to the multiple mobile computing devices and predicting a predicted body characteristic of a part of a body of the client for providing body care to the client, a second artificial intelligence model operatively coupled to the multiple mobile computing devices and predicting a list of products or services for the body care at least by executing a recommendation service on the list of products or services, and a data lake that is integrated with multiple application programming interfaces across multiple computing nodes and stores multiple types of data for predicting the list of products or services by the first artificial intelligence model and for predicting the personalized recommendation by the second artificial intelligence model.

Claims (105)

1. A method for predicting a matching product and recommendation for a body characteristic of a client, comprising:

receiving, at a computing system having a user interface, an input pertaining to a body complexion characteristic of a client;

predicting, by a first artificial intelligence model, a body complexion index at least by:

executing an extraction task on at least the input to obtain an extraction result; and

executing a classification task on the extract result;

predicting, by at least a second artificial intelligence model, a list of products or services at least by executing a recommendation service based at least in part upon the body complexion index for the client;

displaying, in the user interface of the computing system, information pertaining to the list of products or services; and

receiving, at the user interface, an adjustment instruction that alters a value of a first attribute of a plurality of attributes of the body complexion index into an adjusted value, wherein

the body complexion index comprises a body tone color index or value for a subject area of a body of the client, and

the plurality of attributes comprises a first attribute pertaining to lightness of the body complexion characteristic of the client, a second attribute pertaining to saturation of the body complexion characteristic of the client, and a third attribute pertaining to an undertone of the body complexion characteristic of the client.

2. The method of claim 1 further comprising

incorporating information pertaining to the value and the adjusted value to a training dataset, and

training the first artificial intelligence model using at least the training dataset.

3. The method of claim 1 wherein the input pertaining to the body complexion characteristic of the client comprises the body tone color index or value for the subject area of the body of the client.

4. The method of claim 3 wherein the body tone color index or value for the subject area comprises a plurality of color attribute values in a L*A*B* color space of a Lch color space.

5. The method of claim 1 wherein receiving the input comprises

presenting preparation instructions in the user interface for preparing for a body scan session, wherein the preparation instructions comprise a first preparation instruction to prepare a scanning device for the body scan session,

executing the body scan session for determining the body complexion index for the client at least by transmitting a signal to a scanning device operatively coupled to the computing system, and

capturing, by the scanning device, body complexion characteristic data for the subject area on the body of the client.

6. The method of claim 1 comprising

training the first artificial intelligence model using multiple datasets, wherein training the first artificial intelligence model comprises

partitioning a plurality of first datasets for the first artificial intelligence model into a first training dataset, a first testing dataset, and a first validation dataset,

partitioning a plurality of second datasets for the second artificial intelligence model into a second training dataset, a second testing dataset, and a second validation dataset, wherein the plurality of first datasets or the plurality of second datasets comprises a synthetic example,

respectively training the first artificial intelligence model and the second artificial intelligence model with the first training dataset and the second training dataset,

respectively testing the first artificial intelligence model and the second artificial intelligence model with the first testing dataset and the second testing dataset, and

respectively validating the first artificial intelligence model and the second artificial intelligence model with the first validation dataset and the second validation dataset.

7. A method for predicting a matching product and recommendation for a body characteristic of a client, comprising:

receiving, at a computing system having a user interface, an input pertaining to a body complexion characteristic of a client;

predicting, by a first artificial intelligence model, a body complexion index at least by:

executing an extraction task on at least the input to obtain an extraction result; and

executing a classification task on the extract result;

predicting, by at least a second artificial intelligence model, a list of products or services at least by executing a recommendation service based at least in part upon the body complexion index for the client; and

displaying, in the user interface of the computing system, information pertaining to the list of products or services,

wherein predicting the list of products or services comprises:

predicting, by at least the second artificial intelligence model with a first kernel, a first set of products or services from a plurality of products or services at least by applying the first kernel of the second artificial intelligence model to the plurality of products or services based at least in part upon the input, wherein the first kernel comprises a first plurality of filtering values arranged in a first array.

8. The method of claim 7 wherein predicting the list of products or services further comprises

predicting, by at least the recommendation service with a second kernel, a second set of products or services from the first set of products or services at least by

identifying one or more attributes that are specific to the client, and

applying the second kernel to the first set of products or services based at least in part upon the one or more attributes specific to the client, wherein the second kernel comprises a second plurality of filtering values arranged in a second array, and

generating the list of products or services at least by identifying one or more products or services from the second set into the list of products or services.

9. The method of claim 8 further comprising

generating a personalized recommendation for the client based at least in part upon the list of products or services,

wherein the one or more attributes specific to the client include one or more concerns, one or more body complexion characteristic types, one or more preferences pertaining to the body complexion characteristic, a change in the body complexion characteristic after applying a product or service, or a combination thereof.

10. A system for predicting products and recommendations for body care, comprising:

a plurality of mobile computing devices that comprises a unified front end computing device and a mobile computing device of a client;

a first artificial intelligence model that is operatively coupled to the plurality of mobile computing devices and predicts a predicted body characteristic of a part of a body of the client for providing body care to the client;

a second artificial intelligence model that is operatively coupled to the plurality of mobile computing devices and generates a predicted list of products or services for the body care at least by executing a recommendation service on the predicted list of products or services; and

a data lake that is integrated with a plurality of application programming interfaces across multiple computing nodes and stores multiple types of data for predicting the predicted list of products or services by the first artificial intelligence model and for predicting a personalized recommendation by the second artificial intelligence model.

11. The system of claim 10 further comprising

a product catalog service that is operatively coupled to the plurality of mobile computing devices, the first artificial intelligence model, and the second artificial intelligence model.

12. The system of claim 11 wherein

the product catalog service operatively coupled to the first artificial intelligence model receives first inputs from the plurality of mobile computing devices or a product catalog database, and

the product catalog service operatively coupled to the first artificial intelligence model generates the predicted body characteristic of the part of the body of the client based at least in part on the first inputs received from the plurality of mobile computing devices or the product catalog database.

13. The system of claim 11 wherein

the product catalog service is operatively coupled to the second artificial intelligence model to receive second inputs from the plurality of mobile computing devices or a product catalog database, and

the product catalog service is operatively coupled to the second artificial intelligence model to generate the predicted list of products or services for the body care based at least in part on the second inputs received from the plurality of mobile computing devices or the product catalog database.

14. The system of claim 10 further comprising

a business control center database,

a mapping service that is operatively coupled to the business control center database to correlate product information with body care information into correlated product and body care data at least by performing a mapping between the product information of a plurality of body care products in the business control center database and the body care information, wherein the body care information comprises color space data of a L*A*B* color space or a Lch color space, and

an enterprise service bus that comprises a correlation database or correlation knowledgebase and stores the correlated product and body care data generated by the mapping service.

15. The system of claim 10 further comprising

a product ingestion processor that distributes inputs received from a correlation database or correlation knowledgebase of an enterprise service bus among a distributed processing system that comprises a plurality of stream processing nodes, and

a product or user experience management model operatively coupled to the product ingestion processor.

16. The system of claim 15 wherein

the inputs comprise correlated product and body care data that is generated by correlating product information of a plurality of products with body care information,

the body care information comprises color space data of a L*A*B* color space or a Lch color space, and

the plurality of stream processing nodes respectively processes corresponding streams of the inputs into one or more catalog files.

17. The system of claim 16 wherein

the product ingestion processor is operatively coupled to the product or user experience management model to partition the inputs into the corresponding streams of the inputs, and

the product ingestion processor is operatively coupled to the product or user experience management model to respectively assign the corresponding streams of the inputs to the plurality of stream processing nodes.

18. The system of claim 17 further comprising a product catalog database or knowledgebase,

wherein a catalog file of the one or more catalog files is stored in the product catalog database or knowledgebase,

the catalog file of the one or more catalog files comprises the product information of a particular product in the plurality of products, a specific piece of the body care information, and a link between the product information of the particular product and the specific piece of the body care information, and

the specific piece of the body care information comprises a color index or value of a plurality of color indices or values in the L*A*B* color space or the Lch color space.

19. The system of claim 15 wherein

the product or user experience management model is operatively coupled to the product ingestion processor to transform correlation data in the inputs received from the correlation database or the correlation knowledgebase into transformed correlation data at least by associating specific data with the correlation data, and

the specific data pertains to one or more characteristics of the client or experience of the client with a product or service in the predicted list of products or services.

20. A method for predicting a matching product and recommendation for a body characteristic of a client, comprising:

receiving, at a computing system having a user interface, an input pertaining to a body complexion characteristic of a client;

predicting, by a first artificial intelligence model, a body complexion index at least by:

executing an extraction task on at least the input to obtain an extraction result; and

executing a classification task on the extract result;

predicting, by at least a second artificial intelligence model, a list of products or services at least by executing a recommendation service based at least in part upon the body complexion index for the client; and

displaying, in the user interface of the computing system, information pertaining to the list of products or services,

wherein predicting the list of products or services further comprises:

predicting, by at least the recommendation service with a first kernel, a first set of products or services from a second set of products or services at least by:

identifying one or more attributes that are specific to the client; and

applying the first kernel to the second set of products or services based at least in part upon the one or more attributes specific to the client, wherein the first kernel comprises a first plurality of filtering values arranged in a first array; and

generating the list of products or services at least by identifying one or more products or services from the first set into the list of products or services.

21. The method of claim 20 further comprising:

generating a personalized recommendation for the client based at least in part upon the list of products or services,

wherein the one or more attributes specific to the client include one or more concerns, one or more body complexion characteristic types, one or more preferences pertaining to a body complexion characteristic, a change in the body complexion characteristic after applying a product or service, or a combination thereof.

22. The method of claim 20 wherein the input pertaining to a body complexion characteristic of the client comprises a body tone color index or value for a subject area of a body of the client.

23. The method of claim 22 wherein the body tone color index or value for the subject area comprises a plurality of color attribute values in a L*A*B* color space of a Lch color space.

24. The method of claim 20 wherein receiving the input comprises

presenting preparation instructions in the user interface for preparing for a body scan session, wherein the preparation instructions comprise a first preparation instruction to prepare a scanning device for the body scan session,

executing the body scan session for determining the body complexion index for the client at least by transmitting a signal to the scanning device operatively coupled to the computing device, and

capturing, by the scanning device, body complexion characteristic data for a subject area on a body of the client.

25. The method of claim 20 comprising:

training the first artificial intelligence model using multiple datasets, wherein training the first artificial intelligence model comprises

partitioning a plurality of first datasets for the first artificial intelligence model into a first training dataset, a first testing dataset, and a first validation dataset,

partitioning a plurality of second datasets for the second artificial intelligence model into a second training dataset, a second testing dataset, and a second validation dataset, wherein the plurality of first datasets or the plurality of second datasets comprises a synthetic example,

respectively training the first artificial intelligence model and the second artificial intelligence model with the first training dataset and the second training dataset,

respectively testing the first artificial intelligence model and the second artificial intelligence model with the first testing dataset and the second testing dataset, and

respectively validating the first artificial intelligence model and the second artificial intelligence model with the first validation dataset and the second validation dataset.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Oct 3, 2025
From: TENDULKAR, PRASAD; MENSAH, NELLY; BERTONE, JILLIAN; ZHU, LISA; STOKES, JENNIFER; LIUJANTO, WITONO; OELCKERS, AARON; MCCOTTER, MARC; HUDDAR, HARSHA; KAVANAGH, FIONA; NARASIMHA, AMIT ADIRAJU; JANARDHAN, NAROJU; NICHOLAS, ANNA TERESA
To: SEPHORA USA, INC.
Reel/Frame 072468/0703 →
Continuity (2)
Provisional Application 63261060 · Sep 9, 2021
Related Publication 20230177587A1 · Jun 8, 2023
References Cited (81)
US 4648718A · Sadamitsu et al. · 1987 [cited by applicant]
US 4989079A · Ito · 1991 [cited by applicant]
US 5483339A · Van Aken et al. · 1996 [cited by applicant]
US 6437866B1 · Flynn · 2002 [cited by applicant]
US 6522977B2 · Corrigan et al. · 2003 [cited by applicant]
US 6603550B1 · Flynn et al. · 2003 [cited by applicant]
US 7180524B1 · Axelrod · 2007 [cited by applicant]
US 7738032B2 · Kollias et al. · 2010 [cited by applicant]
US 8182091B2 · Foster · 2012 [cited by applicant]
US 8564778B1 · Igarashi · 2013 [cited by applicant]
US 9019296B1 · Abraham · 2015 [cited by applicant]
US 9064279B1 · Tuan et al. · 2015 [cited by applicant]
US 9519927B1 · Tuan et al. · 2016 [cited by applicant]
US 10045820B2 · Youngquist et al. · 2018 [cited by applicant]
US 10387937B1 · Tuan et al. · 2019 [cited by applicant]
US 20020036778A1 · Wagner et al. · 2002 [cited by applicant]
US 20020059248A1 · Farchione · 2002 [cited by applicant]
US 20020062263A1 · Katou · 2002 [cited by applicant]
US 20020065452A1 · Bazin et al. · 2002 [cited by applicant]
US 20030058261A1 · Challa et al. · 2003 [cited by applicant]
US 20030065578A1 · Peyrelevade et al. · 2003 [cited by applicant]
US 20030065636A1 · Peyrelevade · 2003 [cited by applicant]
US 20030149504A1 · Iwaki et al. · 2003 [cited by applicant]
US 20030151611A1 · Turpin et al. · 2003 [cited by applicant]
US 20030235544A1 · Kishida et al. · 2003 [cited by applicant]
US 20040010333A1 · Baker · 2004 [cited by applicant]
US 20050146531A1 · Rice et al. · 2005 [cited by applicant]
US 20060022994A1 · Hussie · 2006 [cited by applicant]
US 20060066629A1 · Norlander et al. · 2006 [cited by applicant]
US 20060095297A1 · Virik · 2006 [cited by applicant]
US 20060178904A1 · Aghassian et al. · 2006 [cited by applicant]
US 20060179045A1 · Grinsfelder et al. · 2006 [cited by applicant]
US 20060253176A1 · Caruso · 2006 [cited by applicant]
US 20070058860A1 · Harville et al. · 2007 [cited by applicant]
US 20070071314A1 · Bhatti et al. · 2007 [cited by applicant]
US 20080003547A1 · Woolfe et al. · 2008 [cited by applicant]
US 20080046410A1 · Lieb · 2008 [cited by applicant]
US 20080079843A1 · Pote et al. · 2008 [cited by applicant]
US 20080194928A1 · Bandic et al. · 2008 [cited by applicant]
US 20080281728A1 · Gomez · 2008 [cited by applicant]
US 20080311061A1 · Heuer · 2008 [cited by applicant]
US 20100020095A1 · Reynolds et al. · 2010 [cited by applicant]
US 20100183528A1 · Maloney et al. · 2010 [cited by applicant]
US 20100185064A1 · Bandic et al. · 2010 [cited by applicant]
US 20100322878A1 · Stella et al. · 2010 [cited by applicant]
US 20110093361A1 · Morales · 2011 [cited by applicant]
US 20110106650A1 · Stone · 2011 [cited by applicant]
US 20110184837A1 · Biro · 2011 [cited by applicant]
US 20110246284A1 · Chaikin et al. · 2011 [cited by applicant]
US 20120045121A1 · Youngman et al. · 2012 [cited by applicant]
US 20120096380A1 · Wagner · 2012 [cited by applicant]
US 20120123759A1 · Cho et al. · 2012 [cited by applicant]
US 20120203668A1 · Howard et al. · 2012 [cited by applicant]
US 20120253224A1 · Mir et al. · 2012 [cited by applicant]
US 20120269743A1 · Smith et al. · 2012 [cited by applicant]
US 20130092805A1 · Funk et al. · 2013 [cited by applicant]
US 20130117137A1 · Klein et al. · 2013 [cited by applicant]
US 20130177137A1 · Eichhorn et al. · 2013 [cited by applicant]
US 20130250322A1 · Kawabata et al. · 2013 [cited by applicant]
US 20130300761A1 · Ahmed · 2013 [cited by applicant]
US 20140323873A1 · Cummins et al. · 2014 [cited by applicant]
US 20150356661A1 · Rousay · 2015 [cited by applicant]
US 20190038211A1 · Rattner · 2019 [cited by applicant]
US 20190125249A1 · Rattner et al. · 2019 [cited by applicant]
US 20190164210A1 · Kornilov · 2019 [cited by examiner]
US 20190318505A1 · LaSalle et al. · 2019 [cited by applicant]
US 20200226657A1 · Ji et al. · 2020 [cited by applicant]
US 20210015240A1 · Elfakhri · 2021 [cited by examiner]
US 20220138972A1 · Rattner et al. · 2022 [cited by applicant]
US 20230169749A1 · Zhou · 2023 [cited by examiner]
CN 111400606A · 2020 [cited by examiner]
WO 2012065037A1 · 2012 [cited by applicant]
WO 2015126988A1 · 2015 [cited by applicant]
WO 2017181293A1 · 2017 [cited by applicant]
English translation of CN-111400606-A. (Year: 2020). [cited by examiner]
Alexander, Antoinette, “Virtual Mirror Kiosks Make Beauty Shopping ‘EZ’,” 2009, Drug Store News, Feb. 9, 2009; vol. 31, No. 2, 66 pages. [cited by applicant]
Weisstein, Eric W., “Subset,” Mathworld, retrieved from http://mathworld.wolfram.com/Subset.html, dated May 16, 2013, retrieved from the Internet Archive on Jul. 12, 2015, 1 page. [cited by applicant]
Hastie, Trevor, et al., “The Elements of Statistical Learning Data Mining, Inference, and Prediction,” Springer Series in Statistics, 2nd ed., Chapters 14.3-14.6 and 16, Jan. 13, 2017, retrieved from https://hastie.su.d… [cited by applicant]
Goodfellow, Ian, et al., “Deep Learning,” MIT Press book, 2016, Part I, Chapters 7 and 10, Part III, Chapters 14 and 20, retrieved from https://www.deeplearningbook.org/ 196 pages. [cited by applicant]
PCT International Search Report and the Written Opinion of the International Searching Authority for International Application PCT/US2022/043155 dated Mar. 26, 2023, 14 pgs. [cited by applicant]
Molnar, Christoph, “Interpretable Machine Learning—A Guide for Making Black Box Models Explainable,” Aug. 26, 2021, Chapters 4.1 and 5, retrieved from https://web.archive.org/web/20210828033303/https://christophm.github… [cited by applicant]