System and method for intelligent context-based personalized beauty product recommendation and matching at retail environments

1 . A system for intelligent context-based personalized beauty product recommendation and matching, comprising:

a computer system comprising a memory and a processor;

a data extraction engine, comprising a first plurality of programming instructions stored in the memory and operating on the processor, wherein the first plurality of programming instructions, when operating on the processor, causes the computer system to:

receive, via a mobile device camera, a photograph of an ingredient list for a beauty product from a customer;

apply optical character recognition (OCR) to transform the photograph of the ingredient list into a machine-readable text data format;

map customer data input and the extracted text data to product requirements, the customer input data comprising goal-based input, concern-based input, and fact-based input;

store the mapped customer input data and text data in an individual profile associated with the customer; and

a data analysis and recommendation engine, comprising a second plurality of programming instructions stored in the memory and operating on the processor, wherein the second plurality of programming instructions stored in the memory and executed by the processor, when operating on the processor, causes the computer system to:

retrieve a fact-based input from the customer input data and extract environmental data based on a locational input included in the fact-based input;

combine the mapped customer input data, the extracted text data, and the extracted environmental data into a requirement vector;

compute a similarity score between the requirement vector and one or more product vectors, the product vectors corresponding to a set of beauty products;

identify one or more products having product vectors with a computed similarity score exceeding a predetermined threshold; and

display the identified products as recommended to the customer; and

a skin analysis engine, comprising a third plurality of programming instructions stored in the memory and operating on the processor, wherein the third plurality of programming instructions, when operating on the processor, causes the computer system to:

receive a photograph of some portion of the customer's body containing visible non-facial skin;

perform body part detection on the photograph using a trained convolutional neural network classifier to identify a body part other than a face;

detect and extract secondary features from the identified body part, the secondary features comprising at least one of tattoos, body hair, makeup, or skin conditions;

detect skin and non-skin areas in the photograph after the body part has been identified and the secondary features extracted;

apply an image segmentation network comprising a convolutional neural network with a trained encoder and decoder to the detected skin areas to generate a segmentation mask identifying one or more skin concern objects, and classify skin concerns based on the identified skin concern objects;

apply a classification network comprising a convolutional neural network to the detected skin areas to generate a predicted skin-tone;

store the classified skin concerns and predicted skin-tone and skin concern in the individual profile; and

forward the classified skin concerns and skin-tone to the data analysis and recommendation engine for inclusion in the requirement vector.

2 . The system of claim 1 , wherein the product requirements include ingredients, chemical composition, functions of ingredients, allergens, ingredient free data, benefits, use, and product source information.

3 . The system of claim 1 , wherein the skin analysis engine is further configured to:

detect and extract a hair boundary from the non-skin areas of the photograph;

analyze the extracted hair boundary to generate as outputs a predicted hair-tone and a predicted hair type;

store the outputted hair-tone and hair type in the individual profile; and

forward the outputted hair-tone and hair type to the data analysis and recommendation engine.

4 . The system of claim 3 , wherein the data analysis and recommendation engine receives the outputted hair-tone and hair type and combines them into the requirement vector.

5 . The system of claim 1 , wherein the data analysis and recommendation engine receives the outputted skin-tone and skin concern and combines them into the requirement vector.

6 . The system of claim 1 , wherein the similarity is calculated using cosine similarity.

7 . The system of claim 1 , wherein the similarity is calculated using Euclidian distance.

8 . The system of claim 1 , further comprising a customer mobile device comprising a second processor, a second memory, and a software application stored in the second memory and operating on the second processor, causes the customer mobile device to:

take a photograph of the ingredient list for the beauty product;

transmit the photograph of the ingredient list for the beauty product to the data extraction engine; and

display the identified products.

9 . The system of claim 8 , wherein the customer is using the customer mobile device while currently located at a beauty or health store.

10 . The system of claim 1 , wherein the individual profile is created using customer input data, the customer input data comprising at least a fact-based input, at least a concern-based input, at least a preference-based input, and at least a goal-based input.

11 . A method for intelligent context-based personalized beauty product recommendation and matching, comprising the steps of:

receiving, via a mobile device camera, a photograph of an ingredient list for a beauty product from a customer;

applying optical character recognition (OCR) to transform the photograph of the ingredient list into a machine-readable text data format;

mapping customer data input and the extracted text data to product requirements, the customer input data comprising goal-based input, concern-based input, and fact-based input;

storing the mapped customer data input and the extracted text data in an individual profile associated with the customer; and

retrieving a fact-based input from the customer data input and extracting any environmental data based on a locational input included in the fact-based input;

combining mapped customer input data, the extracted text data and the extracted environmental data into a requirement vector;

computing a similarity score between the requirement vector and one or more product vectors, the product vectors corresponding to a set of beauty products;

identifying one or more products associated with the product vectors that have similarity scores exceeding a predetermined threshold; and

displaying the identified products as recommended to the customer; and

receiving a photograph of some portion of the customer's body containing visible non-facial skin;

performing body part detection on the photograph using a trained convolutional neural network classifier to identify a body part other than a face that contains visible skin;

detecting and extracting secondary features from the identified body part, the secondary features comprising at least one of tattoos, body hair, makeup, or skin conditions;

detecting skin and non-skin areas in the photograph after the body part has been identified and the secondary features have been extracted;

applying an image segmentation network comprising a convolutional neural network with a trained encoder and decoder to the detected skin areas, generating a segmentation mask identifying one or more skin concern objects, and classifying skin concerns based on the identified skin concern objects;

applying a classification network comprising a convolutional neural network to the detected skin areas to generate a predicted skin tone;

storing the classified skin concerns and predicted skin-tone in the individual profile; and

forwarding the classified skin concerns and skin-tone to the data analysis and recommendation engine for inclusion in the requirement vector.

12 . The method of claim 11 , wherein the product requirements include ingredients, chemical composition, functions of ingredients, allergens, ingredient free data, benefits, use, and product source information.

13 . The method of claim 11 , further comprising the steps of:

detecting and extracting a hair boundary from the non-skin areas of the photograph;

analyzing the extracted hair boundary to generate as outputs a predicted hair-tone and a predicted hair type;

storing the outputted hair-tone and hair type in the individual profile; and

forwarding the outputted hair-tone and hair type to the data analysis and recommendation engine.

14 . The method of claim 13 , wherein the data analysis and recommendation engine receives the outputted hair-tone and hair type and combines them into the requirement vector.

15 . The method of claim 11 , wherein the data analysis and recommendation engine receives the outputted skin-tone and skin concern and combines them into the requirement vector.

16 . The method of claim 11 , wherein the similarity is calculated using cosine similarity.

17 . The method of claim 11 , wherein the similarity is calculated using Euclidian distance.

18 . The method of claim 11 , further comprising the steps of:

taking a photograph of the ingredient list for the beauty product;

transmitting the photograph of the ingredient list for the beauty product to the data extraction engine; and

displaying the identified products.

19 . The method of claim 18 , wherein the customer is using the customer mobile device while currently located at a beauty or health store.

20 . The method of claim 11 , wherein the individual profile is created using customer input data, the customer input data comprising at least a fact-based input, at least a concern-based input, at least a preference-based input, and at least a goal-based input.