Complementary item recommendations based on multi-modal embeddings
Systems and methods for providing suggestions of complementary products responsive to an anchor product are disclosed. The method includes receiving a selection of an anchor product. A similarity score between text embeddings of the anchor product and text embeddings of a plurality of products in a product database is calculated. A similarity score between an image feature of the anchor product and an image feature of the plurality of products in the product database is calculated. A weighted score between the two similarity scores as calculated for the anchor product and the plurality of products in the product database is calculated. At least one of the products from the product database having a highest weighted score is selected and returned responsive to the selection of the anchor product.
1. A system, comprising:
a server; and
a product recommendation system,
wherein the product recommendation system is in electronic communication with the server,
wherein the product recommendation system comprises:
a processor; and
a memory storing instructions that, when executed by the processor, cause the system to perform a method, comprising:
receiving, by the product recommendation system from a user device, a selection of an anchor product;
training, by the processor, a machine learning model with product information for a plurality of products in a product database;
generating, from the machine learning model, text embeddings comprising vector descriptions of text associated with each product of the plurality of products in the product database for each product of the plurality of products, wherein the machine learning model comprises a multi-stream network;
calculating, by the product recommendation system, a similarity score between text embeddings of the anchor product and text embeddings of a plurality of products in a product database;
calculating, by the product recommendation system, a similarity score between an image feature of the anchor product and an image feature of the plurality of products in the product database,
wherein the image features comprise vector descriptions of color content of the images;
calculating, by the product recommendation system, a weighted score between the two similarity scores as calculated for the anchor product and the plurality of products in the product database, wherein a weight of the similarity score between the image feature of the plurality of products in the product database and a weight of the similarity score between text embeddings of the anchor product and text embeddings of the plurality of products in the product database is varied using a product type determined from the text embeddings of the anchor product;
selecting, by the product recommendation system, at least one of the products from the product database having a highest weighted score; and
returning, to the user device by the product recommendation system, the at least one of the products as selected responsive to the selection of the anchor product, automatically causing the user device to display on a display of the user device the at least one of the products as selected responsive to the selection of the anchor product.
2. The system of claim 1 , wherein the weighted score includes a text weight value based on a product type and an image weight value based on the product type.
3. The system of claim 2 , comprising a first product type and a second product type, the first product type having a greater importance of visual features and a lower importance of textual features, the second product type having a lower importance of visual features and a greater importance of textual features.
4. The system of claim 3 , wherein the text weight value of the first product type is lower than the text weight value of the second product type, and the image weight value of the first product type is greater than the image weight value of the second product type.
5. The system of claim 1 , wherein the image feature includes a red-green-blue (RGB) color histogram on the image.
6. The system of claim 5 , comprising determining, by the product recommendation system, the RGB color histogram for a foreground of the anchor product and the plurality of products in the product database.
7. The system of claim 5 , wherein the RGB channels of the image include 8 bins per channel to obtain a 512-dimensional feature vector for the anchor product and the plurality of products in the product database.
8. The system of claim 1 , wherein the selecting at least one of the products from the product database having a highest weighted score includes selecting a plurality of products and returning at least one of the plurality of products having a different product type than a product type of the anchor product.
9. A method, comprising:
receiving, by a product recommendation system from a user device, a selection of an anchor product;
training, by the product recommendation system, a machine learning model with product information for a plurality of products in a product database;
generating, from the machine learning model, text embeddings comprising vector descriptions of text associated with each product of the plurality of products in the product database for each product of the plurality of products, wherein the machine learning model comprises a multi-stream network;
calculating, by the product recommendation system, a similarity score between text embeddings of the anchor product and text embeddings of a plurality of products in a product database,
wherein the text embeddings comprise vector descriptions of text associated with each product of the plurality of products in the product database,
wherein the text embeddings for each product of the plurality of products are generated from a machine learning model, wherein the machine learning model is trained based on product information for each of the plurality of products in the product database for each of the plurality of products;
calculating, by the product recommendation system, a similarity score between an image feature of the anchor product and an image feature of the plurality of products in the product database;
calculating, by the product recommendation system, a weighted score between the two similarity scores as calculated for the anchor product and the plurality of products in the product database,
wherein a weight of the similarity score between the image feature of the plurality of products in the product database and a weight of the similarity score between text embeddings of the anchor product and text embeddings of the plurality of products in the product database is varied using a product type determined from the text embeddings of the anchor product;
selecting, by the product recommendation system, at least one of the products from the product database having a highest weighted score; and
returning, to the user device by the product recommendation system, the at least one of the products as selected responsive to the selection of the anchor product, automatically causing the user device to display on a display of the user device the at least one of the products as selected responsive to the selection of the anchor product.
10. The method of claim 9 , wherein calculating the similarity score between text embeddings of the anchor product and text embeddings of the plurality of products in a database includes calculating a cosine similarity score between the text embeddings of the anchor product and the text embeddings of the plurality of products in the product database.
11. The method of claim 9 , wherein calculating the similarity score between the image feature of the anchor product and the image feature of the plurality of products in the product database includes calculating a cosine similarity score between the image features of the anchor product and the image features of the plurality of products in the product database.
12. The method of claim 9 , comprising separating the background and foreground of the image.
13. The method of claim 12 , wherein the separating comprises a mean adaptive threshold.
14. The method of claim 12 , wherein the image feature includes a red-green-blue (RGB) color histogram on the image, the method comprising determining the RGB color histogram for the foreground of the image following the separating the background and the foreground of the image.
15. The method of claim 14 , wherein the RGB channels of the image include 8 bins per channel to obtain a 512-dimensional feature vector as the image feature for each image.
16. The method of claim 9 , comprising filtering the of the products as selected responsive to selection of the anchor product to be a different product type than the product type of the anchor product.
17. A method, comprising:
receiving, by a product recommendation system from a user device, a selection of an anchor product;
generating, from a machine learning model, text embeddings comprising vector descriptions of text associated with each product of a plurality of products in a product database for each product of the plurality of products, wherein the machine learning model is trained based on product information for each of the plurality of products in the product database for each of the plurality of products, wherein the machine learning model comprises a multi-stream network;
calculating, by the product recommendation system, a cosine similarity score between text embeddings of the anchor product and text embeddings of the plurality of products in the product database;
calculating, by the product recommendation system, a cosine similarity score between an image feature of the anchor product and an image feature of a first subset of the plurality of products in the product database;
calculating, by the product recommendation system, a combined score between the two cosine similarity scores as calculated for the anchor product and the first subset of the plurality of products in the product database,
wherein a weight of the cosine similarity score between the image feature of the anchor product and the image feature of the first subset of the plurality of products in the product database and a weight of the cosine similarity score between text embeddings of the anchor product and text embeddings of the anchor product and text embeddings of the plurality of products in the product database is varied using a product type determined from the text embeddings of the anchor product;
selecting, by the product recommendation system, at least one of the products from the product database having a highest combined score; and
returning, to the user device by the product recommendation system, the at least one of the products as selected responsive to the selection of the anchor product, automatically causing the user device to display on a display of the user device the at least one of the products as selected responsive to the selection of the anchor product.
18. The method of claim 17 , wherein the first subset of the plurality of products includes a selection of the plurality of products having a highest cosine similarity score between the text embeddings of the anchor product and the plurality of products.
19. The method of claim 17 , wherein the combined score is weighted and includes a text weight value based on the product type and an image weight value based on the product type.
20. The method of claim 17 , comprising filtering the products as selected responsive to selection of the anchor product to be a different product type than a product type of the anchor product.