Multi-modal product embedding generator
Described are systems and methods for providing a multi-tasked trained machine learning model that may be configured to generate product embeddings from multiple types of product information. The exemplary product embeddings may be generated for a corpus of products (e.g., products included in a product catalog, etc.) based on both image information and text information associated with each respective product. Accordingly, the generated product embeddings may be compatible with learned representations of the different types of product information (e.g., image information, text information, etc.) and may be used to create a product index, which can be used to determine and serve product recommendations in connection with multiple different recommendation services that may be configured to receive different types of inputs (e.g., a single image, multiple images, text-based information, etc.).
1 . A computer-implemented method, comprising:
accessing a plurality of training data including a plurality of query and engaged product pairs, wherein the plurality of query and engaged product pairs includes queries of different modalities including a plurality of image queries, and a plurality of text queries, and a plurality of engaged products associated with a plurality of engagement types;
training a first machine learning model using the plurality of training data to configure the first machine learning model to generate, based at least in part on at least one of image information or text information associated with a product, a product embedding representative of the product according to multiple task objectives, the product embedding being compatible with multiple modalities such that a distance measure between the product embedding and respective embeddings of different modalities represents a relevance of the product embedding to the respective embeddings;
generating an index based at least in part on a plurality of product embeddings generated by the first trained machine learning model, wherein the plurality of product embeddings corresponds to a first plurality of products, and wherein the plurality of product embeddings are compatible with different query modalities;
receiving a query that includes a representation of at least one of an image query, a text query, or a product query; and
determining, based at least in part on an embedding of the representation and the index, a second plurality of products from the first plurality of products as recommended products, the determining comprising calculating a distance measure between the embedding of the representation and one or more product embeddings in the index.
2 . The computer-implemented method of claim 1 , wherein the plurality of product embeddings are compatible with a first learned representation of an image and a second learned representation of a text string.
3 . The computer-implemented method of claim 2 , wherein the first learned representation of the image and the second learned representation of the text string are incompatible.
4 . The computer-implemented method of claim 1 , wherein one or more product embeddings of the plurality of product embeddings are generated based at least in part on a plurality of image information associated with a respective product of the first plurality of products and a plurality of text information associated with the respective product of the first plurality of products.
5 . The computer-implemented method of claim 1 , wherein determining the second plurality of products includes performing a nearest neighbor technique.
6 . A computing system, comprising:
one or more processors; and
a memory storing program instructions that, when executed by the one or more processors, cause the one or more processors to at least:
receive a plurality of product information associated with a plurality of products, wherein one or more products of the plurality of products is associated with product information that includes image product information and text product information;
determine, using a trained product embedding generator and based at least in part on the image product information and text product information associated with one or more products of the plurality of products, a respective product embedding for one or more products of the plurality of products, the respective product embeddings being compatible with multiple modalities such that a distance measure between a product embedding and respective embeddings of different modalities represents a relevance of the product embedding to the respective embeddings;
generate, based at least in part on the respective product embeddings for the plurality of products, a product index;
receive a request for recommended products from one of a plurality of recommendation services, the plurality of recommendation services including two or more recommendation services configured to receive respective inputs of different modalities, wherein the request includes one of a plurality of modalities including: an image query, a text query, or a product query;
determine, based at least in part on an embedding of the image query or an embedding of the text query and the product index, a plurality of recommended products, the determining comprising calculating a distance measure between the embedding of the image query or the text query and one or more product embeddings in the index; and
provide at least a portion of the plurality of recommended products as a response to the request.
7 . The computing system of claim 6 , wherein the image product information includes a plurality of image embeddings corresponding to a plurality of images associated with the product and the text product information includes a plurality of text strings associated with the product.
8 . The computing system of claim 7 , wherein the plurality of text strings are processed using a hash embedding technique.
9 . The computing system of claim 6 , wherein:
the trained product embedding generator is trained using positive engagements associated with a plurality of engagement types; and
the plurality of engagement types includes at least one of:
clicking a first product;
saving a second product;
adding a third product to a cart; or
purchasing a fourth product.
10 . The computing system of claim 6 , wherein:
the product index supports a plurality of recommendation services; and
at least one of the plurality of recommendation services determines the plurality of recommended products.
11 . The computing system of claim 6 , wherein respective product embeddings are compatible with a first learned representation of an image and a second learned representation of a text string, such that a distance between a first product embedding and the first learned representation and the second learned representation represents a relevance of the first learned representation and the second learned representation to the first product embedding.
12 . The computing system of claim 6 , wherein the product index includes a hierarchical navigable small worlds (HNSW) graph.
13 . The computing system of claim 6 , wherein determining the plurality of recommended products includes performing a nearest neighbor search technique.
14 . The computing system of claim 6 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to at least:
process at least a portion of the respective product embeddings using a ranking service to determine a ranking associated with the plurality of recommended products; and
cause at least a portion of the plurality of recommended products to be presented to a user in accordance with the ranking.
15 . The computing system of claim 6 , wherein the instructions, when executed by the one or more processors, further cause the one or more processors to at least process at least a portion of the respective product embeddings using a trained classifier to determine a further product information associated with at least one of the plurality of products.
16 . One or more non-transitory computer readable storage media having instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
accessing a plurality of training data including a plurality of query and engaged product pairs, wherein the plurality of query and engaged product pairs includes queries of different modalities including a plurality of image queries, and a plurality of text queries, and a plurality of engaged products associated with a plurality of engagement types;
training a first machine learning model using the plurality of training data to configure the first machine learning model to generate, based at least in part on at least one of image information or text information associated with a product, a product embedding representative of the product according to multiple task objectives, the product embedding being compatible with multiple modalities such that a distance measure between the product embedding and respective embeddings of different modalities represents a relevance of the product embedding to the respective embeddings;
generating an index based at least in part on a plurality of product embeddings generated by the first trained machine learning model, wherein the plurality of product embeddings corresponds to a first plurality of products, and wherein the plurality of product embeddings are compatible with different query modalities;
receiving a query that includes a representation of at least one of an image query, a text query, or a product query; and
determining, based at least in part on an embedding of the representation and the index, a second plurality of products from the first plurality of products as recommended products, the determining comprising calculating a distance measure between the embedding of the representation and one or more product embeddings in the index.
17 . The one or more non-transitory computer readable storage media of claim 16 , wherein the plurality of product embeddings are compatible with a first learned representation of an image and a second learned representation of a text string.
18 . The one or more non-transitory computer readable storage media of claim 17 , wherein the first learned representation of the image and the second learned representation of the text string are incompatible.
19 . The one or more non-transitory computer readable storage media of claim 16 , wherein one or more product embeddings of the plurality of product embeddings are generated based at least in part on a plurality of image information associated with a respective product of the first plurality of products and a plurality of text information associated with the respective product of the first plurality of products.
20 . The one or more non-transitory computer readable storage media of claim 16 , wherein determining the second plurality of products includes performing a nearest neighbor technique.