Systems and methods for deep learning model based product matching using multi modal data
Methods and systems for generating a list of products each matching a reference product are disclosed. A user query is first received, and multi-modal attribute data for the reference product are determined, with each data mode being a type of product characterization having a modality selected from a text data class, categorical data, a pre-compared engineered feature, audio, image, and video. Next, a first list of candidate products is determined based on a product match signature, and a second list of candidate products is generated from the first, wherein for at least one given candidate product, a deep learning multi-modal matching model is selected to determine whether a match is found. Lastly, the second list is filtered to remove outliers and to generate the list of matching products. Also disclosed are benefits of the new methods and systems, and alternative embodiments of the implementation.
1. A method for generating a list of matching products, wherein each matching product matches a reference product, comprising:
receiving a user query for the reference product;
determining reference attribute data for the reference product, wherein the reference attribute data is multimodal having at least a first data mode and a second data mode different from the first data mode, wherein each data mode is a type of characterization of the reference product, and wherein each data mode has a modality selected from the group consisting of a text data class, categorical data, a pre-compared engineered feature, audio data, image data, and video data;
determining a first list of candidate products, based on at least one reference product match signature generated from the reference attribute data by a deep learning product signature model;
generating a second list of candidate products from the first list of candidate products, wherein at least one given candidate product in the first list of candidate products has multimodal attribute data with at least the first data mode and the second data mode, and wherein for the given candidate product,
selecting a deep learning multimodal matching model from a plurality of candidate multimodal matching models, based on a reference category associated with the reference product and a first attribute data item for the given candidate product, wherein each candidate multimodal matching model has been previously trained on a category-specific training dataset, wherein the selected deep learning multimodal matching model comprises at least a first data analyzer for processing data having the first data mode and a second data analyzer for processing data having the second data mode, wherein the first data analyzer comprises a first deep learning neural network (DLNN), and wherein the second data analyzer comprises a second DLNN,
matching the given candidate product to the reference product using the selected deep learning multimodal matching model to generate a match score, wherein the match score is computed based on the reference attribute data for the reference product and attribute data for the given candidate product,
determining whether the match score meets a given threshold, and
in response to determining that the match score meets the given threshold, adding the given candidate product to the second list of candidate products; and
generating the list of matching products by filtering the second list of candidate products to remove outlier candidate products.
2. The method of claim 1 , wherein the first attribute data item for the given candidate product is selected from the group consisting of product type, vendor, brand, product category, and target consumer group.
3. The method of claim 2 , wherein the first attribute data item is a product type attribute having a value selected from the group consisting of new, used, refurbished, renewed, open-box, generic, rental, offline sale, local resale, and auction.
4. The method of claim 1 , wherein the selecting of the deep learning multimodal matching model is further based on a user attribute extracted from the user query.
5. The method of claim 4 , wherein the user attribute is selected from the group consisting of a location, a gender, and an age.
6. The method of claim 1 , wherein the user query comprises at least one reference attribute data item for the reference product, the at least one reference attribute data item selected from the group consisting of a search keyword, an image, an audio clip, a video clip, a web address, a barcode, a stock keeping unit (SKU) code, a Universal Product Code (UPC), an International Standard Book Number (ISBN), an Amazon Standard Identification Number (ASIN), an European Article Number (EAN) and a Manufacturer's Part Number (MPN).
7. The method of claim 1 , wherein the determining the reference attribute data for the reference product comprises:
extracting at least one identifier for the reference product from the user query;
determining whether the reference product exists in an internal database by searching the internal database with the at least one identifier for the reference product;
in response to determining that the reference product exists in the internal database, retrieving the reference attribute data for the reference product from the internal database; and
in response to determining that the reference product does not exist in the internal database, performing a search in at least one external database for the reference product to identify the reference attribute data for the reference product, and adding the reference product to the internal database by adding the reference attribute data to the internal database.
8. The method of claim 1 , wherein the text data class is selected from the group consisting of a title text class, a product description text class, a product review text class, a social media post text class, and a ranking label text class.
9. The method of claim 1 ,
wherein the reference attribute data comprises at least a first reference subset of attribute data having the first data mode and a second reference subset of attribute data having the second data mode,
wherein the attribute data for the given candidate product comprises at least a first comparison subset of attribute data having the first data mode, and a second comparison subset of attribute data having the second data mode, and
wherein the matching the given candidate product to the reference product using the selected deep learning multimodal matching model to generate the match score comprises:
determining, for the reference product, a first reference signature from the first reference subset of attribute data using the first DLNN in the first data analyzer, and a second reference signature from the second reference subset of attribute data using the second DLNN in the second data analyzer; and
determining, for the given candidate product, a first comparison signature from the first comparison subset of attribute data using the first DLNN in the first data analyzer, and a second comparison signature from the second comparison subset of attribute data using the second DLNN in the second data analyzer,
wherein the match score is computed based on the first reference signature, the second reference signature, the first comparison signature, and the second comparison signature.
10. The method of claim 1 ,
wherein at least one candidate product in the first list of candidate products has a second engineered feature attribute data item that has been pre-compared to the reference product.
11. The method of claim 1 , wherein the match score is a matching probability.
12. The method of claim 1 , wherein the filtering the second list of candidate products comprises:
clustering candidate products in the second list of candidate products based on at least a second attribute data item;
determining whether an outlier cluster comprising the outlier candidate products exists; and
in response to determining that the outlier cluster exists, removing the outlier candidate products in the outlier cluster from the second list of candidate products.
13. The method of claim 12 , wherein the second attribute data item is a price attribute.
14. The method of claim 1 , further comprising:
receiving a user selection of one target product in the list of matching products; and
providing to the user a link to an online purchase page for the target product.
15. The method of claim 1 , wherein the reference product match signature is generated from at least one reference attribute data item selected from the group consisting of a keyword, an image, and a categorical variable associated with the reference product.
16. The method of claim 15 , wherein the reference product match signature is generated further from the user query.
17. A system for generating a list of matching products, wherein each matching product matches a reference product, comprising:
one or more processors; and
one or more non-transitory, computer-readable storage media for storing program code accessible by the one or more processors, the program code when executed by the one or more processors causes the one or more processors to:
receive a user query for the reference product;
determine reference attribute data for the reference product, wherein the reference attribute data is multimodal having at least a first data mode and a second data mode different from the first data mode, wherein each data mode is a type of characterization of the reference product, and wherein each data mode has a modality selected from the group consisting of a text data class, categorical data, a pre-compared engineered feature, audio data, image data, and video data;
determine a first list of candidate products, based on at least one reference product match signature generated from the reference attribute data by a deep learning product signature model;
generate a second list of candidate products from the first list of candidate products, wherein at least one given candidate product in the first list of candidate products has multimodal attribute data with at least the first data mode and the second data mode, and wherein for the given candidate product,
select a deep learning multimodal matching model from a plurality of candidate multimodal matching models, based on a reference category associated with the reference product and a first attribute data item for the given candidate product, wherein each candidate multimodal matching model has been previously trained on a category-specific training dataset, wherein the selected deep learning multimodal matching model comprises at least a first data analyzer for processing data having the first data mode and a second data analyzer for processing data having the second data mode wherein the first data analyzer comprises a first deep learning neural network (DLNN), and wherein the second data analyzer comprises a second DLNN,
match the given candidate product to the reference product using the selected deep learning multimodal matching model to generate a match score, wherein the match score is computed based on the reference attribute data for the reference product and attribute data for the given candidate product,
determine whether the match score meets a given threshold, and
in response to determining that the match score meets the given threshold, add the given candidate product to the second list of candidate products; and
generate the list of matching products by filtering the second list of candidate products to remove outlier candidate products.
18. The system of claim 17 , wherein the first attribute data item for the given candidate product is selected from the group consisting of product type, vendor, brand, product category, and target consumer group.
19. A non-transitory, computer-readable storage medium for generating a list of matching products, wherein each matching product matches a reference product, the non-transitory, computer-readable storage medium storing program code accessible by one or more processors, the program code when executed by the one or more processors causes the one or more processors to:
receive a user query for the reference product;
determine reference attribute data for the reference product, wherein the reference attribute data is multimodal having at least a first data mode and a second data mode different from the first data mode, wherein each data mode is a type of characterization of the reference product, and wherein each data mode has a modality selected from the group consisting of a text data class, categorical data, a pre-compared engineered feature, audio data, image data, and video data;
determine a first list of candidate products, based on at least one reference product match signature generated from the reference attribute data by a deep learning product signature model;
generate a second list of candidate products from the first list of candidate products, wherein at least one given candidate product in the first list of candidate products has multimodal attribute data with at least the first data mode and the second data mode, and wherein for the given candidate product,
select a deep learning multimodal matching model from a plurality of candidate multimodal matching models, based on a reference category associated with the reference product and a first attribute data item for the given candidate product, wherein each candidate multimodal matching model has been previously trained on a category-specific training dataset, wherein the selected deep learning multimodal matching model comprises at least a first data analyzer for processing data having the first data mode and a second data analyzer for processing data having the second data mode, wherein the first data analyzer comprises a first deep learning neural network (DLNN), and wherein the second data analyzer comprises a second DLNN,
match the given candidate product to the reference product using the selected deep learning multimodal matching model to generate a match score, wherein the match score is computed based on the reference attribute data for the reference product and attribute data for the given candidate product,
determine whether the match score meets a given threshold, and
in response to determining that the match score meets the given threshold, add the given candidate product to the second list of candidate products; and
generate the list of matching products by filtering the second list of candidate products to remove outlier candidate products.
20. The non-transitory, computer-readable storage medium of claim 19 , wherein the first attribute data item for the given candidate product is selected from the group consisting of product type, vendor, brand, product category, and target consumer group.