System and method for product search by embedding visual representation into text sequences

1. A computer-implemented method for searching a product corresponding to a query from a customer, comprising:

embedding the query to obtain a query embedding;

retrieving product information comprising a product text and a product image;

embedding the product text to obtain a product text embedding, embedding the product image to obtain a product image embedding, and combining the product text embedding and the product image embedding to obtain a product embedding, wherein the product image embedding has a same format as the product text embedding,

wherein embedding the product image comprises: normalizing the product image to obtain a normalized product image; splitting the normalized product image into a plurality of grids: concatenating the plurality of grids into a sequence of grids; and extracting product image feature from the sequence of grids to obtain a grid token for each of the sequence of grids,

wherein embedding the product image further comprises: adding a position vector to each of the grid tokens, the position vectors representing positions of the grids in the sequence of grids; adding a segment vector to each of the grid tokens, the segment vectors representing identification of the product image: adding a mask vector to each of the grid tokens, wherein values of the mask vectors are 0 or 1, and value of one of the grid tokens is converted to 0 when the value of corresponding one of the mask vectors is 0; and defining a category identification, the category identification representing category of the product in an e-commerce platform, wherein the product embedding comprises the grid tokens, the position vectors, the segment vectors, the mask vectors, and the category identification;

subjecting the query embedding and the product embedding to a transformer to determine whether the query and the product are relevant; and

providing the product as a search result of the query when the query and the product are relevant.

2. The method of claim 1 , wherein the transformer comprises a query transformer for processing the query embedding and a product transformer for processing the product embedding.

3. The method of claim 1 , wherein the query comprises a query text and a query image, and the query embedding comprises a query text embedding corresponding to the query text and a query image embedding corresponding to the query image.

4. The method of claim 1 , wherein about 10-15% of the mask vectors have a value of 0.

5. The method of claim 1 , wherein the step of extracting the product image feature from the sequence of grids comprises performing a convolutional neural network (CNN) on each of the grids, and the transformer is a pre-trained bidirectional encoder representations from transformers (BERT).

6. A system for searching a product corresponding to a query from a customer, wherein the system comprises a computing device, the computing device comprises a processor and a storage device storing computer executable code, and the computer executable code, when executed at the processor, is configured to:

embed the query to obtain a query embedding;

retrieve product information comprising a product text and a product image;

embed the product text to obtain a product text embedding, embed the product image to obtain a product image embedding, and combine the product text embedding and the product image embedding to obtain a product embedding, wherein the product image embedding has a same format as the product text embedding,

wherein the computer executable code is configured to embed the product image by: normalizing the product image to obtain a normalized product image; splitting the normalized product image into a plurality of grids; concatenating the plurality of grids into a sequence of grids; and extracting product image feature from the sequence of grids to obtain a grid token for each of the sequence of grids,

wherein the computer executable code is further configured to embed the product image by: adding a position vector to each of the grid tokens, the position vectors representing positions of the grids in the sequence of grids; adding a segment vector to each of the grid tokens, the segment vectors representing identification of the product image; adding a mask vector to each of the grid tokens, wherein values of the mask vectors are 0 or 1, and value of one of the grid tokens is converted to 0 when the value of corresponding one of the mask vectors is 0; and defining a category identification, the category identification representing category of the product in an e-commerce platform, wherein the product embedding comprises the grid tokens, the position vectors, the segment vectors, the mask vectors, and the category identification;

subject the query embedding and the product embedding to a transformer to determine whether the query and the product are relevant; and

provide the product as a search result of the query when the query and the product are relevant.

7. The system of claim 6 , wherein the transformer comprises a query transformer for processing the query embedding and a product transformer for processing the product embedding.

8. The system of claim 6 , wherein the query comprises a query text and a query image, and the query embedding comprises a query text embedding corresponding to the query text and a query image embedding corresponding to the query image.

9. The system of claim 6 , wherein about 10-15% of the mask vectors have a value of 0.

10. The system of claim 6 , wherein the computer executable code is configured extract the product image feature from the sequence of grids by performing a convolutional neural network (CNN) on each of the grids, and the transformer is a pre-trained bidirectional encoder representations from transformers (BERT).

11. A non-transitory computer readable medium storing computer executable code, wherein the computer executable code, when executed at a processor of a computing device, is configured to:

embed the query to obtain a query embedding;

retrieve product information comprising a product text and a product image;

embed the product text to obtain a product text embedding, embed the product image to obtain a product image embedding, and combine the product text embedding and the product image embedding to obtain a product embedding, wherein the product image embedding has a same format as the product text embedding,

wherein the computer executable code is configured to embed the product image by: normalizing the product image to obtain a normalized product image; splitting the normalized product image into a plurality of grids; concatenating the plurality of grids into a sequence of grids; and extracting product image feature from the sequence of grids to obtain a grid token for each of the sequence of grids,

wherein the computer executable code is further configured to embed the product image by: adding a position vector to each of the grid tokens, the position vectors representing positions of the grids in the sequence of grids; adding a segment vector to each of the grid tokens, the segment vectors representing identification of the product image; adding a mask vector to each of the grid tokens, wherein values of the mask vectors are 0 or 1, and value of one of the grid tokens is converted to 0 when the value of corresponding one of the mask vectors is 0; and defining a category identification, the category identification representing category of the product in an e-commerce platform, wherein the product embedding comprises the grid tokens, the position vectors, the segment vectors, the mask vectors, and the category identification;

subject the query embedding and the product embedding to a transformer to determine whether the query and the product are relevant; and

provide the product as a search result of the query when the query and the product are relevant.