Method and system of retrieving multimodal assets
A system and method and for retrieving one or more one or more multimodal assets includes receiving a search query for searching for one or more multimodal assets from among a plurality of candidate multimodal assets, encoding the search query into one or more query embedding representations via a trained query representation machine-learning (ML) model, comparing, via a matching unit, the one or more query embedding representations to a plurality of multimodal tensor representations, each of the plurality of multimodal tensor representations being a representation of one of the plurality of candidate multimodal assets, and identifying, based on the comparison, at least one of the plurality of the candidate multimodal assets as a search result for the search query, and providing the at least one of the plurality of the candidate multimodal assets for display as the search result.
1. A data processing system comprising:
a processor; and
a memory in communication with the processor, the memory comprising executable instructions that, when executed by the processor, cause the data processing system to perform:
receiving, via a query representation model, a search query for searching for one or more multimodal assets from among a plurality of candidate multimodal assets, wherein the one or more multimodal assets and the search query each includes multimodal content containing two or more different types of content including graphic or image content;
parsing, via the query representation model, the search query including the multimodal content;
identifying, based on the parsing, a first content type and a second content type in the search query, the second content type being a graphic or image content type;
transmitting the first content type to a first representation model to generate a first set of vector embeddings;
transmitting the second content type to a second representation model to generate a second set of vector embeddings;
transmitting the first and second sets of vector embeddings to a tensor generation unit to generate tensors based on the first and second sets of vector embeddings and to output a query tensor representation;
comparing, via a matching unit, the query tensor representation to a plurality of multimodal tensor representations, each of the plurality of multimodal tensor representations being a representation of one of the plurality of candidate multimodal assets; and
identifying, based on the comparing, at least one of the plurality of the candidate multimodal assets as a search result for the search query; and
providing the at least one of the plurality of the candidate multimodal assets for display as the search result.
2. The data processing system of claim 1 , wherein the graphic or image content comprises at least one of an image element, an icon element, a GIF element, and an illustration element.
3. The data processing system of claim 1 , wherein the multimodal tensor representation for each one of the plurality of candidate multimodal assets includes a vector embedding representation for each of a plurality of elements in each of the plurality of candidate multimodal assets.
4. The data processing system of claim 3 , wherein the executable instructions, when executed by the processor, further cause the data processing system to perform:
providing the plurality of candidate multimodal assets to a trained asset representation machine-learning (ML) model to generate the vector embedding representation for each of the plurality of elements;
receiving the vector embedding representation for each of the plurality of elements as an output from the trained asset representation ML model;
generating the multimodal tensor representation for each one of the plurality of candidate multimodal assets asserts from the received vector embedding representations; and
generating a summarized embedding representation for the multimodal tensor representation.
5. The data processing system of claim 4 , wherein generating the summarized embedding representation for the multimodal tensor representation includes creating one aggregate embedding representation for each different modality in each one of the plurality of candidate multimodal assets.
6. The data processing system of claim 4 , wherein the executable instructions, when executed by the processor, further cause the data processing system to perform:
generating a summarized query embedding representation for the query tensor representation;
applying an Approximate Nearest Neighbor (ANN) technique to the summarized query embedding representation to generate an indexed query representation; and
applying the ANN technique to a summarized tensor representation for each one of the plurality of candidate assets,
wherein comparing the one or more query embedding representations to the plurality of multimodal tensor representations includes:
applying an ANN search technique to the summarized tensor representation for each one of the plurality of candidate assets, based on the summarized query embedding representation, to generate a smaller candidate set of multimodal assets from among the plurality of candidate multimodal assets;
performing a tensor-to-tensor comparison between the query tensor and one or more tensors associated with the smaller candidate set of multimodal assets;
generating a similarity score between the query tensor and each of the multimodal assets in the smaller candidate set of multimodal assets; and
identifying the at least one of the plurality of multimodal assets as the search result based on the similarity score.
7. The data processing system of claim 4 , wherein the summarized embedding representation for the multimodal tensor representation has a fixed size.
8. A method for retrieving one or more multimodal assets comprising:
receiving, via a query representation model, a search query for searching for the one or more multimodal assets from among a plurality of candidate multimodal assets, wherein the one or more multimodal assets and the search query each includes multimodal content containing two or more different types of content including graphic or image content;
parsing, via the query representation model, the search query including the multimodal content;
identifying, based on the parsing, a first content type and a second content type in the search query, the second content type being a graphic or image content type;
transmitting the first content type to a first representation model to generate a first set of vector embeddings;
transmitting the second content type to a second representation model to generate a second set of vector embeddings;
transmitting the first and second sets of vector embeddings to a tensor generation unit to generate tensors based on the first and second sets of vector embeddings and to output a query tensor representation;
comparing, via a matching unit, the query tensor representation to a plurality of multimodal tensor representations, each of the plurality of multimodal tensor representations being a representation of one of the plurality of candidate multimodal asset; and
identifying, based on the comparing, at least one of the plurality of the candidate multimodal assets as a search result for the search query; and
providing the at least one of the plurality of the candidate multimodal assets for display as the search result.
9. The method of claim 8 , wherein the graphic or image content comprises at least one of an image element, an icon element, a GIF element, and an illustration element.
10. The method of claim 8 , wherein a multimodal tensor representation for each one of the plurality of candidate multimodal assets includes a vector embedding representation for each of a plurality of elements in each of the plurality of candidate multimodal assets.
11. The method of claim 10 , further comprising:
providing the plurality of candidate multimodal assets to a trained asset representation machine-learning (ML) model to generate the vector embedding representation for each of the plurality of elements;
receiving the vector embedding representation for each of the plurality of elements as an output from the trained asset representation ML model;
generating the multimodal tensor representation for each one of the plurality of candidate multimodal assets from the received vector embedding representations; and
generating a summarized embedding representation for the multimodal tensor representation.
12. The method of claim 11 , wherein generating the summarized embedding representation for the multimodal tensor representation includes creating one aggregate embedding representation for each different modality in each one of the plurality of candidate multimodal assets.
13. The method of claim 11 , further comprising:
generating a summarized query embedding representation for the query tensor representation;
applying an Approximate Nearest Neighbor (ANN) technique to the summarized query embedding representation to generate an indexed query representation; and
applying the ANN technique to a summarized tensor representation for each one of the plurality of candidate assets,
wherein comparing the one or more query embedding representations to the plurality of multimodal tensor representations includes:
applying an ANN search technique to the summarized tensor representation for each one of the plurality of candidate assets, based on the summarized query embedding representation, to generate a smaller candidate set of multimodal assets from among the plurality of candidate multimodal assets;
performing a tensor-to-tensor comparison between the query tensor and one or more tensors associated with the smaller candidate set of multimodal assets;
generating a similarity score between the query tensor and each of the multimodal assets in the smaller candidate set of multimodal assets; and
identifying the at least one of the plurality of multimodal assets as the search result based on the similarity score.
14. The method of claim 11 , wherein the summarized embedding representation for the multimodal tensor representation has a fixed size.
15. A non-transitory computer readable medium on which are stored instructions that, when executed, cause a programmable device to perform:
receiving, via a query representation model, a search query for searching for one or more multimodal assets from among a plurality of candidate multimodal assets, wherein the one or more multimodal asset and the search query each includes multimodal content containing two or more different types of content including graphic or image content;
parsing, via the query representation model, the search query including the multimodal content;
identifying, based on the parsing, a first content type and a second content type in the search query, the second content type being a graphic or image content type;
transmitting the first content type to a first representation model to generate a first set of vector embeddings;
transmitting the second content type to a second representation model to generate a second set of vector embeddings;
transmitting the first and second sets of vector embeddings to a tensor generation unit to generate tensors based on the first and second sets of vector embeddings and to output a query tensor representation;
comparing, via a matching unit, the query tensor representation to a plurality of multimodal tensor representations, each of the plurality of multimodal tensor representations being a representation of one of the plurality of candidate multimodal asset; and
identifying, based on the comparing, at least one of the plurality of the candidate multimodal assets as a search result for the search query; and
providing the at least one of the plurality of the candidate multimodal assets for display as the search result.
16. The non-transitory computer readable medium of claim 15 , wherein the graphic or image content comprises at least one of an image element, an icon element, a GIF element, and an illustration element.
17. The non-transitory computer readable medium of claim 15 , wherein a multimodal tensor representation for each one of the plurality of candidate multimodal assets includes a vector embedding representation for each of a plurality of elements in each of the plurality of candidate multimodal assets.
18. The non-transitory computer readable medium of claim 17 , further comprising instructions for:
providing the plurality of candidate multimodal assets to a trained asset representation machine-learning (ML) model to generate the vector embedding representation for each of the plurality of elements;
receiving the vector embedding representation for each of the plurality of elements as an output from the trained asset representation ML model;
generating the multimodal tensor representation for each one of the plurality of candidate multimodal assets from the received vector embedding representations; and
generating a summarized embedding representation for the multimodal tensor representation.
19. The non-transitory computer readable medium of claim 18 , wherein generating the summarized embedding representation for the multimodal tensor representation includes creating one aggregate embedding representation for each different modality in each one of the plurality of candidate multimodal assets.
20. The non-transitory computer readable medium of claim 18 , further comprising:
generating a summarized query embedding representation for the query tensor representation;
applying an Approximate Nearest Neighbor (ANN) technique to the summarized query embedding representation to generate an indexed query representation; and
applying the ANN technique to a summarized tensor representation for each one of the plurality of candidate assets,
wherein comparing the one or more query embedding representations to the plurality of multimodal tensor representations includes:
applying an ANN search technique to the summarized tensor representation for each one of the plurality of candidate assets, based on the summarized query embedding representation, to generate a smaller candidate set of multimodal assets from among the plurality of candidate multimodal assets;
performing a tensor-to-tensor comparison between the query tensor and one or more tensors associated with the smaller candidate set of multimodal assets;
generating a similarity score between the query tensor and each of the multimodal assets in the smaller candidate set of multimodal assets; and
identifying the at least one of the plurality of multimodal assets as the search result based on the similarity score.
21. The non-transitory computer readable medium of claim 18 , wherein the summarized embedding representation for the multimodal tensor representation has a fixed size.