Multimodal analysis for content item semantic retrieval and identification
A content retrieval system may receive a query associated with a plurality of content items in a repository. For each content item of the plurality of content items: a respective first and second similarity score may be generated based on a similarity between embeddings indicative of a first and second data type generated from the query and for the content item; and a respective normalized similarity score may be generated based on a combination of the respective first and second similarity scores. A set of content items with respective normalized similarity scores that satisfy a similarity score threshold may be identified. An exact-match (lexical) search may yield respective mapping scores for content items that may also be ranked. An output indicative of content items that are identified in the set of content items with high-ranking similarity scores and identified in the set of content items with high-ranking mapping scores.
1 . A computer-implemented method, comprising:
performing, with a first predictive model, a first vector search on an image data type portion of a multimodal query, input to a user interface of a media device providing a system for searching media content items, to generate a first similarity score indicating a similarity between the image data type portion and a content item, and concurrently performing, with a second predictive model, a second vector search on a text data type portion of the multimodal query to generate a second similarity score indicating a similarity between the text data type portion and the content item, wherein the first predictive model and the second predictive model are machine-learning based classifiers that are trained independently of each other, wherein the first predictive model is trained to identify a similarity between an embedding indicative of the image data type portion and an embedding indicative of data of an image data type generated for the content item, wherein the image data type portion is of the image data type, and wherein the second predictive model is trained to identify a similarity between an embedding indicative of the text data type portion and an embedding indicative of data of a text data type generated for the content item, wherein the text data type portion is of the text data type;
normalizing the first similarity score and the second similarity score;
combining the normalized first similarity score and the normalized second similarity score into a combined normalized similarity score;
generating, based on an amount of tokenized keywords from the multimodal query mapped to tokenized keywords from a description the content item, a mapping score for the content item; and
outputting an indication of the content item to the user interface based on the combined normalized similarity score satisfying a similarity score threshold and the mapping score satisfying a mapping score threshold.
2 . The computer-implemented method of claim 1 , wherein the normalizing the first similarity score and the second similarity score comprises:
transforming, based on a normal distribution of the first similarity score among each content item of a plurality of content items, the first similarity score to a common scale value; and
transforming, based on a normal distribution of the second similarity score among each content item of the plurality of content items, the second similarity score to another common scale value; and
wherein the combining into the combined normalized similarity score comprises outputting, based on a combination of the common scale value and the another common scale value, the combined normalized similarity score.
3 . The computer-implemented method of claim 1 , wherein the outputting the indication of the content item comprises causing the media device from which the multimodal query is received to display the indication of the content item.
4 . The computer-implemented method of claim 3 , wherein the indication of the content item is arranged according to a user preference associated with the media device.
5 . A system, comprising:
one or more memories; and
at least one processor each coupled to at least one of the memories and configured to perform operations comprising:
performing, with a first predictive model, a first vector search on an image data type portion of a multimodal query, input to a user interface of a media device providing a system for searching media content items, to generate a first similarity score indicating a similarity between the image data type portion and a content item, and concurrently performing, with a second predictive model, a second vector search on a text data type portion of the multimodal query to generate a second similarity score indicating a similarity between the text data type portion and the content item, wherein the first predictive model and the second predictive model are machine-learning based classifiers that are trained independently of each other, wherein the first predictive model is trained to identify a similarity between an embedding indicative of the image data type portion and an embedding indicative of data of an image data type generated for the content item, wherein the image data type portion is of the image data type, and wherein the second predictive model is trained to identify a similarity between an embedding indicative of the text data type portion and an embedding indicative of data of a text data type generated for the content item, wherein the text data type portion is of the text data type;
normalizing the first similarity score and the second similarity score;
combining the normalized first similarity score and the normalized second similarity score into a combined normalized similarity score;
generating, based on an amount of tokenized keywords from the multimodal query mapped to tokenized keywords from a description the content item, a mapping score for the content item; and
outputting an indication of the content item to the user interface based on the combined normalized similarity score satisfying a similarity score threshold and the mapping score satisfying a mapping score threshold.
6 . The system of claim 5 , wherein the normalizing the first similarity score and the second similarity score comprises:
transforming, based on a normal distribution of the first similarity score among each content item of a plurality of content items, the first similarity score to a common scale value; and
transforming, based on a normal distribution of the second similarity score among each content item of the plurality of content items, the second similarity score to another common scale value; and
wherein the combining normalized similarity score comprises outputting, based on a combination of the common scale value and the another common scale value, the combined normalized similarity score.
7 . The system of claim 5 , wherein the outputting the indication of the content item comprises causing the media device from which the multimodal query is received to display the indication of the content item.
8 . The system of claim 7 , wherein the indication of the content item is arranged according to a user preference associated with the media device.
9 . A non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device, cause the at least one computing device to perform operations comprising:
performing, with a first predictive model, a first vector search on an image data type portion of a multimodal query, input to a user interface of a media device providing a system for searching media content items, to generate a first similarity score indicating a similarity between the image data type portion and a content item, and concurrently performing, with a second predictive model, a second vector search on a text data type portion of the multimodal query to generate a second similarity score indicating a similarity between the text data type portion and the content item, wherein the first predictive model and the second predictive model are machine-learning based classifiers that are trained independently of each other, wherein the first predictive model is trained to identify a similarity between an embedding indicative of the image data type portion and an embedding indicative of data of an image data type generated for the content item, wherein the image data type portion is of the image data type, and wherein the second predictive model is trained to identify a similarity between an embedding indicative of the text data type portion and an embedding indicative of data of a text data type generated for the content item, wherein the text data type portion is of the text data type;
normalizing the first similarity score and the second similarity score;
combining the normalized first similarity score and the normalized second similarity score into a combined normalized similarity score;
generating, based on an amount of tokenized keywords from the multimodal query mapped to tokenized keywords from a description the content item, a mapping score for the content item; and
outputting an indication of the content item to the user interface based on the combined normalized similarity score satisfying a similarity score threshold and the mapping score satisfying a mapping score threshold.
10 . The non-transitory computer-readable medium of claim 9 , wherein the normalizing the first similarity score and the second similarity score comprises:
transforming, based on a normal distribution of the first similarity score among each content item of a plurality of content items, the first similarity score to a common scale value; and
transforming, based on a normal distribution of the second similarity score among each content item of the plurality of content items, the second similarity score to another common scale value; and
wherein the combining into the combined normalized similarity score comprises outputting, based on a combination of the common scale value and the another common scale value, the combined normalized similarity score.
11 . The non-transitory computer-readable medium of claim 9 , wherein the outputting the indication of the content item comprises causing the media device from which the multimodal query is received to display the indication of the content item.
12 . The non-transitory computer-readable medium of claim 11 , wherein the indication of the content item is arranged according to a user preference associated with the media device.