Unified framework for multi-modal similarity search
Technology is disclosed herein for enhanced similarity search. In an implementation, a search environment includes one or more computing hardware, software, and/or firmware components in support of enhanced similarity search. The one or more components identify a modality for a similarity search with respect to a query object. The components generate an embedding for the query object based on the modality and based on connections between the query object and neighboring nodes in a graph. The embedding for the query object provides the basis for the search for similar objects.
1. One or more computer readable storage media having program instructions stored therein that, when executed by a processing system, direct a computing apparatus to at least:
determine, selected from a plurality of modalities, a modality for a similarity search to be performed with respect to a query object;
identify, based on a nearest neighbor search of objects represented in a graph and based on the selected modality for the similarity search, one or more connections between the query object and a subset of the objects in the graph;
submit input to a machine learning model, wherein the input comprises a vector embedding of the query object and the one or more connections between the query object and the subset of the objects;
obtain output from the machine learning model, wherein the output comprises an initial vector embedding for the query object;
obtain a tag vector embedding for at least one tag associated with the query object;
combine the initial vector embedding and the tag vector embedding, to produce a combined vector embedding, based on a weighting biased towards the selected modality; and
supply input to the similarity search comprising the combined vector embedding for the query object, to obtain results comprising one or more similar objects represented in the graph.
2. The one or more computer readable storage media of claim 1 wherein:
when the selected modality comprises a conceptual modality, the weighting emphasizes the tag vector embedding in the combining of the initial vector embedding with the tag vector embedding, more so than when the selected modality comprises a visual modality; and
when the selected modality comprises the visual modality, the weighting emphasizes the tag vector embedding in the combining of the initial vector embedding with the tag vector embedding, less so than when the selected modality comprises the conceptual modality.
3. The one or more computer readable storage media of claim 1 wherein:
the query object comprises a query image;
the objects comprise existing images represented by nodes in the graph; and
the one or more similar objects represented in the graph comprise nearest neighbors of the query image in the existing images.
4. The one or more computer readable storage media of claim 3 , wherein the program instructions, when executed by the processing system, further direct the computing apparatus to perform the similarity search, for the nearest neighbors of the query image in the existing images, based on the combined vector embedding.
5. The one or more computer readable storage media of claim 1 , wherein the graph comprises:
the existing nodes, wherein existing nodes comprise object nodes corresponding to the objects and tag nodes corresponding to tags; and
existing edges representative of existing connections, wherein the existing connections comprise object-object connections between similar pairs of the objects, and object-tag connections between the tags and at least some of the objects.
6. The one or more computer readable storage media of claim 1 , wherein the program instructions, when executed by the processing system, further direct the computing apparatus to:
identify a subset of the objects most like the query object; and
identify one or more tags associated with the subset of the objects; and
include, in the one or more connections, one or more of new object-object connections between the query object and the subset of the objects, and one or more new object-tag connections between the query object and the one or more tags.
7. The one or more computer readable storage media of claim 6 , wherein:
when the selected modality comprises a visual modality, the one or more connections include the new object-object connections and exclude the one or more new object-tag connections; and
when the selected modality comprises a conceptual modality, the one or more connections include one or more of the new object-object connections and the one or more of the new object-tag connections.
8. The one or more computer readable storage media of claim 1 , wherein the program instructions, when executed by the processing system, further direct the computing apparatus to identify the selected modality based on user input comprising a selection of the selected modality from a set of possible modalities comprising a visual modality and a conceptual modality.
9. A method comprising:
determining a modality, selected from a plurality of modalities, for a similarity search to be performed with respect to a query object;
identifying, based on a nearest neighbor search of objects represented in a graph and based on the selected modality determined for the similarity search, one or more connections between the query object and a subset of the objects in the graph;
submitting input to a machine learning model, wherein the input comprises a vector embedding of the query object and the one or more connections between the query object and the subset of the objects;
obtaining output from the machine learning model, wherein the output comprises an initial vector embedding for the query object;
obtaining a tag vector embedding for at least one tag associated with the query object;
combining the initial vector embedding and the tag vector embedding, to produce a combined vector embedding, based on a weighting biased towards the selected modality; and
supplying input to the similarity search comprising the combined vector embedding for the query object, to obtain results comprising one or more similar objects represented in the graph.
10. The method of claim 9 , wherein the selected modality comprises a conceptual modality, and the weighting emphasizes the tag vector embedding in the combining of the initial vector embedding with the tag vector embedding.
11. The method of claim 9 , wherein the selected modality comprises a visual modality, and the weighting emphasizes the tag vector embedding in the combining of the initial vector embedding with the tag vector embedding.
12. The method of claim 9 , wherein:
the query object comprises a query image;
the objects comprise existing images represented by nodes in the graph; and
the one or more similar objects represented in the graph comprise nearest neighbors of the query image in the existing images.
13. The method of claim 12 , further comprising performing the similarity search, for the nearest neighbors of the query image in the existing images, based on the combined vector embedding.
14. The method of claim 12 , wherein the graph comprises:
the existing nodes, wherein existing nodes comprise object nodes corresponding to the objects and tag nodes corresponding to tags; and
existing edges representative of existing connections, wherein the existing connections comprise object-object connections between similar pairs of the objects, and object-tag connections between the tags and at least some of the objects.
15. The method of claim 12 , further comprising:
identifying a subset of the objects most like the query object; and
identifying one or more tags associated with the subset of the objects; and
including, in the one or more connections, one or more of new object-object connections between the query object and the subset of the objects, and one or more new object-tag connections between the query object and the one or more tags.
16. The method of claim 15 , wherein the selected modality comprises a visual modality, and the one or more connections include the new object-object connections and exclude the one or more new object-tag connections.
17. The method of claim 15 , wherein the selected modality comprises a conceptual modality, and the one or more connections include one or more of the new object-object connections and the one or more of the new object-tag connections.
18. The method of claim 9 , further comprising identifying the selected modality based on user input comprising a selection of the selected modality from a set of possible modalities comprising a visual modality and a conceptual modality.
19. A computing system comprising:
one or more computer readable storage media having program instructions stored therein that, when executed by a processing system, direct a computing apparatus to at least:
determine a modality, selected from a plurality of modalities, for a similarity search to be performed with respect to a query object;
identify, based on a nearest neighbor search of objects represented in a graph and based on the selected modality for the similarity search, one or more connections between the query object and a subset of the objects in the graph;
submit input to a machine learning model, wherein the input comprises a vector embedding of the query object and the one or more connections between the query object and the subset of the objects;
obtain output from the machine learning model, wherein the output comprises an initial vector embedding for the query object;
obtain a tag vector embedding for at least one tag associated with the query object;
combine the initial vector embedding and the tag vector embedding, to produce a combined vector embedding, based on a weighting biased towards the selected modality; and
supply input to the similarity search comprising the combined vector embedding for the query object, to obtain results comprising one or more similar objects represented in the graph.
20. The computing system of claim 19 , wherein:
the query object comprises a query image;
the objects comprise existing images represented by nodes in the graph; and
the one or more similar objects represented in the graph comprise nearest neighbors of the query image in the existing images.