Image search using vectors
A server receives, from a client device, an image search query comprising multiple search components. The server maps each of the multiple search components to a component embedding vector to generate multiple component embedding vectors. The server mathematically combines the multiple component embedding vectors to generate multiple query vectors with different mathematical combinations. The server identifies, for each of the different mathematical combinations, from a vector database storing representations of images, a set of image search results based on a similarity score of a query vector of the multiple query vectors and an image embedding vector for an image of the images. The server transmits, to the client device, a portion of the set of image search results.
1 . A method for image search, the method comprising:
receiving, from a client device, an image search query comprising multiple search components;
mapping at least a subset of the multiple search components to a component embedding vector to generate multiple component embedding vectors;
combining the multiple component embedding vectors to generate multiple query vectors with different combinations, wherein combining the multiple component embedding vectors comprises mathematically combining the multiple component embedding vectors by calculating sums or weighted sums using different weights;
in response to generation of the multiple query vectors with the different combinations, selecting one or more of the multiple query vectors;
identifying one or more images associated with image embedding vectors stored in a vector database based on a comparison of the image embedding vectors with at least one of the multiple query vectors, wherein the comparison comprises a similarity score of the image embedding vectors and the at least one of the multiple query vectors; and
transmitting, to the client device, information to cause display, at the client device, of at least one of the one or more images.
2 . The method of claim 1 , wherein the comparison of the image embedding vectors with the at least one of the multiple query vectors comprises a calculation of a similarity score between the at least one of the multiple query vectors and at least one of the image embedding vectors, and wherein identifying the one or more images comprises selecting images having a similarity score exceeding a threshold value.
3 . The method of claim 1 , wherein combining the multiple component embedding vectors comprises performing a weighted addition of the multiple component embedding vectors, wherein different weights are applied to different component embedding vectors.
4 . The method of claim 1 , wherein the image search query comprises Boolean filtering criteria, the method further comprising:
determining a number of filtering search results that satisfy the Boolean filtering criteria.
5 . The method of claim 4 , further comprising:
when the number of filtering search results is below a threshold, applying the multiple search components to the filtering search results to identify the one or more images.
6 . The method of claim 4 , further comprising:
when the number of filtering search results exceeds a threshold, obtaining a set of search results based on the multiple query vectors and filtering the set of search results according to the Boolean filtering criteria to identify the one or more images.
7 . The method of claim 4 , wherein the Boolean filtering criteria comprise at least one of: a presence of a tag, an absence of a tag, a timestamp range, a geographic location being inside or outside a geographic region, or a numerical range.
8 . The method of claim 1 , wherein the multiple search components comprise a text-based criterion, and wherein mapping at least the subset of the multiple search components comprises:
mapping the text-based criterion to a first component embedding vector using a text embedding model.
9 . The method of claim 1 , wherein the multiple search components comprise a classifier-based criterion, and wherein mapping at least the subset of the multiple search components comprises:
mapping the classifier-based criterion to a first component embedding vector using an embedding-based classifier.
10 . The method of claim 1 , wherein the multiple search components comprise a similarity criterion to an input image, and wherein mapping at least the subset of the multiple search components comprises:
mapping the input image to a first component embedding vector using an image embedding model.
11 . The method of claim 1 , further comprising:
computing, for each image of the one or more images, a product of probabilities that the image is associated with each of the multiple search components; and
ranking the one or more images based on the product of probabilities to determine an order for displaying the one or more images at the client device.
12 . The method of claim 1 , wherein the vector database stores the image embedding vectors and a relational database that stores metadata associated with the one or more images, the method further comprising:
accessing both the vector database and the relational database to identify the one or more images.
13 . The method of claim 1 , further comprising:
receiving, from the client device, an indication of a selected image of the one or more images;
generating a refined set of query vectors based on an image embedding vector associated with the selected image; and
identifying additional images based on a comparison between the image embedding vectors stored in the vector database and the refined set of query vectors.
14 . A non-transitory computer-readable medium storing instructions operable to cause one or more processors to perform operations comprising:
receiving an image search query comprising multiple search components;
mapping at least a subset of the multiple search components to a component embedding vector to generate multiple component embedding vectors;
combining the multiple component embedding vectors to generate multiple query vectors with different combinations, wherein combining the multiple component embedding vectors comprises mathematically combining the multiple component embedding vectors by calculating sums or weighted sums using different weights;
in response to generation of the multiple query vectors with the different combinations, selecting one or more of the multiple query vectors;
identifying one or more images associated with image embedding vectors stored in a vector database based on a comparison of the image embedding vectors with at least one of the selected one or more of the multiple query vectors, wherein the comparison comprises a similarity score of the image embedding vectors and the at least one of the multiple query vectors; and
transmitting information to cause display of at least one of the one or more images.
15 . The non-transitory computer-readable medium of claim 14 , wherein the comparison of the image embedding vectors with the at least one of the multiple query vectors comprises a calculation of a similarity score between the at least one of the multiple query vectors and at least one of the image embedding vectors, and wherein identifying the one or more images comprises selecting images having a similarity score exceeding a threshold value.
16 . The non-transitory computer-readable medium of claim 14 , wherein combining the multiple component embedding vectors comprises performing a weighted addition of the multiple component embedding vectors, wherein different weights are applied to different component embedding vectors.
17 . The non-transitory computer-readable medium of claim 14 , wherein the image search query comprises Boolean filtering criteria, the operations further comprising:
determining a number of filtering search results that satisfy the Boolean filtering criteria.
18 . A system comprising:
a memory subsystem storing instructions; and processing circuitry configured to execute the instructions to perform operations comprising:
receiving, from a client device, a search query comprising multiple search components;
mapping at least a subset of the multiple search components to a component embedding vector to generate multiple component embedding vectors;
combining the multiple component embedding vectors to generate multiple query vectors with different combinations, wherein combining the multiple component embedding vectors comprises mathematically combining the multiple component embedding vectors by calculating sums or weighted sums using different weights;
in response to generation of the multiple query vectors with the different combinations, selecting one or more of the multiple query vectors;
identifying one or more entries associated with entry embedding vectors stored in a vector database based on a comparison of the entry embedding vectors with at least one of the multiple query vectors, wherein the comparison comprises a similarity score of the entry embedding vectors and the at least one of the multiple query vectors; and
transmitting, to the client device, information to cause display, at the client device, of at least one of the one or more entries.
19 . The system of claim 18 , wherein the one or more entries comprise one or more images, and wherein the entry embedding vectors comprise image embedding vectors.
20 . The system of claim 18 , wherein the multiple search components comprise a classifier-based criterion, and wherein mapping at least the subset of the multiple search components comprises:
mapping the classifier-based criterion to a first component embedding vector using an embedding-based classifier.