Semantic image retrieval
View Patent ↗Computer-implemented techniques for sematic image retrieval are disclosed. Digital images are classified into N number of categories based on their visual content. The classification provides a set of N-dimensional image vectors for the digital images. Each image vector contains up to N number of probability values for up to N number of corresponding categories. An N-dimensional image match vector is generated that projects an input keyword query into the vector space of the set of image vectors by computing the vector similarities between a word vector for the input query and a word vector for each of the N number of categories. Vector similarities between the image match vectors and the set of image vectors can be computed to determine images semantically relevant to the input query.
1. A computer-implemented method for semantic image search, the method comprising:
obtaining a word vector representing an input keyword unit, the word vector generated according to an embedding function that projects keyword units in a vector space as N-dimensional vectors and in which distances between vectors in the vector space correspond to semantic similarities between the keyword units;
obtaining a plurality of word vectors representing a plurality of image content class keyword units; wherein each image content class keyword unit, of the plurality of image content class keyword units, describes a respective image content class of a plurality of image content classes; wherein each word vector, of the plurality of words vectors, is generated according to the embedding function;
generating an image match vector comprising a plurality of vector similarity measurements, each vector similarity measurement of the plurality of vector similarity measurements based on a computed vector similarity between (a) the word vector representing the input keyword unit and (b) a respective word vector of the plurality of word vectors;
computing a vector similarity measurement between the image match vector and a particular image vector, the particular image vector being for a particular digital image, the particular image vector for the particular digital image comprising a plurality of probability values for the plurality of image content classes, each probability value of the plurality of probability values reflecting a probability that the particular digital image belongs to a respective image content class of the plurality of image content classes; and
identifying the particular digital image as relevant to the input keyword unit based on the vector similarity measurement between the image match vector and the particular image vector.
2. The method of claim 1 , further comprising:
generating the image match vector comprising the plurality of vector similarity measurements, each vector similarity measurement of the plurality of vector similarity measurements based on a computed cosine similarity between (a) the word vector representing the input keyword unit and (b) the respective word vector of the plurality of word vectors representing the plurality of image content class keyword units.
3. The method of claim 1 , further comprising;
computing a cosine similarity between the image match vector and the particular image vector; and
identifying the particular digital image as relevant to the input keyword unit based on the cosine similarity between the image match vector and the particular image vector.
4. The method of claim 1 , wherein a particular image content class of the plurality of image content classes is associated with two or more image content class keyword units of the plurality of image content class keyword units; and wherein the method further comprises:
for each image content class keyword unit of the two or more image content class keyword units, computing a vector similarity between the word vector representing the input keyword unit and the respective word vector of the image content class keyword unit; and
selecting, for inclusion in the image match vector as a representative vector similarity measure for the particular image content class, the vector similarity, of two or more vector similarities computed for the two or more image content class keyword units, representing a greatest semantic similarity, among the two or more image content class keyword units, with the input keyword unit.
5. The method of claim 1 , further comprising:
based on the vector similarity measurement between the image match vector and the particular image vector, causing a graphical user interface to be presented at a client computing device, the graphical user interface indicating that the particular digital image is relevant to the input keyword unit.
6. The method of claim 1 , further comprising:
computing a vector similarity measurement between the image match vector and each of a plurality of image vectors, each image vector of the plurality of image vectors being for a respective digital image, the image vector for the respective digital image comprising a plurality of probability values for the plurality of image content classes, each probability value of the plurality of probability values reflecting a probability that the respective digital image belongs to a respective image content class of the plurality of image content classes; and
identifying a plurality of digital images as relevant to the input keyword unit based on the vector similarity measurements between the image match vector and the plurality of image vectors.
7. The method of claim 1 , further comprising:
selecting a particular image content class, of the plurality of image content classes, corresponding to a particular element of the image match vector; and
retrieving the particular image vector from an inverted index using an identifier of the particular image content class as a key into the inverted index.
8. One or more non-transitory computer-readable media comprising:
one or more computer programs having instructions configured for:
obtaining a word vector representing an input keyword unit, the word vector generated according to a first embedding function that projects keyword units in a vector space as N-dimensional vectors and in which distances between vectors in the vector space correspond to semantic similarities between the keyword units;
obtaining a plurality of word vectors representing a plurality of image content class keyword units, each image content class keyword unit describing a respective image content class of a plurality of image content classes, each word vector of the plurality of words vectors generated according to a second embedding function that projects keyword units in a vector space as N-dimensional vectors and in which distances between vectors in the vector space correspond to semantic similarities between the keyword units;
generating an image match vector comprising a plurality of vector similarity measurements, each vector similarity measurement of the plurality of vector similarity measurements based on a computed vector similarity between (a) the word vector representing the input keyword unit and (b) a respective word vector of the plurality of word vectors;
computing a vector similarity measurement between the image match vector and a particular image vector, the particular image vector being for a particular digital image, the particular image vector for the particular digital image comprising a plurality of probability values for the plurality of image content classes, each probability value of the plurality of probability values reflecting a probability that the particular digital image belongs to a respective image content class of the plurality of image content classes; and
identifying the particular digital image as relevant to the input keyword unit based on the vector similarity measurement between the image match vector and the particular image vector.
9. The one or more non-transitory computer-readable media of claim 8 , further comprising:
the one or more computer programs having instructions configured for:
generating the image match vector comprising the plurality of vector similarity measurements, each vector similarity measurement of the plurality of vector similarity measurements based on a computed dot product of (a) the word vector representing the input keyword unit and (b) the respective word vector of the plurality of word vectors.
10. The one or more non-transitory computer-readable media of claim 8 , further comprising:
the one or more computer programs having instructions configured for:
computing a dot product of the image match vector and the particular image vector; and
identifying the particular digital image as relevant to the input keyword unit based on the dot product of the image match vector and the particular image vector.
11. The one or more non-transitory computer-readable media of claim 8 , wherein a particular image content class of the plurality of image content classes is associated with two or more image content class keyword units of the plurality of image content class keyword units; and wherein the one or more non-transitory computer-readable media further comprise:
the one or more computer programs having instructions configured for:
for each image content class keyword unit of the two or more image content class keyword units, computing a vector similarity between the word vector representing the input keyword unit and the respective word vector of the image content class keyword unit; and
selecting, for inclusion in the image match vector as a representative vector similarity measure for the particular image content class, the vector similarity, of two or more vector similarities computed for the two or more image content class keyword units, representing a greatest semantic similarity, among the two or more image content class keyword units, with the input keyword unit.
12. The one or more non-transitory computer-readable media of claim 8 , further comprising:
the one or more computer programs having instructions configured for:
based on the vector similarity measurement between the image match vector and the particular image vector, causing a graphical user interface to be presented at a client computing device, the graphical user interface indicating that the particular digital image is relevant to the input keyword unit.
13. The one or more non-transitory computer-readable media of claim 8 , further comprising:
the one or more computer programs having instructions configured for:
computing a vector similarity measurement between the image match vector and each of a plurality of image vectors, each image vector of the plurality of image vectors being for a respective digital image, the image vector for the respective digital image comprising a plurality of probability values for the plurality of image content classes, each probability value of the plurality of probability values reflecting a probability that the respective digital image belongs to a respective image content class of the plurality of image content classes; and
identifying a plurality of digital images as relevant to the input keyword unit based on the vector similarity measurements between the image match vector and the plurality of image vectors.
14. The one or more non-transitory computer-readable media of claim 8 , wherein the first embedding function and the second embedding function are a same embedding function.
15. A computing system comprising:
one or more processors;
storage media;
one or more computer programs stored in the storage media and configured for execution by the one or more processors, the one or more computer programs having instructions configured for:
obtaining a word vector representing an input keyword unit, the word vector generated according to an embedding function that projects keyword units in a vector space as N-dimensional vectors and in which distances between vectors in the vector space correspond to semantic similarities between the keyword units;
obtaining a plurality of word vectors representing a plurality of image content class keyword units, each image content class keyword unit describing a respective image content class of a plurality of image content classes, each word vector of the plurality of words vectors generated according to the embedding function;
generating an image match vector comprising a plurality of vector similarity measurements, each vector similarity measurement of the plurality of vector similarity measurements based on a computed vector similarity between (a) the word vector representing the input keyword unit and (b) a respective word vector of the plurality of word vectors;
computing a vector similarity measurement between the image match vector and a particular image vector, the particular image vector being for a particular digital image, the particular image vector for the particular digital image comprising a plurality of probability values for the plurality of image content classes, each probability value of the plurality of probability values reflecting a probability that the particular digital image belongs to a respective image content class of the plurality of image content classes; and
identifying the particular digital image as relevant to the input keyword unit based on the vector similarity measurement between the image match vector and the particular image vector.
16. The computing system of claim 15 , further comprising:
the one or more computer programs having instructions configured for:
generating the image match vector comprising the plurality of vector similarity measurements, each vector similarity measurement of the plurality of vector similarity measurements based on a computed dot product of (a) the word vector representing the input keyword unit and (b) the respective word vector of the plurality of word vectors.
17. The computing system of claim 15 , further comprising:
the one or more computer programs having instructions configured for:
computing a dot product of the image match vector and the particular image vector; and
identifying the particular digital image as relevant to the input keyword unit based on the dot product of the image match vector and the particular image vector.
18. The computing system of claim 15 , wherein a particular image content class of the plurality of image content classes is associated with two or more image content class keyword units of the plurality of image content class keyword units; and the computing system further comprising:
the one or more computer programs having instructions configured for:
for each image content class keyword unit of the two or more image content class keyword units, computing a vector similarity between the word vector representing the input keyword unit and the respective word vector of the image content class keyword unit; and
selecting, for inclusion in the image match vector as a representative vector similarity measure for the particular image content class, the vector similarity, of two or more vector similarities computed for the two or more image content class keyword units, representing a greatest semantic similarity, among the two or more image content class keyword units, with the input keyword unit.
19. The computing system of claim 15 , further comprising:
the one or more computer programs having instructions configured for:
based on the vector similarity measurement between the image match vector and the particular image vector, causing a graphical user interface to be presented at a client computing device, the graphical user interface indicating that the particular digital image is relevant to the input keyword unit.
20. The computing system of claim 15 , further comprising:
the one or more computer programs having instructions configured for:
computing a vector similarity measurement between the image match vector and each of a plurality of image vectors, each image vector of the plurality of image vectors being for a respective digital image, the image vector for the respective digital image comprising a plurality of probability values for the plurality of image content classes, each probability value of the plurality of probability values reflecting a probability that the respective digital image belongs to a respective image content class of the plurality of image content classes; and
identifying a plurality of digital images as relevant to the input keyword unit based on the vector similarity measurements between the image match vector and the plurality of image vectors.