Query by image
View Patent ↗Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing queries made up of images. In one aspect, a method includes indexing images by image descriptors. The method further includes associating descriptive n-grams with the images. In another aspect, a method includes receiving a query, identifying text describing the query, and performing a search according to the text identified for the query.
1. A method performed by data processing apparatus, the method comprising:
receiving an image as a query from a user device associated with a user;
deriving a textual query corresponding to the received image, wherein deriving the textual query for the image includes:
using the image to determine a group of images that are similar to the received image;
for each similar image of the group of similar images, identifying one or more n-grams associated with the similar image; and
selecting one or more of the identified n-grams as a textual query for the received image;
obtaining search results responsive to the textual query; and
providing for presentation on the user device one or more of the obtained search results.
2. The method of claim 1 , wherein the textual query comprises an n-gram considered a description of the received image.
3. The method of claim 1 , wherein the image is received from a mobile device.
4. The method of claim 1 , where deriving the textual query corresponding to the image comprises:
determining an overall score for each of the one or more of the n-grams; and
selecting one or more of the n-grams as a description of the particular image based on the respective overall scores.
5. The method of claim 1 , wherein obtaining a group of similar images for a particular image comprises querying an index that associates images with descriptors using a vector of for each local descriptor of the image.
6. The method of claim 1 , wherein the search results are image search results comprising a group of one or more images.
7. The method of claim 1 , further comprising:
obtaining additional search results that are visually similar to the image query; and
providing one or more of the additional search results for presentation with the search results.
8. A system comprising:
one or more computers operable to perform operations comprising:
receiving an image as a query from a user device associated with a user;
deriving a textual query corresponding to the received image, wherein deriving the textual query for the image includes:
using the image to determine a group of images that are similar to the received image;
for each similar image of the group of similar images, identifying one or more n-grams associated with the similar image; and
selecting one or more of the identified n-grams as a textual query for the received image;
obtaining search results responsive to the textual query; and
providing for presentation on the user device one or more of the obtained search results.
9. The system of claim 8 , wherein the textual query comprises an n-gram considered a description of the received image.
10. The system of claim 8 , where the image is received from a mobile device.
11. The system of claim 8 , where deriving the textual query corresponding to the image comprises:
determining an overall score for each of the one or more of the n-grams; and
selecting one or more of the n-grams as a description of the particular image based on the respective overall scores.
12. The system of claim 8 , wherein obtaining a group of similar images for a particular image comprises querying an index that associates images with descriptors using a vector of for each local descriptor of the image.
13. The system of claim 8 , where the search results are image search results comprising a group of one or more images.
14. The system of claim 8 , wherein the one or more computers are further operable to perform operations comprising:
obtaining additional search results that are visually similar to the image query; and
providing one or more of the additional search results for presentation with the search results.
15. A non-transitory computer-readable storage medium storing instructions that, when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:
receiving an image as a query from a user device associated with a user;
deriving a textual query corresponding to the received image, wherein deriving the textual query for the image includes:
using the image to determine a group of images that are similar to the received image;
for each similar image of the group of similar images, identifying one or more n-grams associated with the similar image; and
selecting one or more of the identified n-grams as a textual query for the received image;
obtaining search results responsive to the textual query; and
providing for presentation on the user device one or more of the obtained search results.
16. The non-transitory computer-readable storage medium of claim 15 , wherein the textual query comprises an n-gram considered a description of the received image.
17. The non-transitory computer-readable storage medium of claim 15 , where the image is received from a mobile device.
18. The non-transitory computer-readable storage medium of claim 15 , where deriving the textual query corresponding to the image comprises:
determining an overall score for each of the one or more of the n-grams; and
selecting one or more of the n-grams as a description of the particular image based on the respective overall scores.
19. The non-transitory computer-readable storage medium of claim 15 , wherein obtaining a group of similar images for a particular image comprises querying an index that associates images with descriptors using a vector of for each local descriptor of the image.
20. The non-transitory computer-readable storage medium of claim 15 , where the search results are image search results comprising a group of one or more images.