Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing queries made up of images. In one aspect, a method includes indexing images by image descriptors. The method further includes associating descriptive n-grams with the images. In another aspect, a method includes receiving a query, identifying text describing the query, and performing a search according to the text identified for the query.
1. A method performed by data processing apparatus, the method comprising:
obtaining a group of similar images for a particular image, wherein each similar image is associated with one or more n-grams;
determining an overall score for each of the n-grams, wherein for each n-gram the overall score is based at least in part on: i) a combination of image specific scores for the n-gram for images in the group of similar images and ii) the number of similar images in the group of similar images; and
selecting one of the n-grams as a textual description of the particular image according at least in part on the overall scores for the n-grams and the respective n-gram order of each of the n-grams representing a number of tokens in the n-gram.
2. The method of claim 1 , wherein obtaining a group of similar images for a particular image comprises querying an index that associates images with descriptors.
3. The method of claim 1 , further comprising storing a database associating each image with one or more n-grams; using the database to determine an n-gram associated with a received image query; and using the n-gram to obtain one or more search results.
4. The method of claim 1 , wherein each image-specific score for a particular n-gram is calculated based on image affinity calculated between similar images associated with the n-gram.
5. The method of claim 1 , wherein the overall score for an n-gram is based on a sum of the image-specific scores for the n-gram divided by a total number of images associated with the n-gram.
6. The method of claim 1 , further comprising filtering one or more of the n-grams according to a blacklist of terms.
7. The method of claim 1 , further comprising filtering one or more of the n-grams to remove n-grams that contain stop words.
8. The method of claim 1 , wherein selecting one of the n-grams comprises:
selecting a current best n-gram from one or more 1-grams in the n-grams, where the current best n-gram has a highest overall score of all of the one or more 1-grams in the n-grams;
considering each other highest overall scoring n-gram of all of the n-grams at each order of n-grams greater than 1 in increasing order, and determining, for each highest overall scoring n-gram at a given order whether to update the current best n-gram to be the highest overall scoring n-gram of the order, where the current best n-gram is updated when the highest overall scoring n-gram of the order has a higher overall score than a score associated with the current best n-gram or when the highest overall scoring n-gram is a superstring of the current best n-gram and has a score that satisfies an acceptance threshold.
9. The method of claim 8 , further comprising updating the current best n-gram to be the determined highest overall scoring n-gram of the order when it is determined that the current best n-gram should be updated.
10. The method of claim 9 , wherein updating the current best n-gram further comprises associating a score with the updated current best n-gram, wherein the score is the overall score of the updated current best n-gram when the updated current best n-gram has an overall score that is higher than the score associated with the previous best n-gram, and otherwise is the score of the previous best n-gram.
11. A system, comprising:
one or more computers operable to perform operations comprising:
obtaining a group of similar images for a particular image, wherein each similar image is associated with one or more n-grams;
determining an overall score for each of the n-grams, wherein for each n-gram the overall score is based at least in part on: i) a combination of image specific scores for the n-gram for images in the group of similar images and ii) the number of similar images in the group of similar images; and
selecting one of the n-grams as a textual description of the particular image according at least in part on the overall scores for the n-grams and the respective n-gram order of each of the n-grams representing a number of tokens in the n-gram.
12. The system of claim 11 , wherein obtaining a group of similar images for a particular image comprises querying an index that associates images with descriptors.
13. The system of claim 11 , further operable to perform operations comprising storing a database associating each image with one or more n-grams; using the database to determine an n-gram associated with a received image query; and using the n-gram to obtain one or more search results.
14. The system of claim 11 , wherein each image-specific score for an n-gram is based on image affinity calculated between similar images associated with the n-gram.
15. The system of claim 11 , wherein the overall score for an n-gram is based on a sum of the image-specific scores for the n-gram divided by a total number of images associated with the n-gram.
16. The system of claim 11 , further operable to perform operations comprising filtering one or more of the n-grams according to a blacklist of terms.
17. The system of claim 11 further operable to perform operations comprising filtering one or more of the n-grams to remove n-grams that contain stop words.
18. The system of claim 11 , wherein selecting one of the n-grams comprises:
selecting a current best n-gram from one or more 1-grams in the n-grams, where the current best n-gram has a highest overall score of all of the one or more 1-grams in the n-grams;
considering each other highest overall scoring n-gram of all of the n-grams at each order of n-grams greater than 1 in increasing order, and determining, for each highest overall scoring n-gram at a given order whether to update the current best n-gram to be the highest overall scoring n-gram of the order, where the current best n-gram is updated when the highest overall scoring n-gram of the order has a higher overall score than a score associated with the current best n-gram or when the highest overall scoring n-gram is a superstring of the current best n-gram and has a score that satisfies an acceptance threshold.
19. The system of claim 18 , further comprising updating the current best n-gram to be the determined highest overall scoring n-gram of the order when it is determined that the current best n-gram should be updated.
20. The system of claim 19 , wherein updating the current best n-gram further comprises associating a score with the updated current best n-gram, wherein the score is the overall score of the updated current best n-gram when the updated current best n-gram has an overall score that is higher than the score associated with the previous best n-gram, and otherwise is the score of the previous best n-gram.
21. A non-transitory computer-readable storage medium storing instructions that, when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:
obtaining a group of similar images for a particular image, wherein each similar image is associated with one or more n-grams;
determining an overall score for each of the n-grams, wherein for each n-gram the overall score is based at least in part on: i) a combination of image specific scores for the n-gram for images in the group of similar images and ii) the number of similar images in the group of similar images; and
selecting one of the n-grams as a textual description of the particular image according at least in part on the overall scores for the n-grams and the respective n-gram order of each of the n-grams representing a number of tokens in the n-gram.