Mapping images to search queries
Methods, systems, and apparatus for receiving a query image, receiving one or more entities that are associated with the query image, identifying, for one or more of the entities, one or more candidate search queries that are pre-associated with the one or more entities, generating a respective relevance score for each of the candidate search queries, selecting, as a representative search query for the query image, a particular candidate search query based at least on the generated respective relevance scores and providing the representative search query for output in response to receiving the query image.
1. A computing system comprising:
one or more processors;
one or more non-transitory computer readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
obtaining, a data encoding, wherein the data encoding comprises an encoding of a query image, wherein the query image comprises one or more image features;
processing the query image with an image annotator to identify one or more query image labels, wherein the one or more query image labels label one or more objects in the query image and wherein the one or more query image labels comprise coarse grained image labels, wherein the coarse grained image labels label one or more classes of objects;
determining one or more entities associated with the one or more query image labels, wherein the one or more entities comprise specific instances of the one or more classes of objects;
determining a plurality of candidate search queries based at least in part on the one or more entities;
determining a context associated with the query image;
determining a representative search query of the plurality of candidate search queries based at least in part on the context associated the query image;
obtaining a search results page associated with the representative search query; and
providing the search results page for display.
2. The computing system of claim 1 , wherein the operations further comprise:
generating a respective relevance score for each of the plurality of candidate search queries comprises, for each candidate search query:
determining a popularity of the candidate search query; and
based on the determined popularity, generating a respective relevance score for the candidate search query;
wherein the representative search query is determined based at least in part on the one or more entities, the context, and the relevance scores.
3. The computing system of claim 1 , wherein determining the representative search query of the plurality of candidate search queries based at least in part on the context associated the query image comprises:
identifying, for one or more of the entities, one or more candidate search queries, wherein the one or more candidate search queries are textual search queries and wherein the one or more candidate search queries are different than one or more terms associated with the one or more entities.
4. The computing system of claim 1 , wherein determining the representative search query of the plurality of candidate search queries based at least in part on the context associated the query image comprises:
generating a respective relevance score for each of the candidate search queries; and
selecting the representative search query for the query image based at least on the generated respective relevance scores.
5. The computing system of claim 1 , wherein the operations further comprise:
generating a respective relevance score for each of the candidate search queries comprises, for each candidate search query:
generating a search results page using the candidate search query;
analyzing the generated search results page to determine a measure indicative of how interesting and useful the search results page is; and
based on the determined measure, generating a respective relevance score for the candidate search query.
6. The computing system of claim 1 , wherein the one or more query image labels tag the one or more image features in the query image.
7. The computing system of claim 1 , wherein the one or more image features comprise one or more coarse-grained features.
8. The computing system of claim 1 , wherein the one or more image features comprise one or more fine-grained features.
9. The computing system of claim 1 , wherein the search results page comprises a plurality of search results responsive to the representative search query.
10. The computing system of claim 1 , wherein the query image comprises an image found on a website accessed by a user device.
11. The computing system of claim 1 , wherein the search results page comprises a knowledge panel, wherein the knowledge panel comprises general information associated with the one or more entities associated with the one or more query image labels.
12. A computer-implemented method, the method comprising:
obtaining, a data encoding, wherein the data encoding comprises an encoding of by a computing system comprising one or more processors, a query image, wherein the query image comprises one or more image features;
processing, by the computing system, the query image with an image annotator to identify one or more query image labels, wherein the one or more query image labels label one or more objects in the query image and wherein the one or more query image labels comprise coarse grained image labels, wherein the coarse grained image labels label one or more classes of objects;
determining, by the computing system, one or more entities associated with the one or more query image labels, wherein the one or more entities comprise specific instances of the one or more classes of objects;
determining, by the computing system, a plurality of candidate search queries based at least in part on the one or more entities;
determining, by the computing system, a context associated with the query image;
determining, by the computing system, a representative search query of the plurality of candidate search queries based at least in part on the context associated the query image;
obtaining, by the computing system, a search results page associated with the representative search query; and
providing, by the computing system, the search results page for display.
13. The method of claim 12 , further comprising:
receiving a natural language query; and
generating a respective relevance score for each of the candidate search queries based at least on the received natural language query.
14. The method of claim 12 , further comprising:
providing, by the computing system, the one or more entities for display.
15. The method of claim 12 , wherein determining, by the computing system, the context associated with the query image comprises:
determining, by the computing system, at least one of an intent of a user or a location of the user.
16. The method of claim 12 , wherein the representative search query is determined based at least in part on at least one of an intent of a user, a determined popularity of a particular candidate search query, an association with one or more relevant search result pages, or an association with a received natural language query.
17. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:
obtaining a data encoding, wherein the data encoding comprises an encoding of a query image, wherein the query image comprises one or more image features;
processing the query image with an image annotator to identify one or more query image labels, wherein the one or more query image labels label one or more objects in the query image and wherein the one or more query image labels comprise coarse grained image labels, wherein the coarse grained image labels label one or more classes of objects;
determining one or more entities associated with the one or more query image labels, wherein the one or more entities comprise specific instances of the one or more classes of objects;
determining a plurality of candidate search queries based at least in part on the one or more entities;
determining a context associated with the query image;
determining a representative search query of the plurality of candidate search queries based at least in part on the context associated the query image;
obtaining a search results page associated with the representative search query; and
providing the search results page for display.
18. The one or more non-transitory computer-readable media of claim 17 , wherein the search results page comprises one or more images and one or more textual search results responsive to the representative search query.
19. The one or more non-transitory computer-readable media of claim 17 , wherein determining the plurality of candidate search queries based at least in part on the one or more entities comprises determining the plurality of candidate search queries with a knowledge engine, wherein the knowledge engine is configured to identify candidate search queries that are associated with the one or more entity in a language that matches a user language.
20. The one or more non-transitory computer-readable media of claim 19 , wherein the user language is indicated by a user device associated with the query image.