Contextually disambiguating queries
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for contextually disambiguating queries are disclosed. In an aspect, a method includes receiving an image being presented on a display of a computing device and a transcription of an utterance spoken by a user of the computing device, identifying a particular sub-image that is included in the image, and based on performing image recognition on the particular sub-image, determining one or more first labels that indicate a context of the particular sub-image. The method also includes, based on performing text recognition on a portion of the image other than the particular sub-image, determining one or more second labels that indicate the context of the particular sub-image, based on the transcription, the first labels, and the second labels, generating a search query, and providing, for output, the search query.
1. A method implemented by one or more processors, the method comprising:
identifying an image that is displayed at a computing device;
receiving data indicating a user request at the computing device, wherein the user request is received while the image is displayed on the computing device;
processing the image to identify a plurality of sub-images included in the image and displayed on the computing device when the user request is received, wherein each sub-image of the plurality of sub-images included in the image comprises a separate portion of the image;
generating, based on the data indicating the user request, a plurality of confidence scores, wherein each of the confidence scores is for a corresponding one of the sub-images displayed on the computing device when the user request is received and indicates a corresponding likelihood that the corresponding one of the sub-images is a focus of the user request;
selecting a particular sub-image, of the sub-images, based on the confidence scores;
generating, based on performing image recognition on the particular sub-image, a plurality of labels, wherein each of the labels represents attributes of at least one entity included in a separate portion of the sub-image;
generating, for each of the labels, a corresponding label confidence score indicating a likelihood that the corresponding label corresponds to the separate portion of the sub-image that is relevant to the user request;
selecting a subset of the labels based on the corresponding label confidence scores;
generating, based on the subset of the labels and the data indicating the user request, a search request; and
providing, for output at the computing device, one or more search results responsive to the search request.
2. The method of claim 1 , wherein generating the search request based on the labels and the data indicating the user request comprises:
generating, based on the labels and the data indicating the user request, multiple candidate search requests, including the search request;
accessing historical user request data of a plurality of users;
comparing the multiple candidate search requests to the historical user request data; and
selecting the search request based on the comparison.
3. The method of claim 1 , further comprising:
identifying metadata associated with the particular sub-image;
wherein generating the labels is further based on the metadata associated with the particular sub-image.
4. The method of claim 1 , wherein generating the search request is further based on the corresponding label confidence scores for the subset of the labels.
5. A computing device comprising:
a display;
one or more processors; and
one or more storage devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform actions comprising:
identifying an image that is displayed at the display of the computing device;
receiving data indicating a user request at the computing device, wherein the user request is received while the image is displayed at the display;
processing the image to identify a plurality of sub-images included in the image and displayed at the display when the user request is received, wherein each sub-image of the plurality of sub-images included in the image comprises a separate portion of the image;
generating, based on the data indicating the user request, a plurality of confidence scores, wherein each of the confidence scores is for a corresponding one of the sub-images displayed at the display when the user request is received and indicates a corresponding likelihood that the corresponding one of the sub-images is a focus of the user request;
selecting a particular sub-image, of the sub-images, based on the confidence scores;
generating, based on performing image recognition on the particular sub-image, a plurality of labels, wherein each of the labels represents attributes of at least one entity included in a separate portion of the sub-image;
generating, for each of the labels, a corresponding label confidence score indicating a likelihood that the corresponding label corresponds to the separate portion of the sub-image that is relevant to the user request;
selecting a subset of the labels based on the corresponding label confidence scores;
generating, based on the subset of the labels and the data indicating the user request, a search request; and
providing, for output at the computing device, one or more search results responsive to the search request.
6. The computing device of claim 5 , wherein generating the search request based on the labels and the data indicating the user request comprises:
generating, based on the labels and the data indicating the user request, multiple candidate search requests, including the search request;
accessing historical user request data of a plurality of users;
comparing the multiple candidate search requests to the historical user request data; and
selecting the search request based on the comparison.
7. The computing device of claim 5 , comprising further instructions that, when executed by the one or more processors, cause the one or more processors to perform further actions comprising:
identifying metadata associated with the particular sub-image;
wherein generating the labels is further based on the metadata associated with the particular sub-image.
8. The computing device of claim 5 , wherein generating the search request is further based on the corresponding label confidence scores for the subset of the labels.
9. One or more non-transitory computer readable storage media storing thereon a computer program comprising instructions that when executed by a computer system, cause the computer system to perform operations including:
identifying an image that is displayed at a computing device;
receiving data indicating a user request at the computing device, wherein the user request is received while the image is displayed on the computing device;
processing the image to identify a plurality of sub-images included in the image and displayed on the computing device when the user request is received, wherein each sub-image of the plurality of sub-images included in the image comprises a separate portion of the image;
generating, based on the data indicating the user request, a plurality of confidence scores, wherein each of the confidence scores is for a corresponding one of the sub-images displayed on the computing device when the user request is received and indicates a corresponding likelihood that the corresponding one of the sub-images is a focus of the user request;
selecting a particular sub-image, of the sub-images, based on the confidence scores;
generating, based on performing image recognition on the particular sub-image, a plurality of labels, wherein each of the labels represents attributes of at least one entity included in a separate portion of the sub-image;
generating, for each of the labels, a corresponding label confidence score indicating a likelihood that the corresponding label corresponds to the separate portion of the sub-image that is relevant to the user request;
selecting a subset of the labels based on the corresponding label confidence scores;
generating, based on the subset of the labels and the data indicating the user request, a search request; and
providing, for output at the computing device, one or more search results responsive to the search request.
10. The one or more non-transitory computer readable storage media of claim 9 , wherein generating the search request based on the labels and the data indicating the user request comprises:
generating, based on the labels and the data indicating the user request, multiple candidate search requests, including the search request;
accessing historical user request data of a plurality of users;
comparing the multiple candidate search requests to the historical user request data; and
selecting the search request based on the comparison.
11. The one or more non-transitory computer readable storage media of claim 9 , comprising further instructions that, when executed by the one or more processors, cause the one or more processors to perform further actions comprising:
identifying metadata associated with the particular sub-image;
wherein generating the labels is further based on the metadata associated with the particular sub-image.
12. The one or more non-transitory computer readable storage media of claim 9 , wherein generating the search request is further based on the corresponding label confidence scores for the subset of the labels.