Efficiently augmenting images with related content
The subject matter of this specification generally relates to providing content related to text depicted in images. In one aspect, a system includes a data processing apparatus configured to extract text from an image. The extracted text is partitioned into multiple blocks. The multiple blocks are presented as respective first user-selectable targets on a user interface at a first zoom level. A user selection of a first block of the multiple blocks is detected. In response to detecting the user selection of the first block, portions of the extracted text in the first block are presented as respective second user-selectable targets on the user interface at a second zoom level greater than the first zoom level. In response to detecting a user selection of a portion of the extracted text within the first block, an action is initiated based on content of the user-selected text.
1. A method comprising:
obtaining, by a computing system comprising one or more processors, an image;
extracting, by the computing system, text from the image;
partitioning, by the computing system, the text into a plurality of blocks;
providing, by the computing system, data descriptive of a user interface that presents the plurality of blocks as respective first user-selectable targets;
obtaining, by the computing system, a user selection of a first block of the plurality of blocks, wherein the first block comprises a first set of text;
generating, by the computing system, a search query based on the first set of text;
determining, by the computing system, a context for the image, wherein the context is descriptive of category classification for the image;
processing, by the computing system, the search query and the context to determine search result content, wherein the search result content is determined based on the first set of text and the context; and
providing, by the computing system, data descriptive of an updated user interface that is descriptive the search result content presented with a portion of the image that includes the first block.
2. A method as claimed in claim 1 , wherein determining the context for the image comprises: processing, by the computing system, the image with one or more machine learning models to classify the image.
3. A method as claimed in claim 2 , wherein an output of the one or more machine learning models comprises a predefined category.
4. A method as claimed in claim 1 , wherein the context of the image is determined based on other text depicted in the image.
5. A method as claimed in claim 1 , wherein the search result content is associated with a resource responsive to the search query and associated with the context.
6. A method as claimed in claim 1 , wherein the category classification comprises at least one of: a menu classification, a music classification, and/or a sign classification.
7. The method as claimed in claim 1 , wherein processing the search query and the context to determine the search result content comprises:
modifying, by the computing system, the search query to include one or more additional terms based on the context.
8. The method of claim 1 , wherein processing the search query and the context to determine the search result content comprises:
identifying, by the computing system, a plurality of electronic resources based on the search query; and
modifying, by the computing system, a ranking of the plurality of electronic resources based on the context.
9. A method of claim 8 , wherein the plurality of electronic resources comprises one or more web pages.
10. The method of claim 8 , wherein the plurality of electronic resources comprises one or more images.
11. The method of claim 8 , wherein the plurality of electronic resources comprises one or more videos.
12. The method of claim 1 , wherein the category classification is one of a predefined set of categories.
13. The method of claim 1 , wherein determining the context for the image comprises: processing the image with one or more machine learning models to determine the category classification.
14. A system comprising:
a data processing apparatus; and
a memory apparatus in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising:
obtaining an image;
extracting text from the image;
partitioning the text into a plurality of blocks;
providing data descriptive of a user interface that presents the plurality of blocks as respective first user-selectable targets;
obtaining a user selection of a first block of the plurality of blocks, wherein the first block comprises a first set of text;
generating a search query based on the first set of text;
determining a context for the image, wherein the context is descriptive of category classification for the image;
processing the search query and the context to determine search result content, wherein the search result content is determined based on the first set of text and the context; and
providing data descriptive of an updated user interface that is descriptive the search result content presented with a portion of the image that includes the first block.
15. The system of claim 14 , wherein partitioning the extracted text into the blocks is based at least partially on semantic analysis of the extracted text.
16. The system of claim 14 , wherein the category classification comprises a menu classification.
17. The system of claim 14 , wherein the category classification comprises a music classification.
18. One or more non-transitory computer-readable storage media having instructions stored thereon, which, when executed by a data processing apparatus, cause the data processing apparatus to perform operations comprising:
obtaining an image;
extracting text from the image;
partitioning the text into a plurality of blocks;
providing data descriptive of a user interface that presents the plurality of blocks as respective first user-selectable targets;
obtaining a user selection of a first block of the plurality of blocks, wherein the first block comprises a first set of text;
generating a search query based on the first set of text;
determining a context for the image, wherein the context is descriptive of category classification for the image;
processing the search query and the context to determine search result content, wherein the search result content is determined based on the first set of text and the context; and
providing data descriptive of an updated user interface that is descriptive the search result content presented with a portion of the image that includes the first block.
19. The one or more non-transitory computer-readable storage media of claim 18 , wherein the category classification comprises a menu classification, and wherein the first set of text is descriptive of a food item.
20. The one or more non-transitory computer-readable storage media of claim 18 , wherein the category classification comprises a sign classification.