Systems and methods for analyzing text extracted from images and performing appropriate transformations on the extracted text
The present disclosure provides computer-implemented methods, systems, and devices for responding to requests associated with an image. A computing system obtains, wherein the image depicts a first set of textual content. The computing system determines one or more characteristics of the first set of textual content. The computing system determines a response type from a plurality of response types based on the one or more characteristics. The computing system generates a model input, wherein the model input comprises data descriptive of the first set of textual content and a prompt associated with the response type. The computing system provides providing the model input as an input to a machine-learned language model. The computing system receives a second set of text as an output of the machine-learned language model as a result of the machine-learned language model processing the model input. The computing system provides the second set of text for display to a user, wherein the second set of textual content is associated with the response type.
1 . A computing system, the system comprising:
one or more processors; and
one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
accessing an image displayed by the computing system, wherein the image depicts a first set of displayed textual content;
determining one or more characteristics of the first set of displayed textual content, wherein the one or more characteristics of the first set of displayed textual content includes a density of the first set of displayed textual content;
responsive to a determination that the density for the first set of displayed textual content within the image satisfies a threshold, updating a user interface to include a summarize user interface element;
generating a model input, wherein the model input comprises data descriptive of the first set of displayed textual content and a prompt including instructions to create a summary of the first set of displayed textual content;
providing the model input as an input to a machine-learned language model;
receiving a second set of text as an output of the machine-learned language model as a result of the machine-learned language model processing the model input; and
providing the second set of text for display to a user, wherein the second set of textual content is a summary of the first set of displayed textual content.
2 . The computing system of claim 1 , wherein determining the density of the first set of displayed textual content within the image further comprises:
determining an area of the image that includes the first set of displayed textual content;
determining a total area of the image; and
determining a percentage of the image that includes the first set of displayed textual content.
3 . The computing system of claim 2 , wherein determining the density of the first set of displayed textual content within the image further comprises:
determining a total number of words visible in the image.
4 . The computing system of claim 3 , wherein determining the density of the first set of displayed textual content within the image further comprises:
determining a number of words per pixel in the image.
5 . The computing system of claim 4 , wherein the computing system generates the model input in response to a user request.
6 . The computing system of claim 5 , wherein the user request is input by a user selecting the summarize user interface element displayed proximate to the image in the user interface of a user computing device.
7 . The computing system of claim 1 , wherein the second set of textual content has less textual content than the first set of displayed textual content.
8 . The computing system of claim 1 , wherein an optical character recognition process is used to generate text data representing the content of the first set of displayed textual content from the image.
9 . The computing system of claim 1 , wherein the machine-learned language model is a large language model.
10 . The computing system of claim 1 , the operations further comprising:
updating the user interface to display the second set of textual content.
11 . The computing system of claim 1 , wherein the input to the machine-learned model is multimodal.
12 . The computing system of claim 1 , wherein the machine-learned language model is operated at a remote server system and the model input is transmitted to the remote server system and the second set of textual content is received from the remote server system.
13 . A computer-implemented method for summarizing text in an image, the method comprising:
accessing, by a computing system with one or more processors, an image displayed by the computing system, wherein the image depicts a first set of textual content;
determining, by the computing system, one or more characteristics of the first set of displayed textual content, wherein the one or more characteristics of the first set of displayed textual content includes a density of the first set of displayed textual content;
responsive to a determination that the density for the first set of displayed textual content within the image satisfies a threshold, updating, by the computing system, a user interface to include a summarize user interface element;
generating, by the computing system, a model input, wherein the model input comprises data descriptive of the first set of displayed textual content and a prompt including instructions to create a summary of the first set of displayed textual content;
providing, by the computing system, the model input as an input to a machine-learned language model;
receiving, by the computing system, a second set of textual content as an output of the machine-learned language model as a result of the machine-learned language model processing the model input; and
providing, by the computing system, the second set of text for display to a user, wherein the second set of textual content is a summary of the first set of displayed textual content.
14 . The computer-implemented method of claim 13 , wherein determining the density of the first set of displayed textual content within the image further comprises:
determining, by the computing system, an area of the image that includes the first set of displayed textual content;
determining, by the computing system, a total area of the image; and
determining, by the computing system, a percentage of the image that includes the first set of displayed textual content.
15 . The computer-implemented method of claim 14 , wherein determining the density of the first set of displayed textual content within the image further comprises:
determining, by the computing system, a total number of words visible in the image.
16 . The computer-implemented method of claim 15 , wherein determining the density of the first set of displayed textual content within the image further comprises:
determining, by the computing system, a number of words per pixel in the image.
17 . One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:
accessing an image displayed by a computing device in the one or more computing devices, wherein the image depicts a first set of displayed textual content;
determining one or more characteristics of the first set of displayed textual content, wherein the one or more characteristics of the first set of displayed textual content includes a density of the first set of displayed textual content;
responsive to a determination that the density for the first set of displayed textual content within the image satisfies a threshold, updating a user interface to include a summarize user interface element;
generating a model input, wherein the model input comprises data descriptive of the first set of displayed textual content and a prompt including instructions to create a summary of the first set of displayed textual content;
providing the model input as an input to a machine-learned language model;
receiving a second set of text as an output of the machine-learned language model as a result of the machine-learned language model processing the model input; and
providing the second set of text for display to a user, wherein the second set of textual content is a summary of the first set of displayed textual content.
18 . The one or more non-transitory computer-readable media of claim 17 , wherein determining the density of the first set of displayed textual content within the image further comprises:
determining an area of the image that includes the first set of displayed textual content;
determining a total area of the image; and
determining a percentage of the image that includes the first set of displayed textual content.
19 . The one or more non-transitory computer-readable media of claim 18 , wherein determining the density of the first set of displayed textual content within the image further comprises:
determining a total number of words visible in the image.
20 . The one or more non-transitory computer-readable media of claim 19 , wherein determining the density of the first set of displayed textual content within the image further comprises:
determining a number of words per pixel in the image.