System for enhanced ink gesture support
Systems, methods, and software are disclosed herein for automating the editing of documents based on ink gestures via an LLM integration in various implementations. In an implementation, a computing device receives an ink gesture made by a user with respect to a document displayed in the user interface of an application. The ink gesture includes ink strokes made on a portion of the document. The computing device generates a prompt for an LLM to obtain suggested revisions to the document, wherein the prompt includes an image of the ink strokes made on the portion of the document. The prompt also includes a document tree indicative of a structure of the document. The computing device surfaces a suggested revision generated by the LLM in response to the prompt in the user interface.
1 . A computing apparatus comprising:
one or more computer readable storage media;
one or more processors operatively coupled with the one or more computer readable storage media; and
program instructions stored on the one or more computer readable storage media that, when executed by the one or more processors, direct the computing apparatus to at least:
receive, in a user interface of an application, an ink gesture made by a user with respect to a document displayed in the user interface, wherein the ink gesture comprises ink strokes made on a portion of the document;
capture an image of the ink strokes on the portion of the document;
generate a prompt for a large language model to obtain suggested revisions to the document, wherein the prompt includes the image and a document tree indicative of a structure of the document; and
surface, in the user interface, a suggested revision generated by the large language model in response to the prompt.
2 . The computing apparatus of claim 1 , wherein the prompt tasks the large language model with generating the suggested revisions to the document based on the image of the ink strokes on the portion of the document and in view of the document tree.
3 . The computing apparatus of claim 2 , wherein the prompt further comprises context information relating to the ink gesture, wherein the context information comprises a location in the document tree of the portion of the document subject to the ink gesture.
4 . The computing apparatus of claim 3 , wherein the portion of the document comprises a sentence, a paragraph, or a page of the document.
5 . The computing apparatus of claim 4 , wherein the prompt further includes a different portion of the document comprising a sentence, a paragraph, or a page of the document not subject to the ink gesture.
6 . The computing apparatus of claim 1 , wherein the response comprises a request by the large language model for a different portion of the document, wherein the different portion of the document is not captured in the image, and wherein the program instructions further direct the computing apparatus to send a second prompt to the large language model including the different portion of the document.
7 . The computing apparatus of claim 1 , wherein the program instructions further direct the computing apparatus to parse the ink gesture to determine a meaning represented by the ink gesture and include the meaning in the prompt.
8 . The computing apparatus of claim 1 , wherein the image comprises a screen capture in a JPEG or PNG format.
9 . One or more computer readable storage media having program instructions stored thereon that, when executed by one or more processors, direct a computing device to at least:
receive, in a user interface of an application, an ink gesture made by a user with respect to a document displayed in the user interface, wherein the ink gesture comprises ink strokes made on a portion of the document;
generate a prompt for a large language model to obtain suggested revisions to the document, wherein the prompt includes an image of the ink strokes on the portion of the document, and wherein the prompt further includes a document tree indicative of a structure of the document; and
surface, in the user interface, a suggested revision generated by the large language model in response to the prompt.
10 . The one or more computer readable storage media of claim 9 , wherein the prompt tasks the large language model with generating the suggested revisions to the document based on the image of the ink strokes on the portion of the document in view of the document tree.
11 . The one or more computer readable storage media of claim 10 , wherein the prompt further comprises context information relating to the ink gesture, wherein the context information comprises a location in the document tree of the portion of the document subject to the ink gesture.
12 . The one or more computer readable storage media of claim 11 , wherein the portion of the document comprises a sentence, a paragraph, or a page of the document.
13 . The one or more computer readable storage media of claim 12 , wherein the prompt further includes a different portion of the document comprising a sentence, a paragraph, or a page of the document not subject to the ink gesture.
14 . The one or more computer readable storage media of claim 9 , wherein the response comprises a request by the large language model for a different portion of the document, wherein the different portion of the document is not captured in the image, and wherein the program instructions further direct the computing device to send a second prompt to the large language model including the different portion of the document.
15 . The one or more computer readable storage media of claim 9 , wherein the program instructions further direct the computing device to parse the ink gesture to determine a meaning represented by the ink gesture and include the meaning in the prompt.
16 . The one or more computer readable storage media of claim 9 , wherein the image comprises a screen capture in a JPEG or PNG format.
17 . A method of operating a computing device comprising:
receiving, in a user interface of an application, an ink gesture made by a user with respect to a document displayed in the user interface, wherein the ink gesture comprises ink strokes made on a portion of the document;
capturing an image of the ink strokes on the portion of the document;
generating a prompt for a large language model to obtain suggested revisions to the document, wherein the prompt includes the image and a document tree indicative of a structure of the document; and
surfacing, in the user interface, a suggested revision generated by the large language model in response to the prompt.
18 . The method of claim 17 , wherein the prompt tasks the large language model with generating the suggested revisions to the document based on the image of the ink strokes on the portion of the document in view of the document tree.
19 . The method of claim 18 , wherein the prompt further comprises context information relating to the ink gesture, wherein the context information comprises a location in the document tree of the portion of the document subject to the ink gesture.
20 . The method of claim 19 , wherein the prompt further includes a different portion of the document not subject to the ink gesture.