PROCESSING MULTI-TYPE DOCUMENT FOR MACHINE LEARNING COMPREHENSION

1 . A computer-implemented method comprising:

receiving, by a processor set, a digital image of a document and a workflow describing an automation task;

converting, by the processor set, the digital image of the document into rich text that includes layout information in the document;

creating, by the processor set, based on the rich text and the workflow, a tree of thoughts that includes nodes and edges connecting the nodes and that binds at least some nodes representing the rich text with a task node representing the automation task;

converting, by the processor set, the nodes and edges of the tree of thoughts into a natural language text; and

inputting, by the processor set, the natural language text into a language machine learning model with attention given to a token in the natural language text representing the task node, the language machine learning model, in response, outputting a result of completing the automation task.

2 . The computer-implemented method of claim 1 , wherein the inputting the natural language text into the language machine learning model includes iteratively inputting the natural language text to complete the automation task.

3 . The computer-implemented method of claim 1 , wherein computer vision and optical character recognition techniques are used to convert the digital image of the document into the rich text.

4 . The computer-implemented method of claim 1 , wherein the result output by the language machine learning model is presented on a user interface.

5 . The computer-implemented method of claim 1 , wherein the result output by the large language model is fed into another automation task.

6 . The computer-implemented method of claim 1 , wherein the layout information includes images and associated captions contained in the document.

7 . The computer-implemented method of claim 1 , wherein the layout information includes formatting of document content contained in the document.

8 . A computer program product comprising:

a set of one or more computer-readable storage media; and

program instructions, collectively stored in the set of one or more storage media, for causing a processor set to perform the following computer operations:

receive a digital image of a document and a workflow describing an automation task;

convert the digital image of the document into rich text that includes layout information in the document;

create, based on the rich text and the workflow, a tree of thoughts that includes nodes and edges connecting the nodes and that binds at least some nodes representing the rich text with a task node representing the automation task;

convert the nodes and edges of the tree of thoughts into a natural language text; and

input the natural language text into a language machine learning model with attention given to a token in the natural language text representing the task node, the language machine learning model, in response, outputting a result of completing the automation task.

9 . The computer program product of claim 8 , wherein the natural language text is iteratively input into the language machine learning model to complete the automation task.

10 . The computer program product of claim 8 , wherein computer vision and optical character recognition techniques are used to convert the digital image of the document into the rich text.

11 . The computer program product of claim 8 , wherein the result output by the language machine learning model is presented on a user interface.

12 . The computer program product of claim 8 , wherein the result output by the language machine learning model is fed into another automation task.

13 . The computer program product of claim 8 , wherein the layout information includes images and associated captions contained in the document.

14 . The computer program product of claim 8 , wherein the layout information includes formatting of document content contained in the document.

15 . A computer system comprising:

a processor set;

a set of one or more computer-readable storage media; and

program instructions, collectively stored in the set of one or more computer-readable storage media, for causing the processor set to perform the following computer operations:

receive a digital image of a document and a workflow describing an automation task;

convert the digital image of the document into rich text that includes layout information in the document;

create, based on the rich text and the workflow, a tree of thoughts that includes nodes and edges connecting the nodes and that binds at least some nodes representing the rich text with a task node representing the automation task;

convert the nodes and edges of the tree of thoughts into a natural language text; and

input the natural language text into a language machine learning model with attention given to a token in the natural language text representing the task node, the language machine learning model, in response, outputting a result of completing the automation task.

16 . The computer system of claim 15 , wherein the natural language text is iteratively input into the language machine learning model to complete the automation task.

17 . The computer system of claim 15 , wherein computer vision and optical character recognition techniques are used to convert the digital image of the document into the rich text.

18 . The computer system of claim 15 , wherein the result output by the language machine learning model is presented on a user interface.

19 . The computer system of claim 15 , wherein the result output by the language machine learning model is fed into another automation task.

20 . The computer system of claim 15 , wherein the layout information includes images and associated captions contained in the document.