Prediction of table column items in unstructured documents using a hybrid model
One example method includes collecting annotated unstructured documents that each include a table with words whose respective column indices are known, using the documents to train a model to detect a table header in a given document, identifying, by the model, a region of a document that corresponds to a table header in a new document that is not part of the training data, using an algorithm to perform a segmentation process on the table header that identifies column boundaries in the table header, and to use the identified column boundaries to preliminarily assign a respective column index to each word in the table header. Finally, a graph neural network model is run on a graph that includes the words in the table, and running the graph neural network generates a refined prediction of a respective column index for each of the words in the table of the new document.
1 . A method, comprising:
collecting training data that comprises annotated unstructured documents that each include one or more tables with words whose respective column indices are known;
training a model, using the training data, to detect a table body and a table header in a given document;
identifying, by the model, a region of a document that corresponds to a table header of a table in a new document that is not part of the training data;
using a rule-based algorithm to:
perform a segmentation process on the table header that returns a list of column boundaries identified in the table header; and
use the identified column boundaries to preliminarily assign a respective column index to each word in the table header; and
running a graph neural network model on a graph that includes nodes based on the words in the table and edges based on the identified column boundaries,
wherein running the graph neural network generates a refined prediction of a respective column index for each of the words in the table of the new document.
2 . The method as recited in claim 1 , wherein the new document is an unstructured, unannotated, document.
3 . The method as recited in claim 1 , wherein the model is a computer vision model.
4 . The method as recited in claim 1 , wherein the graph includes a respective node for each word in the table, and each of the nodes is associated with a respective feature vector.
5 . The method as recited in claim 1 , further comprising training the graph neural network prior to running the graph neural network on the graph.
6 . The method as recited in claim 5 , wherein training the graph neural network comprises:
performing a training process that comprises generating the graph, each of the words in the table corresponds to a node in the graph, and each node is associated with a respective feature vector; and
performing an inference process to predict a column index for each word in the table.
7 . The method as recited in claim 1 , wherein the model identifies the region of the document by identifying one or more bounding boxes of the table header.
8 . The method as recited in claim 1 , further comprising using the refined prediction of a respective column index for each of the words in the table of the new document to extract one or more of the words from the document, and processing the extracted words.
9 . The method as recited in claim 1 , wherein training the model comprises using an error function to minimize an error between a column index value assigned by the model and a column index value in the training data.
10 . The method as recited in claim 9 , wherein the error function comprises a first error function and a second error function.
11 . A non-transitory storage medium having stored therein instructions that are executable by one or more hardware processors to perform operations comprising:
collecting training data that comprises annotated unstructured documents that each include one or more tables with words whose respective column indices are known;
training a model, using the training data, to detect a table body and a table header in a given document;
identifying, by the model, a region of a document that corresponds to a table header of a table in a new document that is not part of the training data;
using a rule-based algorithm to:
perform a segmentation process on the table header that returns a list of column boundaries identified in the table header; and
use the identified column boundaries to preliminarily assign a respective column index to each word in the table header; and
running a graph neural network model on a graph that includes nodes based on the words in the table and edges based on the identified column boundaries,
wherein running the graph neural network generates a refined prediction of a respective column index for each of the words in the table of the new document.
12 . The non-transitory storage medium as recited in claim 11 , wherein the new document is an unstructured, unannotated, document.
13 . The non-transitory storage medium as recited in claim 11 , wherein the model is a computer vision model.
14 . The non-transitory storage medium as recited in claim 11 , wherein the graph includes a respective node for each word in the table, and each of the nodes is associated with a respective feature vector.
15 . The non-transitory storage medium as recited in claim 11 , further comprising training the graph neural network prior to running the graph neural network on the graph.
16 . The non-transitory storage medium as recited in claim 15 , wherein training the graph neural network comprises:
performing a training process that comprises generating the graph, each of the words in the table corresponds to a node in the graph, and each node is associated with a respective feature vector; and
performing an inference process to predict a column index for each word in the table.
17 . The non-transitory storage medium as recited in claim 11 , wherein the model identifies the region of the document by identifying one or more bounding boxes of the table header.
18 . The non-transitory storage medium as recited in claim 11 , further comprising using the refined prediction of a respective column index for each of the words in the table of the new document to extract one or more of the words from the document, and processing the extracted words.
19 . The non-transitory storage medium as recited in claim 11 , wherein training the model comprises using an error function to minimize an error between a column index value assigned by the model and a column index value in the training data.
20 . The non-transitory storage medium as recited in claim 19 , wherein the error function comprises a first error function and a second error function.