Graph based technique for automatically identifying a transactions table within a document
A method for automatically identifying a transactions table in a document includes obtaining a document object model of the document. The document object model includes a plurality of nodes and a plurality of edges, with each of the nodes corresponding to a respective element of the document. The method includes generating a plurality of hash values, with each of the plurality of hash values corresponding to a respective node of the plurality of nodes. The method includes determining the document includes a candidate table based on the plurality of hash values. The method includes generating a textual table based on the candidate table. The method includes analyzing one or more columns of the textual table to determine whether the textual table satisfies one or more criteria. The method includes determining the textual table is the transactions table based on determining the textual table satisfies the one or more criteria.
1 . A method for automatically identifying a transactions table in a document, the method comprising:
obtaining a document object model (DOM) of the document, the DOM comprising a plurality of nodes and a plurality of edges, each of the plurality of nodes corresponding to a respective element of the document;
generating a plurality of hash values, each of the plurality of hash values corresponding to a respective node of the plurality of nodes;
determining the document includes a candidate table based on the plurality of hash values;
generating a textual table based on the candidate table;
analyzing one or more columns of a plurality of columns of the textual table to determine whether the textual table satisfies one or more criteria, wherein the analyzing the one or more columns of the plurality of columns comprises:
providing text from one or more cells of a plurality of cells of a column of the textual table as an input to a machine learning model trained using a supervised learning process to classify text as descriptive of a transaction or not descriptive of a transaction; and
obtaining an output from the machine learning model, the output indicating the text from the one or more cells of the column is descriptive of a transaction; and
determining the textual table is the transactions table based on determining the textual table satisfies the one or more criteria.
2 . The method of claim 1 , wherein:
the plurality of nodes comprises a parent node and a plurality of child nodes that depend directly from the parent node; and
generating the plurality of hash values comprises:
concatenating the plurality of child nodes to the parent node to generate a concatenated node; and
generating a hash value for the concatenated node.
3 . The method of claim 2 , wherein determining the document includes a candidate table based on the plurality of hash values comprises determining the parent node has a threshold number of identical hash children.
4 . The method of claim 1 , wherein analyzing the one or more columns comprises determining whether the one or more columns comprises a timestamp column.
5 . The method of claim 1 , wherein analyzing one or more columns of the plurality of columns comprises:
removing financial symbols from the textual table to generate an updated textual table; and
subsequent to the removing, determining whether the updated textual table includes a column comprising only numerical values.
6 . The method of claim 1 , wherein the machine learning model comprises a large language model (LLM).
7 . The method of claim 1 , further comprising:
prior to the analyzing, filtering the textual table to remove one or more financial symbols included in one or more columns of the textual table.
8 . The method of claim 1 , further comprising:
in response to determining the textual table is the transactions table, extracting one or more transactions included in the transactions table.
9 . The method of claim 1 , wherein the document comprises a web page.
10 . A system for automatically identifying a transactions table in a document, the system comprising:
a memory including computer executable instructions; and
a processor configured to execute the computer executable instructions and cause the system to:
obtain a document object model (DOM) of the document, the DOM comprising a plurality of nodes and a plurality of edges, each of the plurality of nodes corresponding to a respective element of the document;
generate a plurality of hash values, each of the plurality of hash values corresponding to a respective node of the plurality of nodes;
determine the document includes a candidate table based on the plurality of hash values;
generate a textual table based on the candidate table;
analyze one or more columns of a plurality of columns of the textual table to determine whether the textual table satisfies one or more criteria, wherein the analyzing the one or more columns of the plurality of columns comprises:
providing text from one or more cells of a plurality of cells of a column of the textual table as an input to a machine learning model trained using a supervised learning process to classify text as descriptive of a transaction or not descriptive of a transaction; and
obtaining an output from the machine learning model, the output indicating the text from the one or more cells of the column is descriptive of a transaction; and
determine the textual table is the transactions table based on determining the textual table satisfies the one or more criteria.
11 . The system of claim 10 , wherein:
the plurality of nodes comprises a parent node and a plurality of child nodes that depend directly from the parent node; and
to generate the plurality of hash values, the processor is configured to execute the computer executable instructions to cause the system to:
concatenate the plurality of child nodes to the parent node to generate a concatenated node; and
generate a hash value for the concatenated node.
12 . The system of claim 11 , wherein to determine the document includes a candidate table based on the plurality of hash values, the processor is configured to execute the computer executable instructions to cause the system to:
determine the parent node has a threshold number of identical hash children.
13 . The system of claim 10 , wherein to analyze the one or more columns of the plurality of columns, the processor is configured to execute the computer executable instructions to cause the system to determine whether the one or more columns comprises a timestamp column.
14 . The system of claim 10 , wherein to analyze the one or more columns of the plurality of columns, the processor is configured to execute the computer executable instructions to cause the system to:
remove financial symbols from the textual table to generate an updated textual table; and
subsequent to the removing, determining whether the updated textual table includes a column comprising only numerical values.
15 . The system of claim 10 , wherein the machine learning model comprises a large language model (LLM).
16 . The system of claim 10 , wherein the processor is configured to execute the computer executable instructions to cause the system to:
prior to the analyzing, filter the textual table to remove one or more financial symbols included in one or more columns of the textual table.
17 . The system of claim 10 , wherein the processor is configured to execute the computer executable instructions to cause the system to:
in response to determining the textual table is the transactions table, extract one or more transactions included in the transactions table.
18 . A non-transitory computer-readable medium comprising instructions to be executed in a computer system to automatically identify a transactions table in a document, wherein the instructions when executed in the computer system cause the computer system to:
obtain a document object model (DOM) of the document, the DOM comprising a plurality of nodes and a plurality of edges, each of the plurality of nodes corresponding to a respective element of the document;
generate a plurality of hash values, each of the plurality of hash values corresponding to a respective node of the plurality of nodes;
determine the document includes a candidate table based on the plurality of hash values;
generate a textual table based on the candidate table;
analyze one or more columns of a plurality of columns of the textual table to determine whether the textual table satisfies one or more criteria, wherein the analyzing the one or more columns of the plurality of columns comprises:
providing text from one or more cells of a plurality of cells of a column of the textual table as an input to a machine learning model trained using a supervised learning process to classify text as descriptive of a transaction or not descriptive of a transaction; and
obtaining an output from the machine learning model, the output indicating the text from the one or more cells of the column is descriptive of a transaction; and
determine the textual table is the transactions table based on determining the textual table satisfies the one or more criteria.