Retrieval augmentation system for unstructured tabular documents
A system for extracting a number of data elements from one or more data sources. A text representing tables using a markdown language is extracted from spreadsheets and or other grid-based documents. The text is provided to a language model with a prompt. The prompt may be a chain-of-thoughts prompt. The prompt includes several requests and/or steps that cause the language model to extract one or more tables from the spreadsheet and output the tables using the markdown language or a different markdown language. The tables extracted from the spreadsheet are converted into table chunks and indexed for retrieval by a retrieval augmented architecture. When a prompt to extract particular information from the spreadsheet is provided, one or more relevant table chunks are identified and provided to a language model for extraction. Using the language model to separate tables improves information extraction accuracy while maintaining downstream instructions.
1 . A method for information extraction using a retrieval augmentation architecture, the method comprising:
receiving, by one or more processors, a document comprising a plurality of tables;
prompting, by the one or more processors, a first language model with a request to identify one or more individual tables in the document and extract metadata from the one or more individual tables;
identifying, by the one or more processors, a relevant table of the one or more individual tables the first language model by performing a keyword search of the metadata extracted for the one or more individual tables, the keyword search related to the information to be extracted by an extraction prompt; and
storing a response from the first language model or a second language model to the extraction prompt and the relevant table.
2 . The method of claim 1 , further comprising generating, by the one or more processors, a vector embedding for each of the one or more individual tables based on the metadata extracted using a text embedding model,
wherein identifying the relevant table comprises calculating a distance between the vector embedding for each of the one or more individual tables and a prompt embedding generated for the extraction prompt using the text embedding model and generating a score based on the distance and results of the keyword search.
3 . The method of claim 1 , further comprising generating, by the one or more processors, a markdown representation of the plurality of tables, wherein the request to identify the one or more individual tables in the document and extract the metadata includes the markdown representation.
4 . The method of claim 1 , wherein the metadata comprises at least one of column headers, footnotes, titles, or units related to the one or more individual tables.
5 . The method of claim 1 , wherein the first language model is a multi-modal language model configured to accept an image-based representation of the document and the request to identify the one or more individual tables in the document and extract the metadata comprises a step to identify boundaries of the one or more individual tables.
6 . The method of claim 5 , wherein:
the request to identify the one or more individual tables in the document and extract the metadata is a first request; and
the method further comprises prompting, by the one or more processors, the first language model with a second request to generate a markdown representation of the one or more individual tables.
7 . The method of claim 1 , wherein:
the request to identify the one or more individual tables in the document and extract the metadata is a first request; and
the method further comprises prompting, by the one or more processors, the first language model with a second request to convert data of at least a table of the one or more individual tables to a first unit system from a second unit system in the metadata.
8 . The method of claim 1 , wherein the request to identify the one or more individual tables in the document and extract the metadata is a chain-of-thoughts prompt, the chain-of-thoughts prompt comprising:
a first step to quantify a number of the plurality of tables in the document;
a second step to identify the one or more individual tables from the plurality of tables, wherein a quantity of the one or more individual tables is equal to the number quantified in the first step; and
a third step to extract the metadata of the one or more individual tables.
9 . A method for information extraction using a retrieval augmentation architecture, the method comprising:
receiving, by one or more processors, a document comprising a plurality of tables;
prompting, by the one or more processors, a first language model with a request to identify one or more individual tables in the document and extract metadata from the one or more individual tables;
identifying, by the one or more processors, a relevant table of the one or more individual tables based on a comparison between a vector embedding for an extraction prompt for the first language model or a second language model and a corresponding vector embedding based on the metadata for each of the one or more individual tables; and
storing a response from the first language model or the second language model to the extraction prompt and the relevant table.
10 . The method of claim 9 , further comprising performing a keyword search of the metadata extracted for the one or more individual tables using keywords related to the information to be extracted by the extraction prompt, wherein identifying the relevant table comprises calculating a distance between the vector embedding for the extraction prompt and the corresponding vector embedding for each of the one or more individual tables and generating a score based on the distance and results of the keyword search.
11 . The method of claim 9 , further comprising generating, by the one or more processors, a markdown representation of the plurality of tables, wherein the request to identify the one or more individual tables in the document and extract the metadata includes the markdown representation.
12 . The method of claim 9 , wherein the metadata comprises at least one of column headers, footnotes, titles, or units related to the one or more individual tables.
13 . The method of claim 9 , wherein:
the first language model is a multi-modal language model configured to accept an image-based representation of the document and the request to identify the one or more individual tables in the document and extract the metadata is a first request;
the request to identify the one or more individual tables in the document and extract the metadata comprises a step to identify boundaries of the one or more individual tables; and
the method further comprises prompting, by the one or more processors, the first language model with a second request to generate a markdown representation of the one or more individual tables.
14 . The method of claim 9 , wherein:
the request to identify the one or more individual tables in the document and extract the metadata is a first request; and
the method further comprises prompting, by the one or more processors, the first language model with a second request to convert data of at least a table of the one or more individual tables to a first unit system from a second unit system in the metadata.
15 . A method for information extraction using a retrieval augmentation architecture, the method comprising:
receiving, by one or more processors, a document comprising a plurality of tables;
prompting, by the one or more processors, a first language model with a chain-of-thoughts prompt comprising:
a first step having a first request to determine a quantity of the plurality of tables in the document;
a second step having a second request to identify the quantity of individual tables from the plurality of tables;
a third step having a third request to extract metadata of at least an identified table of the quantity of the individual tables; and
a fourth step having a fourth request to reconstruct the identified table using a markdown language;
identifying, by the one or more processors, a relevant table of the individual tables based on a search criterion related to an extraction prompt for the first language model or a second language model; and
storing a response from the first language model or the second language model to the extraction prompt and the relevant table.
16 . The method of claim 15 , further comprising generating, by the one or more processors, a vector embedding for each of the quantity of individual tables based on the metadata extracted using a text embedding model,
wherein identifying the relevant table comprises calculating a distance between the vector embedding for each of the quantity of individual tables and a prompt embedding generated for the extraction prompt using the text embedding model.
17 . The method of claim 15 , wherein identifying the relevant table comprises performing a keyword search of the metadata extracted for the quantity of individual tables using keywords related to the information to be extracted by the extraction prompt.
18 . The method of claim 15 , wherein the first language model is a multi-modal language model configured to accept an image-based representation of the document and the chain-of-thoughts prompt comprises a fifth step to identify boundaries of the quantity of individual tables.
19 . The method of claim 15 , wherein the metadata comprises at least one of column headers, footnotes, titles, or units related to the quantity of individual tables.
20 . The method of claim 15 , wherein the chain-of-thoughts prompt comprises a fifth request to convert data of at least a table of the quantity of individual tables to a first unit system from a second unit system in the metadata.