Systems and methods for structure and header extraction
The present disclosure is directed towards systems and methods for extracting structure and headers from a body of text. This computational extraction is based on the visual and logical similarities between portions of text. Structure is derived from a programmatic and methodic computation of similarities between header pairs.
1. A system comprising memory and one or more processors communicatively coupled to the memory, the one or more processors configured to:
identify an orthography feature related to orthography characteristics for a respective header from a plurality of headers included in a document, wherein the orthography feature is determined based on capitalization formatting for a string of characters in a data chunk that corresponds to the respective header included in the document;
identify two or more typography features related to typography characteristics for the respective header from the plurality of headers included in the document, wherein the two or more typography features comprise at least (a) a first typography feature configured with a first binary value or a second binary value based on a font setting for at least one character in the data chunk and (b) a second typography feature configured based on a page layout setting for the document;
generate a graph representation of the plurality of headers based at least in part on the orthography feature and the two or more typography features, wherein respective vertices of the graph representation correspond to the respective headers;
compare the graph representation of the plurality of headers to a predefined graph representation to determine a performance metric for the graph representation; and
perform one or more computer-implemented processing tasks with respect to the document based at least in part on the graph representation and the performance metric.
2. The system of claim 1 , wherein the one or more processors are further configured to:
determine a similarity between pairs of headers from the plurality of headers based at least in part on the orthography feature and the two or more typography features; and
transform the graph representation of the plurality of headers into a transformed graph representation of the plurality of headers that segments the headers into groups of one or more similar adjacent headers based on orthography similarities and typography similarities between adjacent headers in the plurality of headers.
3. The system of claim 2 , wherein the one or more processors are further configured to:
match non-adjacent groups of similar adjacent headers based on the groups of the one or more similar adjacent headers associated with the transformed graph representation of the plurality of headers.
4. The system of claim 1 , wherein the two or more typography features further comprise a third typography feature configured with a non-binary value based on the font setting for the at least one character in the data chunk.
5. The system of claim 1 , wherein a path through the graph representation of the plurality of headers represents a sequence of headers included in the document.
6. The system of claim 1 , wherein the graph representation is configured as a tree-structured hierarchy of the plurality of headers included in the document.
7. A method, comprising:
identifying an orthography feature related to orthography characteristics for a respective header from a plurality of headers included in a document, wherein the orthography feature is determined based on capitalization formatting for a string of characters in a data chunk that corresponds to the respective header included in the document;
identifying two or more typography features related to typography characteristics for the respective header from the plurality of headers included in the document, wherein the two or more typography features comprise at least (a) a first typography feature configured with a first binary value or a second binary value based on a font setting for at least one character in the data chunk and (b) a second typography feature configured based on a page layout setting for the document;
generating a graph representation of the plurality of headers based at least in part on the orthography feature and the two or more typography features, wherein respective vertices of the graph representation correspond to the respective headers;
comparing the graph representation of the plurality of headers to a predefined graph representation to determine a performance metric for the graph representation; and
performing one or more computer-implemented processing tasks with respect to the document based at least in part on the graph representation and the performance metric.
8. The method of claim 7 , further comprising:
determining a similarity between pairs of headers from the plurality of headers based at least in part on the orthography feature and the two or more typography features; and
transforming the graph representation of the plurality of headers into a transformed graph representation of the plurality of headers that segments the headers into groups of one or more similar adjacent headers based on orthography similarities and typography similarities between adjacent headers in the plurality of headers.
9. The method of claim 8 , further comprising:
matching non-adjacent groups of similar adjacent headers based on the groups of the one or more similar adjacent headers associated with the transformed graph representation of the plurality of headers.
10. The method of claim 7 , wherein the two or more typography features further comprise a third typography feature configured with a non-binary value based on the font setting for the at least one character in the data chunk.
11. The method of claim 7 , wherein a path through the graph representation of the plurality of headers represents a sequence of headers included in the document.
12. The method of claim 7 , wherein the graph representation is configured as a tree-structured hierarchy of the plurality of headers included in the document.
13. A computer program product, stored on a non-transitory computer readable storage medium, comprising instructions that when executed by one or more processors cause the one or more processors to:
identify an orthography feature related to orthography characteristics for a respective header from a plurality of headers included in a document, wherein the orthography feature is determined based on capitalization formatting for a string of characters in a data chunk that corresponds to the respective header included in the document;
identify two or more typography features related to typography characteristics for the respective header from the plurality of headers included in the document, wherein the two or more typography features comprise at least (a) a first typography feature configured with a first binary value or a second binary value based on a font setting for at least one character in the data chunk and (b) a second typography feature configured based on a page layout setting for the document;
generate a graph representation of the plurality of headers based at least in part on the orthography feature and the two or more typography features, wherein respective vertices of the graph representation correspond to the respective headers;
compare the graph representation of the plurality of headers to a predefined graph representation to determine a performance metric for the graph representation; and
perform one or more computer-implemented processing tasks with respect to the document based at least in part on the graph representation and the performance metric.
14. The computer program product of claim 13 , wherein the one or more processors are further configured to:
determine a similarity between pairs of headers from the plurality of headers based at least in part on the orthography feature and the two or more typography features; and
transform the graph representation of the plurality of headers into a transformed graph representation of the plurality of headers that segments the headers into groups of one or more similar adjacent headers based on orthography similarities and typography similarities between adjacent headers in the plurality of headers.
15. The computer program product of claim 14 , wherein the one or more processors are further configured to:
match non-adjacent groups of similar adjacent headers based on the groups of the one or more similar adjacent headers associated with the transformed graph representation of the plurality of headers.
16. The computer program product of claim 13 , wherein the two or more typography features further comprise a third typography feature configured with a non-binary value based on the font setting for the at least one character in the data chunk.
17. The computer program product of claim 13 , wherein a path through the graph representation of the plurality of headers represents a sequence of headers included in the document.