Systems and methods for structure and header extraction
The present disclosure is directed towards systems and methods for extracting structure and headers from a body of text. This computational extraction is based on the visual and logical similarities between portions of text. Boilerplate is removed from chunks of text making up potential headers and the cleaned result is compared against other potential headers and the remainder of the body of text.
1. A method for extracting headers, comprising:
receiving an input body of text containing a plurality of chunks of text;
identifying a set of features of each chunk;
classifying each text chunk as a potential header depending on whether the chunk includes a mark or title text;
identifying any boilerplate in each potential header and removing identified boilerplate in each potential header, forming a remainder of each potential header by removing the identified boilerplate; and
comparing the potential header remainders to each other and to a remainder of the input body of text not included in the potential header remainders to confirm whether each potential header is a header, wherein
the comparing step includes determining a similarity among all of the potential header remainders based on comparing an average number of characters in a group of potential headers with similar features to a threshold and to a number of character edits required to transform each potential header in the group into a subsequent potential header in the group.
2. The method of claim 1 , wherein the features include typography characteristics.
3. The method of claim 2 , wherein the features include at least two or more of font family, font size, italic, bold, underline, space above, space left, space left first line, and justification.
4. The method of claim 1 , wherein the features include orthography characteristics.
5. The method of claim 1 , wherein the features include page layout.
6. The method of claim 1 , wherein the features include at least two or more of typography characteristics, orthography characteristics and page layout.
7. The method of claim 1 further comprising determining if a chunk includes title text by at least comparing features of the chunk to features of a remainder of the input body of text and identifying title text if its features differ from those of a majority of the remainder.
8. The method of claim 1 , wherein the comparison of potential header remainders includes comparing the number of characters included in the potential header remainders and chunks of text in the input body of text covered by the potential header remainders to a total number of characters in the input body of text.
9. The method of claim 1 , wherein the comparison of potential header remainders includes discounting groups of similar cleaned potential headers based on an average number of characters among the potential header remainders.
10. The method of claim 1 wherein identifying boilerplate includes comparing potential headers to a set of one or more predetermined non-boilerplate words.
11. The method of claim 1 wherein identifying boilerplate includes identifying sequences of potential headers that cross other sequences of potential headers when analyzed in reading order.
12. The method of claim 1 wherein boilerplate is removed from a remainder of a potential header within a single potential header.
13. The method of claim 1 wherein identifying any boilerplate in each potential header includes analyzing formatting and layout information with which a potential header is annotated.