Method to identify common structures in formatted text documents
A computer implemented method, computer program product and data processing system, for identifying common structures shared across a plurality of formatted text documents. The common structure is presented as a sequence of landmarks, each of which has a starting and ending marker to describe the borders of text. The common structure is identified by counting the occurrences of repeating text segments across documents. Frequently co-occurred adjacent segments become candidates for markers of landmarks. In addition, styling information of textual content within a landmark is extracted and mapped to rules. The rules are used to merge and summarize content from multiple documents, which gives an advantage over current practice of content concatenation.
1. A computerized method to identify a common structure from a collection of formatted text documents, said method comprising:
creating a two dimensional array to record an occurrence of text segments in the formatted documents, using a processor on a computer;
sequentially retrieving documents from said collection of formatted documents;
parsing each said retrieved document, using said processor, into text segments according to a segmentation scheme and record scheme attributes of a format used in said formatted documents;
entering each occurrence of said text segments in said retrieved documents into said two dimensional array;
selecting common text segments across a majority of the documents;
creating a one dimensional array and recording therein frequencies of adjacent common segment pairs across the documents;
selecting high frequency pairs as starting and ending markers of landmarks; and
providing, as an output, a sequence of said landmarks as comprising a common structure of said collection of formatted text documents.
2. The method of claim 1 , wherein a segmentation scheme is defined as rules to identify character sequences for segment separators in the formatted text document.
3. The method of claim 1 , wherein scheme attributes comprise styling information.
4. The method of claim 3 , wherein said styling information comprises attributes including heading and fonts types including one or more of bold, italic, and underline.
5. The method of claim 1 , wherein scheme attributes include level references if the document is organized as a hierarchy of text nodes at multiple levels.
6. A non-transitory storage medium tangibly embodying a set of computer instructions that execute the method of claim 1 .
7. The method of claim 1 , wherein each said landmark is assigned its starting and ending markers based on one or more of specific text strings, symbols, and text styling.
8. The method of claim 1 , wherein said landmarks derived from said formatted documents potentially comprise template subject headings having associated therewith a text field to be recognized and filled in by a user using a template.
9. The method of claim 1 , wherein said segmentation scheme defines boundaries between text segments in a formatted text document.
10. The method of claim 9 , wherein said boundaries between text segments comprise one or more of at least: paragraphs; empty lines; table cells; and other semantically meaningful separators used in said formatted text documents.
11. The method of claim 1 , further comprising recording scheme attributes for each segmented text, said scheme attributes comprising presentation formatting instructions for semantic interpretation.
12. The method of claim 11 , wherein said scheme attributes comprise one or more of at least an italic font; a bold font; a bullet format; a numbered format; a heading format; and a table format.
13. The method of claim 12 , wherein said documents are hierarchical and a scheme attribute comprises a path from a root node of a hierarchy of a document to a current text segment.
14. The method of claim 1 , wherein said common structure of said collection of formatted text documents potentially comprises a project-based template of at least a portion of said documents, said project-based template comprising a template that has evolved over time for a project.
15. The method of claim 1 , further comprising initially receiving a user input that defines the text segmentation scheme, dependent upon a text document formatting of said documents.
16. The method of claim 1 , as comprising an automated cleansing tool used to streamline a process of deriving a common template as cleansed documents from said collection of formatted text documents.
17. An apparatus, comprising:
a processor executing a set of machine-readable instructions to identify a common structure from a collection of formatted text documents; and
an interface with a memory storing said collection of formatted text documents,
wherein said machine-readable instructions cause said processor to:
create a two dimensional array to record an occurrence of text segments in the formatted documents;
sequentially retrieve documents from said collection of formatted documents from said memory;
parse each said retrieved document into text segments according to a segmentation scheme and record scheme attributes of a format used in said formatted documents;
enter each occurrence of said text segments in said retrieved documents into said two dimensional array;
select common text segments across a majority of the documents;
create a one dimensional array and record therein frequencies of adjacent common segment pairs across the documents;
select high frequency pairs as starting and ending markers of landmarks; and
provide as an output a sequence of said landmarks as being a common structure of said collection of formatted text documents.
18. The apparatus of claim 17 , wherein a segmentation scheme is defined as rules to identify character sequences for segment separators in the formatted text document.
19. The apparatus of claim 17 , wherein scheme attributes comprise styling information including attributes of heading and fonts.
20. The apparatus of claim 17 , wherein scheme attributes include level references if the document is organized as a hierarchy of text nodes at multiple levels.