Computer-based system and method for generating, classifying, searching, and analyzing standardized text templates and deviations from standardized text templates

1. A method for representing similar text documents and similar segments of text documents as standardized text templates and deviations from standardized text templates, the method comprising:

partitioning documents into segments;

generating standardized text segment templates from segments of documents;

determining the deviations of individual text segments from the standardized text segment templates so generated;

representing the individual text segments as the combination of standardized text segment templates and deviations from standardized text segment templates;

generating standardized text templates representing text documents as sequences of standardized text segment templates;

determining the deviations of the individual text documents from the standardized text templates so generated; and

representing the individual text documents as a combination of the standardized text templates and the deviations from the standardized text templates.

2. The method of claim 1 , where the documents are partitioned into segments corresponding to paragraphs.

3. The method of claim 1 , where the standardized text template generated is the document or portion of a document with the least average edit distance with respect to each other text document in a family of similar text documents.

4. The method of claim 1 , further comprising:

identifying elements or segments of the generated template that frequently vary with respect to the other similar text documents; and

coding those elements or segments as “blanks.”

5. The method of claim 1 , where the documents are of the file types used in a computer word processor.

6. The method of claim 1 , where the similar text documents are business or legal documents.

7. The method of claim 1 , further comprising:

clustering text documents into families of similar text inputs or portions of text inputs;

comparing each text input in the family to each other text input in the family; and

selecting the text input with the least average distance from all other text inputs in the family as the standardized text template.

8. The method of claim 7 , where the documents are partitioned into segments corresponding to paragraphs.

9. The method of claim 7 , where the template selected is the document or portion of a document with the least average edit distance with respect to each other text document in the family of similar text documents.

10. The method of claim 7 , further comprising:

identifying elements or segments of the generated template that frequently vary with respect to other matching documents; and

coding those elements or segments as “blanks.”

11. The method of claim 7 , where the documents are of the file types used in a computer word processor.

12. Computer readable media having computer readable program code thereon which, when loaded into and executed by a computer, performs the steps of:

partitioning documents into segments;

generating a standardized text segment templates from segments of documents;

determining the deviations of individual text segments from the standardized text templates so generated;

representing the individual text segments as the combination of standardized text segment templates and the deviations from the standardized text segment templates;

generating a standardized text templates representing text documents as sequences of standardized text segment templates;

determining the deviations of individual text documents from the standardized text templates so generated; and

representing the individual text documents as a combination of the standardized text templates and the deviations from the standardized text templates.