METHOD AND APPARATUS FOR CALCULATING TOPICAL CATEGORIZATION OF ELECTRONIC DOCUMENTS IN A COLLECTION

1 . A computer implemented method for calculating topical categorization of electronic documents in a collection, comprising:

processor application of a metric to categorize semantic distance between two sections of a document or between two documents;

said processor executing a topic algorithm using the categorization provided by said metric to automatically determine topic boundaries;

said processor extracting topics based upon said topic boundaries; and

said processor comparing said extracted topics for similarity with topics in other documents for organizational and research purposes.

2 . The method of claim 1 , said topic algorithm using variational behavior of semantic distances to calculate topical categorization.

3 . The method of claim 1 , said topic algorithm using recursive division and differential adhesion based on semantic distances to calculate topical categorization.

4 . The method of claim 1 , said topic algorithm using both variational behavior of semantic distances, and recursive division and differential adhesion based on semantic distances to calculate topical categorization.

5 . The method of claim 1 , said topic algorithm detecting topic changes within clear breaks indicated by any of sentence breaks, chapter headings and metadata.

6 . The method of claim 1 , said document comprising any data in the form of a sequence of meaningful tokens that can be represented digitally or that can be expressed in the form of such a sequence.

7 . The method of claim 6 , said document comprising any of text, musical passages, choreography, and mathematics.

8 . The method of claim 1 , said topic algorithm determining topic boundaries for any of:

detecting similarity and/or transitions of meaning in passages where an intended meaning is opaque to an analyzer;

detecting similarity and/or transitions and related passages in an unknown script;

analyzing similarity and/or transitions of purported extraterrestrial signals;

detecting similarity and/or transitions in technical and mathematical papers;

providing an element of search or document discovery;

detecting unexpected, more valuable results based on topics;

identifying unexpected correspondences in research;

finding related passages in an unknown script;

supporting social recommendation engines; and

detecting plagiarism.

9 . The method of claim 1 , said topic algorithm supplying chapter and/or heading generation for documents that do not possess chapters and/or headings.

10 . The method of claim 1 , said topic algorithm categorizing said topics in a multidimensional space by a distance determined relative to a canonical set, or are used as generators of a canonical set, of document topics which serve as axes in said multidimensional space.

11 . The method of claim 1 , said topic algorithm stochastically selecting a canonical set of document topics.

12 . The method of claim 1 , further comprising:

applying one or more compression algorithms to compute compression size alone.

13 . The method of claim 12 , said one or more compression algorithms taking into account self-compression overhead of compressing absolutely identical data.

14 . The method of claim 1 , said topic algorithm taking into account a scaling measure that is independent of the size of objects of comparison.

15 . The method of claim 1 , further comprising:

using said topic algorithm to carry out a calculation of topic boundaries pursuant to a document sketch technique.

16 . The method of claim 1 , further comprising:

testing the effectiveness of parameters used by said topic algorithm by variation against a non-sequitur document term of art.

17 . The method of claim 1 , wherein said topic algorithm is independent of a normalized compression metric.

18 . The method of claim 1 , further comprising:

adjusting a computed topic boundary using a topic algorithm in text documents to a nearest sentence or section bound.

19 . The method of claim 1 , further comprising:

using an embodied metric to isolate most typical gist sentences within topics.

20 . The method of claim 1 , further comprising:

creating an orthonormal basis for a multi-dimensional topic space of a linear combination of topics using a random or prescribed topic sample and normalization using orthogonalization schemes comprising any of a stabilized Gram-Schmidt process, Householder transformations, and Givens rotations.

21 . The method of claim 1 , further comprising:

using a Euclidean metric for search and discovery of related topics in a collection of documents.

22 . The method of claim 1 , further comprising:

using cosine distance and, thereafter, a Euclidean metric for search and discovery of related topics in a collection.

23 . The method of claim 1 , further comprising:

using range threshold, within which each component of the coordinate needs to fall, and, thereafter, a Euclidean metric for search and discovery of related topics in a collection.

24 . The method of claim 1 , further comprising:

using topics to determine one of a number of types of significant document transitions;

using significant document transitions to predict user behavior by assigning weights to transition types and/or by calculating a transition matrix of probabilities using said weights; and

constructing cost/benefit models for storing and retrieving sections of documents from large collections of documents.

25 . An apparatus for calculating topical categorization of electronic documents in a collection, comprising:

a processor configured for applying a metric to categorize semantic distance between two sections of a document or between two documents;

said processor configured for executing a topic algorithm using the categorization provided by said metric to automatically determine topic boundaries;

said processor configured for extracting topics based upon said topic boundaries; and

said processor configured for comparing said extracted topics for similarity with topics in other documents for organizational and research purposes.

26 . The apparatus of claim 25 , said topic algorithm using variational behavior of semantic distances to calculate topical categorization.

27 . The apparatus of claim 25 , said topic algorithm using recursive division and differential adhesion based on semantic distances to calculate topical categorization.

28 . The apparatus of claim 25 , said topic algorithm using both variational behavior of semantic distances, and recursive division and differential adhesion based on semantic distances to calculate topical categorization.