Document clustering based on cohesive terms
View Patent ↗A method and a storage medium, that includes instructions for causing a computer to implement the method, for document categorization is presented. The method includes identifying terms occurring in a collection of documents, and determining a cohesion score for each of the terms. The cohesion score is a function of a cosine difference between each of the documents containing the term and a centroid of all the documents containing the term. The method further includes sorting the terms based on the cohesion scores. The method also includes creating categories based on the cohesion scores of the terms, wherein each of the categories includes only documents (i) containing a selected one of the terms and (ii) that have not already been assigned to a category. The method still further includes moving each of the documents to a category of a nearest centroid, thereby refining the categories.
1. A program storage device encoded with machine-readable computer program code for document categorization, the storage device including instructions for causing a computer to implement a method comprising:
identifying with the computer terms occurring in a collection of documents;
determining a cohesion score for each of the terms, the cohesion score is a function of a cosine difference between each of the documents containing the term and a centroid of all the documents containing the term;
sorting the terms based on the cohesion scores;
creating categories based on the cohesion scores of the terms, wherein each of the categories includes only documents (i) containing a selected one of the terms and (ii) that have not already been assigned to a category; and
moving each of the documents to a category of a nearest centroid, thereby refining the categories;
wherein creating the categories, comprises:
(a) selecting the term with the highest of the cohesion scores that has not been selected;
(b) if a number of the documents that are uncategorized contain the term selected as having the highest of the cohesion scores is greater than a minimum category size, creating another category consisting of all documents that contain the term selected as having the highest of the cohesion scores when the number of documents that are uncategorized contain the term selected as having the highest of the cohesion scores is greater than a minimum category size;
(c) ceasing creating the categories when the number of the documents that are uncategorized is less than an uncategorized document size or when the number of the other categories created is greater than a maximum number of categories; and
(d) repeating steps (a)-(c) until step (c) is performed or none of the terms remain when, at least one of, (i) the number of the documents that are uncategorized is not less than the uncategorized document size and (ii) the number of the other categories created is not greater than maximum number of categories.
2. The storage device of claim 1 , wherein the method further comprises:
normalizing the cohesion scores.
3. The storage device of claim 1 , wherein the method further comprises:
re-categorizing any of the documents belonging to more than one the categories.
4. The storage device of claim 1 , wherein the sorting the terms comprises sorting the terms in a decreasing order of the cohesion scores.
5. The storage device of claim 1 , wherein the identifying the terms, comprises:
identifying the terms from a dictionary by, representing each of the documents as a vector of weighted frequencies of the terms, and
using a term-weighting scheme to emphasizes the terms with a high frequency in the document, and
normalizing each the vectors to have a unit Euclidean norm.
6. The storage device of claim 1 , wherein the cohesion score is given by:
cohesion
(
T
,
n
)
=
∑
x
∈
T
cos
(
centroid
(
T
)
,
x
)
T
n
where T is the documents and n is a normalization parameter.