Text categorization method and apparatus
View Patent ↗A method for categorizing a text is disclosed. A weight for at least one term of the text with respect to a category is computed and the determined weight is multiplied in order to obtain a weight of the category for the plurality of terms.
1. A method for categorizing a text comprising a plurality of terms, comprising:
determining a term score of at least one term of said plurality of terms with respect to a category of a taxonomy, which comprises:
computing a similitude ratio between at least two terms; and
accessing a database using the computed similitude ratio to provide the term score of the at least one term with respect to the category, which comprises retrieving from the database at least one category together with a coefficient for the at least one term, wherein the term score of the at least one term with respect to a category is computed using at least the coefficient of the at least one term and the similitude ratio of the at least one term with respect to the category, and
computing the product of said determined term score of the at least one term of said plurality of terms with respect to the category in order to obtain a category score for the plurality of terms.
2. The method as claimed in claim 1 , wherein the computing of the term score of the at least one term with respect to a category comprises computing a degree of ambiguity of the at least one term, computing a degree of association between the at least one term and a category and multiplying the computed degree of association with the computed degree of ambiguity to provide the term score of the at least one term with respect to the category.
3. The method as claimed in claim 2 , wherein the term is a simple information unit, and wherein the degree of ambiguity of the term is equal to
(
1
+
n
m
(
log
10
(
10
+
NC
)
)
2
)
,
where nm is a coefficient of importance of said term and NC is the total number of categories associated to said term.
4. The method as claimed in claim 2 , wherein the term is a complex information unit, and wherein the degree of ambiguity of the term is equal to
(
1
+
n
m
(
log
10
(
10
+
NC
)
)
2
)
,
where nm is a coefficient of importance of said term and NC is the total number of categories associated to said term.
5. The method as claimed in claim 2 , wherein the term is a simple information unit, and wherein said degree of association is equal to
(
∑
i
=
1
n
CFi
+
CF
∑
i
=
1
n
CFi
)
a
,
where CFi (i=1, . . . , n) relates to frequencies of the term with respect to n categories to which it is associated and wherein a is a coefficient.
6. The method as claimed in claim 5 , wherein a is equal to 1.
7. The method as claimed in claim 2 , wherein the term is a complex information unit, and wherein said degree of association is equal to
(
1
+
CF
NTC
)
b
,
where CF relates to frequencies of the term with respect to said category, where b is a coefficient and where NTC is the number of texts associated to said category.
8. The method as claimed in claim 7 , wherein b is equal to 2.
9. The method as claimed in claim 1 , wherein the determining of a term score of at least one term of the plurality of terms is performed with respect to at least one category of a plurality of categories, and wherein the product of said determined term score of said plurality of terms with respect to the category is performed in order to obtain a plurality of category scores, each of the plurality of term scores of the at least one term being related to a category of the plurality of categories, other comprising filtering the plurality of category scores wherein:
the filtering of the plurality of category scores for a plurality of terms comprises selecting at least one category of the plurality of categories according to a dynamic filtering threshold; and
the text comprises a plurality of paragraphs, wherein the filtering of the plurality of category scores is performed for each paragraph of the plurality of paragraphs, further comprising filtering the filtered plurality of category scores for each paragraph to provide a filtered plurality of text category scores, which comprises normalizing the filtered plurality of category scores according to statistical data related to each paragraph.
10. The method as claimed in claim 9 , wherein the filtering of the filtered plurality of text category scores is performed using a z-filter.
11. The method as claimed in claim 9 , wherein the filtering of the filtered plurality of text category scores is performed using a rule-based filter.
12. The method as claimed in claim 11 , wherein the rule-based filter rejects a category having a high text category score only in one or in a few paragraphs of the text.
13. The method as claimed in claim 11 , wherein the rule-based filter checks if a first category is compatible with a second category and rejects the first category in the case where the first category is not compatible with the second category.
14. The method as claimed in claim 11 , wherein the rule-based filter operates according to a hierarchy in a taxonomy.