Method and system for classifying text
View Patent ↗A content classification system, method and computer product is presented. In connection with the invention, a data structure is created by identifying a plurality of words and mapping each word to one or more categories. The data structure is indexed. An item of content is identified and classified based on the data structure. The classification includes identifying all one—or more—word combinations in the item of content; for each word of at least a pre-determined number of characters in length in each of the word combinations, identifying each of the categories to which it is mapped; and determining a weight for each of the words based on an inverse proportion to the number of categories to which it is mapped.
1. A method comprising:
using a programmed computer,
creating a data structure by identifying a plurality of words and mapping each word to one or more categories;
storing the data structure in one or more databases;
indexing the data structure;
identifying an item of electronic content;
classifying the item of electronic content using the data structure, the classifying comprising:
identifying all single words and word combinations comprising two or more words in the item of electronic content that include one or more words;
for each of the single words of at least a pre-determined number of characters in length and each of the words in the word combinations words of at least a pre-determined number of characters in length in each of the word combinations, identifying each of the categories to which the word is mapped;
assigning a weight for each of the words based on an inverse proportion to the number of categories to which the word is mapped, and
assigning a weight based on a direct proportion to a number of words in the word combination using a multiplier; and
adding a result of classifying the electronic content to the data structure.
2. The method of claim 1 wherein the pre-determined number of characters is three.
3. The method of claim 1 wherein the programmed computer further:
groups one or more of the categories into one or more channels.
4. The method of claim 3 wherein determining the weight further comprises assigning an additional weight based on the one or more channels associated with the one or more categories to which the word is mapped.
5. The method of claim 1 wherein the programmed computer further:
assigns a value to the weight based on a relative relatedness between words using a semantic distance measure.
6. The method of claim 1 wherein the index is derived from existing structures configured in accordance with cognitive human notions of relatedness.
7. The method of claim 1 wherein the index is generated based on counts of categories and/or keywords in a page of electronic content.
8. The method of claim 1 wherein the item of electronic content comprises multilingual content and the classification is performed without translating the content.
9. The method of claim 1 wherein electronic content is discovered using one or more crawling engines.
10. The method of claim 1 wherein the item of electronic content comprises a web page and the classifying is performed by analyzing only a URL of the web page.
11. The method of claim 1 wherein the item of electronic content comprises a web page and wherein the classification is performed on both the web page as rendered and a language used to create the web page.
12. A system comprising:
one or more processors that are programmed to:
create a data structure by identifying a plurality of words and mapping each word to one or more categories;
store the data structure in one or more databases;
index the data structure;
identify an item of electronic content;
classify the item of electronic content using the data structure by identifying all single words and word combinations comprising two or more words in the item of electronic content; for each of the single words of at least a pre-determined number of characters in length and each of the words in the word combinations of at least a pre-determined number of characters in length, identifying each of the categories to which the word is mapped; assigning a weight for each of the words based on an inverse proportion to the number of categories to which the word is mapped; assigning a weight based on a direct proportion to a number of words in the word combination using a multiplier; and
adding a result of classifying the electronic content to the data structure.
13. The system of claim 12 wherein the pre-determined number of characters is three.
14. The system of claim 12 wherein the one or more processors are further programmed to group one or more of the categories into one or more channels.
15. The system of claim 14 wherein determining the weight further comprises assigning an additional weight based on the one or more channels associated with the one or more categories to which the word is mapped.
16. The system of claim 12 wherein the one or more processors are further programmed to assign a value to the weight based on a relative relatedness between words using a semantic distance measure.
17. The system of claim 12 wherein the index is derived from existing structures configured in accordance with cognitive human notions of relatedness.
18. The system of claim 12 wherein the index is generated based on counts of categories and/or keywords in a page of electronic content.
19. The system of claim 12 wherein the item of electronic content comprises multilingual content and the classification is performed without translating the content.
20. The system of claim 12 wherein one or more crawling engines are used to discover content.
21. The system of claim 12 wherein the item of electronic content comprises a web page and the classifying is performed by analyzing only a URL of the web page.
22. The system of claim 12 wherein the item of electronic content comprises a web page and wherein the classification is performed on both the web page as rendered and a language used to create the web page.
23. A non-transitory computer readable medium having stored thereon computer executable instructions that, when executed by a computer, direct the computer to perform a method comprising the steps of:
creating a data structure by identifying a plurality of words and mapping each word to one or more categories;
storing the data structure in one or more databases;
indexing the data structure;
identifying an item of electronic content;
classifying the item of electronic content using the data structure, the classifying comprising:
identifying all single words and word combinations comprising two or more words in the item of electronic content;
for each of the single words of at least a pre-determined number of characters in length and each of the words in the word combinations of at least a pre-determined number of characters in length, identifying each of the categories to which the word is mapped;
assigning a weight for each of the words based on an inverse proportion to the number of categories to which the word is mapped; and
assigning a weight based on a direct proportion to a number of words in the word combination using a multiplier; and
adding a result of classifying the electronic content to the data structure.
24. The non-transitory computer readable medium of claim 23 wherein the pre-determined number of characters is three.
25. The non-transitory computer readable medium of claim 23 , the method further comprising:
grouping one or more of the categories into one or more channels.
26. The non-transitory computer readable medium of claim 25 wherein determining the weight further comprises assigning an additional weight based on the one or more channels associated with the one or more categories to which the word is mapped.
27. The non-transitory computer readable medium of claim 23 , the method further comprising:
assigning a value to the weight based on a relative relatedness between words using a semantic distance measure.
28. The non-transitory computer readable medium of claim 23 wherein the index is derived from existing structures configured in accordance with cognitive human notions of relatedness.
29. The non-transitory computer readable medium of claim 23 wherein the index is generated based on counts of categories and/or keywords in a page of electronic content.
30. The non-transitory computer readable medium of claim 23 wherein the item of electronic content comprises multilingual content and the classification is performed without translating the content.
31. The non-transitory computer readable medium of claim 23 wherein electronic content is discovered using one or more crawling engines.
32. The non-transitory computer readable medium of claim 23 wherein the item of electronic content comprises a web page and the classifying is performed by analyzing only a URL of the web page.
33. The non-transitory computer readable medium of claim 23 wherein the item of electronic content comprises a web page and wherein the classification is performed on both the web page as rendered and a language used to create the web page.