Granted Patent B2

US 8,489,523 · App. 13/077,696 · Granted Jul 16, 2013

Abstract

Claims (81)

1. A method for categorization using a plurality of categories, comprising:

obtaining, by a hardware processor, a plurality of uniform resource locators (URLs) associated with the plurality of categories;

generating, by the hardware processor, a plurality of classified URLs by labeling the plurality of URLs based on the plurality of categories;

collecting, by the hardware processor, a plurality of web pages identified by the plurality of classified URLs;

generating, by the hardware processor, a plurality of vocabulary terms, each comprising a plurality of words, based on the plurality of web pages;

generating, by the hardware processor, an N-gram file including the plurality of vocabulary terms and comprising a starting word of the plurality of words and a flag corresponding to a cardinality of the plurality of words;

generating, by the hardware processor, a plurality of feature vectors by processing the classified URLs and the plurality of web pages against the N-gram file;

generating, by the hardware processor, a categorization model by applying a machine learning algorithm to the plurality of feature vectors; and

loading a classifier with the categorization module and the N-gram file,

wherein the classifier is configured to select a category of the plurality of categories for an input associated with a user.

2. The method of claim 1 , further comprising:

selecting a vocabulary term of the plurality of vocabulary terms;

identifying a document frequency of the vocabulary term within the plurality of web pages; and

calculating an inverse document frequency (IDF) for the vocabulary term by applying a logarithm function to a ratio based on the plurality of web pages and the document frequency of the vocabulary term,

wherein the N-gram file includes the IDF of the vocabulary term.

3. The method of claim 1 , wherein generating the plurality of feature vectors comprises:

selecting a first vocabulary term and a second vocabulary term of the plurality of vocabulary terms;

identifying a term frequency of the first vocabulary term within a web page of the plurality of web pages;

identifying a term frequency of the second vocabulary term within the web page of the plurality of web pages;

identifying an inverse document frequency (IDF) of the first vocabulary term and an IDF of the second vocabulary term from the N-gram file;

calculating a weight for the first vocabulary term based on the IDF of the first and second vocabulary term and the term frequency of the first and second vocabulary term; and

calculating a weight for the second vocabulary term based on the IDF of the first and second vocabulary term and the term frequency of the first and second vocabulary term.

4. The method of claim 3 , wherein calculating the weight for the first vocabulary term comprises:

calculating a product of the term frequency of the first vocabulary term by the IDF of the first vocabulary term;

calculating a distance of the plurality of vocabulary terms; and

dividing the product by the distance.

5. The method of claim 3 , further comprising:

assigning identification numbers to the first vocabulary term and second vocabulary term.

6. The method of claim 1 , wherein the input is text within a web page accessed by the user, and wherein the advertisement is displayed on the web page.

7. The method of claim 1 , further comprising:

selecting an advertisement corresponding to the category; and

presenting the advertisement to the user.

8. A non-transitory computer readable storage medium storing instructions for categorization using a plurality of categories, the instructions comprising functionality to:

obtain a plurality of uniform resource locators (URLs) associated with the plurality of categories;

generate a plurality of classified URLs by labeling the plurality of URLs based on the plurality of categories;

collect a plurality of web pages identified by the plurality of classified URLs;

generate a plurality of vocabulary terms, each comprising a plurality of words, based on the plurality of web pages;

generate an N-gram file including the plurality of vocabulary terms and comprising a starting word of the plurality of words and a flag corresponding to a cardinality of the plurality of words;

generate a plurality of feature vectors by processing the classified URLs and the plurality of web pages against the N-gram file;

generate a categorization model by applying a machine learning algorithm to the plurality of feature vectors; and

load a classifier with the categorization module and the N-gram file,

wherein the classifier is configured to select a category of the plurality of categories for an input associated with a user.

9. The non-transitory computer readable storage medium of claim 8 , the instructions further comprising functionality to:

select a vocabulary term of the plurality of vocabulary terms;

identify a document frequency of the vocabulary term within the plurality of web pages; and

calculate an inverse document frequency (IDF) for the vocabulary term by applying a logarithm function to a ratio based on the plurality of web pages and the document frequency of the vocabulary term,

wherein the N-gram file includes the IDF of the vocabulary term.

10. The non-transitory computer readable medium of claim 8 , the instructions to generate the plurality of feature vectors comprising functionality to:

select a first vocabulary term and a second vocabulary term of the plurality of vocabulary terms;

identify a term frequency of the first vocabulary term within a web page of the plurality of web pages;

identify a term frequency of the second vocabulary term within the web page of the plurality of web pages;

identify an inverse document frequency (IDF) of the first vocabulary term and an IDF of the second vocabulary term from the N-gram file;

calculate a weight for the first vocabulary term based on the IDF of the first and second vocabulary term and the term frequency of the first and second vocabulary term; and

calculate a weight for the second vocabulary term based on the IDF of the first and second vocabulary term and the term frequency of the first and second vocabulary term.

11. The non-transitory computer readable medium of claim 10 , the instructions to calculate the weight for the first vocabulary term comprising functionality to:

calculate a product of the term frequency of the first vocabulary term by the IDF of the first vocabulary term;

calculate a distance of the plurality of vocabulary terms; and

divide the product by the distance.

12. The non-transitory computer readable medium of claim 10 , the instructions further comprising functionality to:

assign identification numbers to the first vocabulary term and second vocabulary term.

13. The non-transitory computer readable medium of claim 8 , the instructions further comprising functionality to:

select an advertisement corresponding to the category; and

present the advertisement to the user.

14. A system for categorization using a plurality of categories, comprising:

a hardware processor;

a data generation module configured to obtain a plurality of URLs associated with the plurality of categories and collect a plurality of web pages identified by the plurality of uniform resource locators, wherein the plurality of URLs are labeled based on the plurality of categories to generate a plurality of classified URLs;

a vocabulary generator operatively connected to the data collector and configured to generate a plurality of vocabulary terms, each comprising a plurality of words, based on the plurality of web pages;

an N-gram generator operatively connected to the vocabulary generator, and configured to generate an N-gram file including the plurality of vocabulary terms and comprising a starting word of the plurality of words and a flag corresponding to a cardinality of the plurality of words;

a feature vector generator executing on the hardware processor, operatively connected to the N-gram generator, and configured to generate a plurality of feature vectors by processing the plurality of web pages and the plurality of classified URLs against the N-gram file;

a categorization model generator configured to generate a categorization model by applying a machine learning algorithm to the plurality of feature vectors; and

a classifier loaded with the categorization model and the N-gram file and configured to predict a category of the plurality of categories based on an input associated with a user.

15. The system of claim 14 , wherein the feature vector generator is further configured to:

select a first vocabulary term and a second vocabulary term of the plurality of vocabulary terms;

identify a term frequency of the first vocabulary term within a web page of the plurality of web pages;

identify a term frequency of the second vocabulary term within the web page of the plurality of web pages;

identify an inverse document frequency (IDF) of the first vocabulary term and an IDF of the second vocabulary term from the N-gram file;

calculate a weight for the first vocabulary term based on the IDF of the first and second vocabulary term and the term frequency of the first vocabulary term and the second vocabulary term; and

calculate a weight for the second vocabulary term based on the IDF of the first and second vocabulary term and the term frequency of the first and second vocabulary term.

16. The system of claim 15 , wherein the N-gram generator is further configured to:

assign identification numbers to the first vocabulary term and the second vocabulary term.

17. The system of claim 14 , wherein the text is input within a web page accessed by the user.

Assignments (18)