Method and system for efficient and exhaustive URL categorization
The present method and system relate to categorizing URLs (Uniform Resource Locators) of web pages accessed by multiple users over an IP (Internet Protocol) based data network. The method and system collect real time data from IP data traffic occurring on the IP based data network, and extract parameters from the collected real time data, the parameters including an URL of a web page. The URL is processed by a rule based categorization engine, to associate a matching category to the URL of the web page. When no matching category is inferred, the URL is transferred to a semantic based categorization engine. A matching category is associated to the transferred URL by the semantic based categorization engine, based on a semantic analysis of the textual content extracted from the web page associated to the URL.
1. A method for categorizing URLs (Uniform Resource Locators) of web pages accessed by users over an IP (Internet Protocol) based data network, the method comprising:
collecting by means of at least one monitoring probe real time data from IP data traffic occurring on the IP based data network;
extracting from said collected real time data parameters related to a web page, said parameters including an URL of the web page;
processing said URL with a rule based categorization engine, to associate a matching category to the URL of said web page, the matching category being inferred from a pre-defined list of categories;
when no matching category is inferred, transferring said URL of said web page to a semantic based categorization engine; and
processing said transferred URL by the semantic based categorization engine, said processing consisting in:
extracting textual content from content of said web page associated to said URL,
performing a semantic analysis of said textual content, and
associating a matching category to the transferred URL of the web page based on the semantic analysis of the textual content extracted from the web page, the matching category being inferred from a pre-defined list of categories,
wherein the URLs for which no matching category has been inferred by the rule based categorization engine over a determined period of time are memorized, wherein only the N URLs having the highest occurrence for which no matching category has been inferred by the rule based categorization engine over the determined period of time are transferred to the semantic based categorization engine, and wherein N is a pre-defined number of URLs.
2. The method of claim 1 , wherein the processing of an URL by the rule based categorization engine to associate a matching category to the URL, relies on a set of text based rules defining a match between the URLs and the categories.
3. The method of claim 2 , wherein the rule based categorization engine and the semantic based categorization engine use the same pre-defined list of categories.
4. The method of claim 3 , wherein the matching category associated to a transferred URL by the semantic based categorization engine is transferred back to the rule based categorization engine, and the set of text based rules of the rule based categorization engine is automatically updated in order to acquire the capability to associate said matching category to said transferred URL.
5. The method of claim 1 , wherein additional parameters related to an access to a web page are extracted from the collected real time data; said parameters including a timestamp of occurrence of the access to a web page, and an identifier of the user accessing the web page.
6. The method of claim 5 , wherein an analysis from a Business Intelligence perspective of the matching categories associated to the URLs by the rule based categorization engine and the semantic based categorization engine is performed by an analytic system; said analysis taking into consideration the additional parameters extracted from the collected real time data.
7. The method of claim 6 , wherein metrics are generated by the analytic system; said metrics including: total number of accesses to web pages which URLs have been associated to a specific category, over a determined period of time; total number of unique users accessing web pages which URLs have been associated to a specific category, over a determined period of time.
8. A system for categorizing URLs of web pages accessed by users over an IP based data network, the system comprising:
at least one monitoring probe for collecting real time data from IP data traffic occurring on the IP based data network, and for extracting from said collected real time data parameters related to a web page, said parameters including an URL of the web page;
a computer hardware processor for executing instructions implementing:
a rule based categorization engine for processing said URL, to associate a matching category to the URL of said web page, the matching category being inferred from a pre-defined list of categories;
a semantic based categorization engine for further processing said URL of said web page, when no matching category is inferred by the rule based categorization engine, the further processing consisting in:
extracting textual content from content of said web page associated to said URL,
performing a semantic analysis of said textual content, and
associating a matching category to the URL of the web page, based on the semantic analysis of the textual content extracted from the web page, the matching category being inferred from a pre-defined list of categories,
wherein the URLs for which no matching category has been inferred by the rule based categorization engine over a determined period of time are memorized, wherein only the N URLs having the highest occurrence for which no matching category has been inferred by the rule based categorization engine over the determined period of time are transferred to the semantic based categorization engine, and wherein N is a pre-defined number of URLs.
9. The system of claim 8 , wherein the processing of an URL by the rule based categorization engine, to associate a matching category to the URL, relies on a set of text based rules defining a match between the URLs and the categories.
10. The system of claim 9 , wherein the rule based categorization engine and the semantic based categorization engine use the same pre-defined list of categories.
11. The system of claim 10 , wherein the matching category associated to an URL by the semantic based categorization engine, when no matching category has been inferred by the rule based categorization engine, is used to automatically update the set of text based rules of the rule based categorization engine, in order to acquire the capability to associate said matching category to said URL.
12. The system of claim 8 , wherein the at least one monitoring probe extracts additional parameters related to an access to a web page from the collected real time data; said parameters including a timestamp of occurrence of the access to a web page, and an identifier of the user accessing the web page.
13. The system of claim 12 , wherein an analytic system performs an analysis from a Business Intelligence perspective of the matching categories associated to the URLs by the rule based categorization engine and the semantic based categorization engine; said analysis taking into consideration the additional parameters extracted from the collected real time data.
14. The system of claim 13 , wherein the analytic system generates metrics; said metrics including: total number of accesses to web pages which URLs have been associated to a specific category, over a determined period of time; total number of unique users accessing web pages which URLs have been associated to a specific category, over a determined period of time.