Method and apparatus for detecting web-based electronic mail in network traffic
View Patent ↗Method and apparatus for detecting web-based electronic mail in network traffic is described. In some examples, web pages are extracted from the network traffic. Fields in each page of a group of the web pages that share a documents structure are identified. A statistical analysis of the fields of each page in the group of web pages is performed to identify any electronic mail (e-mail) fields. The group of web pages is indicated to include web-based e-mail messages if the fields of each page in the group of web pages include at least one e-mail field.
1. A method of processing network traffic, comprising:
extracting web pages from network traffic;
identifying fields in each page of a group of the web pages that share a document structure, wherein the step of identifying comprises:
executing a document clustering algorithm using the web pages as parametric input to establish the group of web pages;
executing a structure extraction algorithm configured to:
identify structural portions of the pages in the group of web pages;
determine at least one common structural portion in the structural portions as being shared across all pages in the group of web pages, the at least one common structural portion comprising the document structure; and
determine each of the structural portions other than the at least one structural portion to be the fields;
performing a statistical analysis of the fields of each page in the group of web pages to identify any electronic mail (e-mail) fields; and
indicating the group of web pages to include web-based e-mail messages if the fields of each page in the group of web pages include at least one e-mail field.
2. The method of claim 1 , wherein the step of determining the at least one common structural portion comprises:
selecting two pages of the group of web pages as a set;
iteratively determining common text between all pages in the set and adding at least one additional page of the group of web pages to the set until the common text does not change between two successive iterations; and
identifying the common text between all pages in the set as the at least one common structural portion.
3. The method of claim 1 , wherein the step of performing the statistical analysis comprises:
analyzing each of the fields across all of the pages in the group of web pages against at least one statistical requirement.
4. The method of claim 3 , wherein the step of performing the statistical analysis further comprises:
analyzing at least one of the fields across all of the pages in the group of web pages against at least one regular expression requirement.
5. The method of claim 1 , further comprising:
extracting text content from the fields of each page in the group of web pages if the group of web pages is indicated as including web-based e-mail messages.
6. An apparatus for processing network traffic, comprising:
one or more computer processors communicatively coupled to a network wherein the one or more computer processors are configured to:
extract web pages from the network traffic;
identify fields in each page of a group of the web pages that share a document structure, wherein the step of identifying comprises:
executing a document clustering algorithm using the web pages as parametric input to establish the group of web pages;
executing a structure extraction algorithm configured to:
identify structural portions of the pages in the group of web pages;
determine at least one common structural portion in the structural portions as being shared across all pages in the group of web pages, the at least one common structural portion comprising the document structure; and
determine each of the structural portions other than the at least one structural portion to be the fields;
perform a statistical analysis of the fields of each page in the group of web pages to identify any electronic mail (e-mail) fields; and
indicat the group of web pages to include web-based e-mail messages if the fields of each page in the group of web pages include at least one e-mail field.
7. The apparatus of claim 6 , wherein the determining the at least one common structural portion comprises:
selecting two pages of the group of web pages as a set;
iteratively determining common text between all pages in the set and adding at least one additional page of the group of web pages to the set until the common text does not change between two successive iterations; and
identifying the common text between all pages in the set as the at least one common structural portion.
8. The apparatus of claim 6 , wherein the one or more computer processors configured to perform the statistical analysis comprises:
analyzing each of the fields across all of the pages in the group of web pages against at least one statistical requirement.
9. The apparatus of claim 8 , wherein the one or more computer processors configured to perform the statistical analysis further comprises:
analyzing at least one of the fields across all of the pages in the group of web pages against at least one regular expression requirement.
10. The apparatus of claim 6 , wherein the one or more computer processors are configured to:
extract text content from the fields of each page in the group of web pages if the group of web pages is indicated as including web-based e-mail messages.
11. A computer readable medium having instructions stored thereon that when executed by a processor cause the processor to perform a method of processing network traffic, comprising:
extracting web pages from the network traffic;
identifying fields in each page of a group of the web pages that share a document structure, wherein the step of identifying comprises:
executing a document clustering algorithm using the web pages as parametric input to establish the group of web pages;
executing a structure extraction algorithm configured to:
identify structural portions of the pages in the group of web pages;
determine at least one common structural portion in the structural portions as being shared across all pages in the group of web pages, the at least one common structural portion comprising the document structure; and
determine each of the structural portions other than the at least one structural portion to be the fields;
performing a statistical analysis of the fields of each page in the group of web pages to identify any electronic mail (e-mail) fields; and
indicating the group of web pages to include web-based e-mail messages if the fields of each page in the group of web pages include at least one e-mail field.
12. The computer readable medium of claim 11 , wherein the step of determining the at least one common structural portion comprises:
selecting two pages of the group of web pages as a set;
iteratively determining common text between all pages in the set and adding at least one additional page of the group of web pages to the set until the common text does not change between two successive iterations; and
identifying the common text between all pages in the set as the at least one common structural portion.
13. The computer readable medium of claim 11 , wherein the step of performing the statistical analysis comprises:
analyzing each of the fields across all of the pages in the group of web pages against at least one statistical requirement.
14. The computer readable medium of claim 13 , wherein the step of performing the statistical analysis further comprises:
analyzing at least one of the fields across all of the pages in the group of web pages against at least one regular expression requirement.