Discerning web content and services based on real-time DNS tagging

1. A method for profiling network traffic of a network, comprising:

obtaining, by a processor of a computer system and from the network traffic, a plurality of domain name service (DNS) flows comprising a plurality of client Internet protocol (IP) addresses, a plurality of fully qualified domain names (FQDNs), and a plurality of server IP addresses,

wherein the plurality of DNS flows comprise a DNS query that is initiated from a client IP address of the plurality of client IP addresses to request service based on a FQDN of the plurality of FQDNs that is contained in the DNS query, and

wherein the plurality of server IP addresses comprise a server IP address that is resolved by a DNS server in the network based on the FQDN as requested from the client IP address and returned in a DNS response contained in the plurality of DNS flows;

analyzing, by the processor, the plurality of DNS flows to generate a mapping between the plurality of client IP addresses, the plurality of server IP addresses, and the plurality of FQDNs;

storing the mapping in a data structure separate from the DNS server, wherein each of the plurality of client IP addresses and each of the plurality of server IP addresses form a client/server IP address pair that is used as input to the data structure for looking up one of the plurality of FQDNs as output of the data structure; and

analyzing the network traffic based on the data structure to generate a result of profiling the network traffic, wherein analyzing the network traffic based on the data structure comprises:

obtaining, from the network traffic, a plurality of flows comprising a first portion of the plurality of client IP addresses, a second portion of the plurality of server IP addresses, and a plurality of server ports;

tagging the plurality of flows using corresponding FQDNs that are looked up using the data structure based on the first portion of the plurality of client IP addresses and the second portion of the plurality of server IP addresses contained in the plurality of flows;

storing the plurality of flows in association with the corresponding FQDNs in a repository as a plurality of tagged flows; and

tallying a plurality of portions of the plurality of tagged flows to generate a plurality of tallies,

wherein at least one portion of the plurality of portions is defined based on a common FQDN among the plurality of FQDNs and a common server port among the plurality of server ports that are shared by all flows in the at least one portion, and

wherein the result comprises the plurality of tallies.

2. The method of claim 1 ,

wherein at least two of the plurality of server IP addresses are resolved by the DNS server to provide service based on the FQDN as requested from the plurality of client IP addresses, and

wherein the server IP address is resolved by the DNS server to provide service based on any of at least two of the plurality of FQDNs as requested from the plurality of client IP addresses.

3. The method of claim 1 , further comprising:

extracting a plurality of FQDN labels from the plurality of FQDNs,

wherein the plurality of FQDN labels are delimited by periods in the plurality of FQDNs, and

wherein at least another one portion of the plurality of portions is defined based on a common FQDN label among the plurality of FQDN labels and another common server port among the plurality of server ports that are shared by all flows in the at least another one portion.

4. The method of claim 1 ,

wherein the second portion of the plurality of server IP addresses of the plurality of flows are obtained from cached local DNS records without initiating any DNS query to the DNS server,

wherein the cached local DNS records are originated from the DNS server, and

wherein validity of the cached local DNS records is based on a time-to-live parameter determined by the DNS server.

5. The method of claim 1 , further comprising:

extracting, in response to the analyzing and based on the FQDN, a portion of the plurality of flows to form a training set; and

generating a signature associated with the FQDN based on the training set,

wherein the result comprises the signature.

6. The method of claim 1 , further comprising:

storing at least one selected from a group consisting of the plurality of server IP addresses and the plurality of server ports in a database as a plurality of entries tagged by the plurality of FQDNs.

7. The method of claim 1 , further comprising:

determining, in response to extracting the FQDN from the DNS response, that the FQDN is not yet stored in the data structure,

extracting the client IP address and the server IP address from the DNS query and the DNS response; and

storing, in response to determining that the FQDN is not yet stored in the data structure, the FQDN in the data structure indexed by the client IP address and the server IP address,

wherein tagging the plurality of flows using the corresponding FQDNs comprises:

determining that a flow of the plurality of flows is associated with the client IP address and the server IP address;

retrieving, in response to the determining, the FQDN from the data structure based on the client IP address and the server IP address; and

tagging, in response to the retrieving, the flow using the FQDN, and

wherein the result is generated in real time prior to completion of the flow.

8. A system for profiling network traffic of a network, comprising:

a data collector configured to:

obtain, from the network traffic, a plurality of domain name service (DNS) flows comprising a plurality of client Internet protocol (IP) addresses, a plurality of fully qualified domain names (FQDNs), and a plurality of server IP addresses,

wherein the plurality of DNS flows comprise a DNS query that is initiated from a client IP address of the plurality of client IP addresses to request service based on a FQDN of the plurality of FQDNs that is contained in the DNS query, and

wherein the plurality of server IP addresses comprise a server IP address that is resolved by a DNS server in the network based on the FQDN as requested from the client IP address and returned in a DNS response contained in the plurality of DNS flows; and

an analyzer configured to:

analyze the plurality of DNS flows to generate a mapping between the plurality of client IP addresses, the plurality of server IP addresses, and the plurality of FQDNs;

store the mapping in a data structure separate from the DNS server, wherein each of the plurality of client IP addresses and each of the plurality of server IP addresses form a client/server IP address pair that is used as input to the data structure for looking up one of the plurality of FQDNs as output of the data structure; and

analyze the network traffic based on the data structure to generate a result of profiling the network traffic, wherein analyzing the network traffic based on the data structure comprises:

obtaining, from the network traffic, a plurality of flows comprising a first portion of the plurality of client IP addresses, a second portion of the plurality of server IP addresses, and a plurality of server ports;

tagging the plurality of flows using corresponding FQDNs that are looked up using the data structure based on the first portion of the plurality of client IP addresses and the second portion of the plurality of server IP addresses contained in the plurality of flows;

storing the plurality of flows in association with the corresponding FQDNs in a repository as a plurality of tagged flows; and

tallying a plurality of portions of the plurality of tagged flows to generate a plurality of tallies,

wherein at least one portion of the plurality of portions is defined based on a common FQDN among the plurality of FQDNs and a common server port among the plurality of server ports that are shared by all flows in the at least one portion, and

wherein the result comprises the plurality of tallies.

9. The system of claim 8 ,

wherein at least two of the plurality of server IP addresses are resolved by the DNS server to provide service based on the FQDN as requested from the plurality of client IP addresses, and

wherein the server IP address is resolved by the DNS server to provide service based on any of at least two of the plurality of FQDNs as requested from the plurality of client IP addresses.

10. The system of claim 8 , the data collector further configured to:

extract a plurality of FQDN labels from the plurality of FQDNs,

wherein the plurality of FQDN labels are delimited by periods in the plurality of FQDNs, and

wherein at least another one portion of the plurality of portions is defined based on a common FQDN label among the plurality of FQDN labels and another common server port among the plurality of server ports that are shared by all flows in the at least another one portion.

11. The system of claim 8 ,

wherein the second portion of the plurality of server IP addresses of the plurality of flows are obtained from cached local DNS records without initiating any DNS query to the DNS server,

wherein the cached local DNS records are originated from the DNS server, and

wherein validity of the cached local DNS records is based on a time-to-live parameter determined by the DNS server.

12. The system of claim 8 , the analyzer further configured to:

extract, in response to the analyzing and based on the FQDN, a portion of the plurality of flows to form a training set; and

generate a signature associated with the FQDN based on the training set,

wherein the result comprises the signature.

13. The system of claim 8 , further comprising:

a database for storing at least one selected from a group consisting of the plurality of server IP addresses and the plurality of server ports as a plurality of entries tagged by the plurality of FQDNs.

14. The system of claim 8 , the data collector further configured to:

determine, in response to extracting the FQDN from the DNS response, that the FQDN is not yet stored in the data structure,

extract the client IP address and the server IP address from the DNS query and the DNS response; and

store, in response to determining that the FQDN is not yet stored in the data structure, the FQDN in the data structure indexed by the client IP address and the server IP address,

wherein tagging the plurality of flows using the corresponding FQDNs comprises:

determining that a flow of the plurality of flows is associated with the client IP address and the server IP address;

retrieving, in response to the determining, the FQDN from the data structure based on the client IP address and the server IP address; and

tagging, in response to the retrieving, the flow using the FQDN, and

wherein the result is generated in real time prior to completion of the flow.

15. A non-transitory computer readable medium embodying instructions for profiling network traffic of a network, the instructions when executed by a processor comprising functionality for:

obtaining, from the network traffic, a plurality of domain name service (DNS) flows comprising a plurality of client Internet protocol (IP) addresses, a plurality of fully qualified domain names (FQDNs), and a plurality of server IP addresses,

wherein the plurality of DNS flows comprise a DNS query that is initiated from a client IP address of the plurality of client IP addresses to request service based on a FQDN of the plurality of FQDNs that is contained in the DNS query, and

wherein the plurality of server IP addresses comprise a server IP address that is resolved by a DNS server in the network based on the FQDN as requested from the client IP address and returned in a DNS response contained in the plurality of DNS flows;

analyzing the plurality of DNS flows to generate a mapping between the plurality of client IP addresses, the plurality of server IP addresses, and the plurality of FQDNs;

storing the mapping in a data structure separate from the DNS server, wherein each of the plurality of client IP addresses and each of the plurality of server IP addresses form a client/server IP address pair that is used as input to the data structure for looking up one of the plurality of FQDNs as output of the data structure; and

analyzing the network traffic based on the data structure to generate a result of profiling the network traffic, wherein analyzing the network traffic based on the data structure comprises:

obtaining, from the network traffic, a plurality of flows comprising a first portion of the plurality of client IP addresses, a second portion of the plurality of server IP addresses, and a plurality of server ports;

tagging the plurality of flows using corresponding FQDNs that are looked up using the data structure based on the first portion of the plurality of client IP addresses and the second portion of the plurality of server IP addresses contained in the plurality of flows;

storing the plurality of flows in association with the corresponding FQDNs in a repository as a plurality of tagged flows; and

tallying a plurality of portions of the plurality of tagged flows to generate a plurality of tallies,

wherein at least one portion of the plurality of portions is defined based on a common FQDN among the plurality of FQDNs and a common server port among the plurality of server ports that are shared by all flows in the at least one portion, and

wherein the result comprises the plurality of tallies.

16. The non-transitory computer readable medium of claim 15 , wherein analyzing the network traffic based on the data structure comprises:

obtaining, from the network traffic, a plurality of flows comprising a first portion of the plurality of client IP addresses, a second portion of the plurality of server IP addresses, and a plurality of server ports;

tagging the plurality of flows by corresponding FQDNs that are looked up using the data structure based on the first portion of the plurality of client IP addresses and the second portion of the plurality of server IP addresses contained in the plurality of flows;

storing the plurality of flows in association with the corresponding FQDNs in a repository as a plurality of tagged flows; and

tallying a plurality of portions of the plurality of tagged flows to generate a plurality of tallies,

wherein at least one portion of the plurality of portions is defined based on a common FQDN among the plurality of FQDNs and a common server port among the plurality of server ports that are shared by all flows in the at least one portion, and

wherein the result comprises the plurality of tallies.