Aggregating select network traffic statistics
Disclosed herein are systems and methods for the collection, aggregation, and processing of network traffic statistics for a plurality of network appliances in a wide area network. Select network traffic statistics can be collected and associated with a hierarchical string, and aggregated over time. In this way, only information that is likely to be relevant is gathered and maintained, allowing for the maintenance of select network traffic statistics for large-scale operations.
1. A method for aggregating select network traffic statistics for each of a plurality of network appliances connected in a communication network, the method comprising:
for each flow from a first network appliance, extracting a first attribute value of a first flow attribute;
for each flow from the first network appliance, extracting a second attribute value of a second flow attribute;
building at least one hierarchical string with the extracted first attribute value and the extracted second attribute value,
wherein the hierarchical string represents a subset of network traffic statistics collected for the first network appliance, and
the second attribute of the hierarchical string is more specific than the first attribute of the hierarchical string;
extracting at least one network metric for at least one network characteristic associated with the at least one hierarchical string;
aggregating the at least one network metric for the at least one network characteristic over a plurality of flows to and from the first network appliance in the communication network;
generating an accumulating map that is updated in substantially real time, the accumulating map comprising the at least one hierarchical string and associated aggregated network metrics for the first flow attribute and the second flow attribute of the hierarchical string,
wherein the accumulating map has a target number of entries for a specified time period and an eviction policy determines how information is aggregated once the accumulating map reaches its target number of entries for the specified time period,
the eviction policy determining that a record is aggregated into a higher level record of the accumulating map and is evicted from the accumulating map; and
transmitting the accumulating map to a network information collector in communication with the plurality of network appliance.
2. The method of claim 1 , wherein information regarding each flow to or from a given network appliance is collected in a flow table.
3. The method of claim 1 , wherein the first and the second flow attributes are extracted at a first time interval.
4. The method of claim 1 , wherein the accumulating map is transmitted to the network information collector at a second time interval, the second time interval being a different amount of time than a first time interval.
5. The method of claim 1 , wherein a new accumulating map is started at the first network appliance after the aggregated information is transmitted to the network information collector.
6. The method of claim 1 , wherein the accumulating map comprises an eviction log for collected information in excess of the target number of entries for the specified time period, the eviction log comprising a summary of strings of information in excess of the target number of entries for the specified time period.
7. The method of claim 1 , wherein the eviction policy determines that once the target number of entries is reached for the specified time period, any new information collected will be discarded, and not aggregated during that time period.
8. The method of claim 1 , wherein the eviction policy further determines that an evicted record is moved to an eviction log when aggregated into a higher level record of the accumulating map.
9. The method of claim 1 , wherein the eviction policy determines that a portion of at least one hierarchical string of information is removed from the accumulating map to reduce the number of entries below a maximum number of entries for the specified time period.
10. The method of claim 1 , wherein the eviction policy removes a predetermined number of records from the accumulating map and moves them to an eviction log, when a maximum number of entries for the specified time period is reached.
11. The method of claim 1 , further comprising:
in response to a query regarding network traffic from a user, displaying a portion of the information collected from each network appliance on a graphical user interface to the user.
12. The method of claim 6 , wherein the eviction log is post-processed to minimize information loss.
13. The method of claim 1 , wherein the aggregated information is stored in bins.
14. The method of claim 1 , further comprising:
for each flow from the first network appliance, extracting a second network metric of the first flow attribute and its corresponding value.
15. A system for aggregating select network traffic statistics, comprising:
a plurality of network appliances in a communication network, each of the plurality of network appliances configured to:
collect a plurality of flow attributes for network traffic through each network appliance;
build at least one hierarchical string of network traffic flow attributes with an extracted first attribute value and an extracted second attribute value of the collected flow attributes, wherein the hierarchical string represents a subset of the collected flow attributes for the network appliance, and the second attribute of the hierarchical string is more specific than the first attribute of the hierarchical string;
extract at least one network metric for at least one network characteristic associated with each of the at least one hierarchical string;
aggregate the at least one network metric for the at least one network characteristic over a plurality of flows to or from the network appliance;
generate an accumulating map that is updated in substantially real time, the accumulating map comprising the at least one hierarchical string and associated aggregated network metrics for a first flow attribute and a second flow attribute of the hierarchical string,
wherein the accumulating map has a target number of entries for a specified time period and an eviction policy determines that a record is aggregated into a higher level record of the accumulating map and is evicted from the accumulating map when the accumulating map reaches the target number of entries for the specified time period; and
transmit the accumulating map to a network information collector in communication with each network appliance; and
the network information collector configured to receive information from each network appliance, and provide the information to a user on a graphical user display.
16. The system of claim 15 wherein the network information collector is further configured to store the information in one or more databases.
17. The system of claim 15 wherein each of the plurality of network appliances further generates at least one indexing data structure for the accumulating map.
18. The system of claim 15 , wherein the extracted first attribute value and the extracted second attribute value are extracted at a first time interval.
19. The system of claim 15 , wherein the accumulating map is transmitted to the network information collector at a second time interval, the second time interval being a different amount of time than a first time interval at which the extracted first attribute value and the extracted second attribute value are extracted.
20. The system of claim 15 , wherein the network appliance is further configured to:
generate a new accumulating map, after a previous accumulating map is transmitted to the network information collector.