Method and apparatus of estimating conversation in a distributed netflow environment
A distributed network traffic flow monitoring system is provided. The system includes a plurality of network packet flow collector devices for receiving netflow packets associated with a traffic flow activity data at a network connection. The traffic flow activity data includes one or more transmissions between particular source and destination IP addresses. The system further includes a network packet flow coordinator device connected to the plurality of network packet flow collector devices and configured to aggregate probabilistic information related to the traffic flow activity data from the one or more of network packet flow collector devices and to estimate cardinality of the traffic flow activity data based on the aggregated probabilistic information.
1. A distributed network traffic flow monitoring system, comprising:
a plurality of network packet flow collector devices for receiving netflow packets associated with a traffic flow activity data at a network connection, the traffic flow activity data comprising one or more transmissions between particular source and destination IP addresses wherein each of the plurality of network packet flow collector devices maintains a plurality of entries corresponding to individual transmissions observed by each of the plurality of network packet flow collector devices; and
a network packet flow coordinator device connected to the plurality of network packet flow collector devices and configured to aggregate probabilistic counter information related to the traffic flow activity data from the plurality of network packet flow collector devices and configured to estimate cardinality of the traffic flow activity data based on the aggregated probabilistic counter information wherein the network packet flow coordinator device is further configured to estimate cardinality of the traffic flow activity data by applying a HyperLogLog technique to the aggregated plurality of entries using one or more probabilistic counters.
2. The system of claim 1 , wherein the network packet flow coordinator estimates cardinality of the traffic flow activity data responsive to a request received from a client device.
3. The system of claim 2 , wherein the network packet flow coordinator device is further configured to send the estimated cardinality of the traffic flow activity data to the client device.
4. The system of claim 2 , wherein the request received from the client device specifies criteria for filtering the traffic flow activity data.
5. The system of claim 1 , wherein the network packet flow coordinator device is further configured to aggregate the plurality of entries from the plurality of network packet flow collector devices and configured to estimate cardinality of the traffic flow activity data based on the aggregated plurality of entries.
6. The system of claim 1 , wherein at least one of the plurality of network packet flow collector devices is further configured to estimate cardinality of a subset of the plurality of entries that match a filter by applying a HyperLogLog technique to the plurality of entries using one or more probabilistic counters, responsive to a request received from the network packet flow coordinator.
7. The system of claim 1 , wherein the network packet flow coordinator device configured to estimate cardinality of the traffic flow activity data is further configured to perform a set operation on the one or more probabilistic counters.
8. The system of claim 7 , wherein the set operation is union.
9. A method for estimating cardinality of traffic flow activity data in a distributed network traffic flow monitoring system, the method comprising:
sending, by a network packet flow coordinator device, one or more requests to a plurality of network packet flow collector devices connected to the network packet flow coordinator device, responsive to receiving a request from a client device, the request including criteria for filtering the traffic flow activity data;
aggregating, by the network packet flow coordinator device, responses received from the plurality of network packet flow collector devices, the responses comprising probabilistic counter information related to the traffic flow activity data; and
estimating, by the network packet flow coordinator device, cardinality of the traffic flow activity data based on the aggregated probabilistic counter information wherein estimating cardinality of the traffic flow activity data further comprises applying a HyperLogLog technique to the aggregated probabilistic information using one or more probabilistic counters and performing a set operation on the one or more probabilistic counters.
10. The method as recited in claim 9 , wherein each of the plurality of network packet flow collector devices maintains a plurality of entries corresponding to individual transmissions observed by each of the plurality of network packet flow collector devices.
11. The method as recited in claim 10 , wherein at least one of the plurality of network packet flow collector devices is configured to estimate cardinality of a subset of the plurality of entries that match the filtering criteria by applying a HyperLogLog technique to the plurality of entries using one or more probabilistic counters, responsive to the one or more requests received from the network packet flow coordinator.
12. The method as recited in claim 9 , further comprising sending the estimated cardinality of the traffic flow activity data to the client device.
13. The method as recited in claim 9 , wherein the set operation is union.
14. The method as recited in claim 9 , wherein the traffic flow activity data comprises one or more transmissions between particular source and destination IP addresses.
15. A non-transitory computer readable storage medium and one or more computer programs embedded therein, the computer programs comprising instructions, which when executed by a computer system, cause the computer system to:
send one or more requests to a plurality of network packet flow collector devices connected to the computer system, responsive to receiving a request from a client device, the request including criteria for filtering traffic flow activity data;
maintain a plurality of entries corresponding to individual transmissions observed by each of the plurality of network packet flow collector devices;
aggregate responses received from the plurality of network packet flow collector devices, the responses comprising probabilistic counter information related to the traffic flow activity data; and
estimate cardinality of the traffic flow activity data based on the aggregated probabilistic counter information by applying a HyperLogLog technique to the aggregated plurality of entries using one or more probabilistic counters to perform a set operation on the one or more probabilistic counters.
16. The non-transitory computer readable storage medium as recited in claim 15 , wherein each of the plurality of network packet flow collector devices maintains a plurality of entries corresponding to individual transmissions observed by each of the plurality of network packet flow collector devices.