Systems and methods for detecting large network flows
In a system for efficiently detecting large/elephant flows in a network, the rate at which the received packets are sampled is adjusted according to the measured heavy tailedness of the arriving traffic, such that the measured heavy tailedness reaches a specified target level. The heavy tailedness is measured using the estimated sizes of different flows associated with the arriving packets. When the measured heavy tailedness reaches and remains at the specified target level, the flows having the largest estimated sizes are likely to be the largest/elephant flows in the network.
1. A method for classifying network flows, the method comprising:
selecting a sampling rate, and sampling a plurality of packets arriving at a processing node at the sampling rate;
identifying a plurality of flows, each flow being associated with at least one sampled packet;
selecting a cache of flows from the plurality of flows, and estimating a respective size of each of the flows in the cache, each respective size comprising a rate of arrival of packets at the processing node;
computing a heavy tailedness measure of the cache of flows based on a standardized moment of a distribution of respective sizes of the flows in the cache;
adjusting the sampling rate for all flows in the cache based on the computed heavy tailedness measure if the computed heavy tailedness measure is outside of a specified range from a specified target heavy tailedness, the sampling rate being a same sampling rate for all flows in the cache of flows; and
classifying a subset of flows having respective sizes larger than respective sizes of all other flows in the cache, as large flows.
2. The method of claim 1 , wherein:
prior to adjustment, the sampling rate is in a range from 0.000001 up to 1.0; and
after at least one adjustment, the sampling rate is in a range from 0.00001 up to 0.1.
3. The method of claim 1 , wherein the rate of arrival of packets at the processing node is in a range from 100 Mbps up to 10 Tbps.
4. The method of claim 1 , wherein selecting the sampling rate comprises initially selecting a sampling rate that is inversely proportional to the rate of arrival of packets at the processing node.
5. The method of claim 1 , wherein identifying a flow associated with a sampled packet comprises designating the sampled packet to a particular flow based on at least one of:
a pair of source and destination addresses in a header of the sampled packet;
a pair of source and destination port numbers in the header of the sampled packet; and
a virtual local area network (VLAN) identifier included in the header of the sampled packet.
6. The method of claim 1 , wherein selecting the cache of flows comprises including each flow in the identified plurality of flows in the cache of flows.
7. The method of claim 6 , wherein selecting the cache of flows further comprises:
determining that no additional sampled packets were associated with a particular flow during an inactivity timeout period; and
removing that particular flow from the cache of flows.
8. The method of claim 1 , wherein:
the size of a flow is based on a metric selected from the group consisting of: (i) a number of packets belonging to the flow at a time of estimation, (ii) a number of bytes belonging to the flow at the time of estimation, (iii) an average number of packets belonging to the flow per unit time, (iv) an average number of bytes belonging to the flow per the unit time, and (v) bustiness of the flow; and
estimating the size of the flow comprises computing the metric.
9. The method of claim 1 , wherein computing the heavy tailedness measure of the cache of flows comprises computing at least one of:
variance of the respective flow sizes;
skewness of the respective flow sizes; and
kurtosis of the respective flow sizes.
10. The method of claim 1 , wherein after adjusting the sampling rate, the sampling rate is not adjusted at least for a specified rate adjustment interval.
11. The method of claim 1 , wherein adjusting the sampling rate comprises one of:
increasing the sampling rate if the computed heavy tailedness measure is less than the specified target heavy tailedness; and
decreasing the sampling rate if the computed heavy tailedness measure is greater than the specified target heavy tailedness.
12. The method of claim 11 , wherein:
increasing the sampling rate comprises increasing the sampling rate by a selected step size; and
decreasing the sampling rate comprises decreasing the sampling rate by the selected step size.
13. A system for classifying network flows, the system comprising:
a first processor; and
a first memory in electrical communication with the first processor, the first memory comprising instructions which, when executed by a processing unit comprising at least one of the first processor and a second processor, and in electronic communication with a memory module comprising at least one of the first memory and a second memory, configure the processing unit to:
select a sampling rate, and sampling a plurality of packets arriving at a processing node at the sampling rate;
identify a plurality of flows, each flow being associated with at least one sampled packet;
select a cache of flows from the plurality of flows, and estimating a respective size of each of the flows in the cache, each respective size comprising a rate of arrival of packets at the processing node;
compute a heavy tailedness measure of the cache of flows based on a standardized moment of a distribution of respective sizes of the flows in the cache;
adjust the sampling rate for all flows in the cache based on the computed heavy tailedness measure if the computed heavy tailedness measure is outside of a specified range from a specified target heavy tailedness, the sampling rate being a same sampling rate for all flows in the cache of flows; and
classify a subset of flows having respective sizes larger than respective sizes of all other flows in the cache, as large flows.
14. The system of claim 13 , wherein:
prior to adjustment, the sampling rate is in a range from 0.000001 up to 1.0; and
after at least one adjustment, the sampling rate is in a range from 0.00001 up to 0.1.
15. The system of claim 13 , wherein the rate of arrival of packets at the processing node is in a range from 100 Mbps up to 10 Tbps.
16. The system of claim 13 , wherein to select the sampling rate the instructions program the processing unit to select initially a sampling rate that is inversely proportional to the rate of arrival of packets at the processing node.
17. The system of claim 13 , wherein to identify a flow associated with a sampled packet the instructions program the processing unit to designate the sampled packet to a particular flow based on at least one of:
a pair of source and destination addresses in a header of the sampled packet;
a pair of source and destination port numbers in the header of the sampled packet; and
a virtual local area network (VLAN) identifier included in the header of the sampled packet.
18. The system of claim 13 , wherein to select the cache of flows the instructions program the processing unit to include each flow in the identified plurality of flows in the cache of flows.
19. The system of claim 18 , wherein to select the cache of flows the instructions further program the processing unit to:
determine that no additional sampled packets were associated with a particular flow during an inactivity timeout period; and
remove that particular flow from the cache of flows.
20. The system of claim 13 , wherein:
the size of a flow is based on a metric selected from the group consisting of: (i) a number of packets belonging to the flow at a time of estimation, (ii) a number of bytes belonging to the flow at the time of estimation, (iii) an average number of packets belonging to the flow per unit time, (iv) an average number of bytes belonging to the flow per the unit time, and (v) bustiness of the flow; and
to estimate the size of the flow the instructions program the processing unit to compute the metric.
21. The system of claim 13 , wherein to compute the heavy tailedness measure of the cache of flows the instructions program the processing unit to compute at least one of:
variance of the respective flow sizes;
skewness of the respective flow sizes; and
kurtosis of the respective flow sizes.
22. The system of claim 13 , wherein the instructions program the processing unit not to adjust the sampling rate at least for a specified rate adjustment interval, after adjusting the sampling rate.
23. The system of claim 13 , wherein to adjust the sampling rate the instructions program the processing unit to one of:
increase the sampling rate if the computed heavy tailedness measure is less than the specified target heavy tailedness; and
decrease the sampling rate if the computed heavy tailedness measure is greater than the specified target heavy tailedness.
24. The system of claim 23 , wherein:
to increase the sampling rate the instructions program the processing unit to increase the sampling rate by a selected step size; and
to decrease the sampling rate the instructions program the processing unit to decrease the sampling rate by the selected step size.