Flow sampling with top talkers
An improved network flow sampling apparatus and corresponding method more intelligently selects flows to be reported beyond random flow or packet sampling intervals. Evaluation of flows selects top talkers or heavy hitters as important flows that most impact traffic and therefore should be reported. Top talkers are flows defined based on a customized definition according to, for example, number of packets in a flow, number of bytes in a flow, or bandwidth consumed by a flow. An external collector can thus proceed with traffic profiling or other flow monitoring purposes with a more intelligently determined network flow sample.
1. A method in a network device for performing sampling-based monitoring of flows in a plurality of buckets, the method comprising the steps of:
receiving a packet at a forwarding plane of the network device;
determining whether the packet is part of an existing flow already tracked in one of the plurality of buckets ranging from a least-significant bucket to a most-significant bucket in terms of a predetermined criteria that identifies top talkers, wherein the top talkers are the flows that most impact traffic of the network device, wherein each of the plurality of buckets sort the flows tracked therein by order of least recently used at a tail of each bucket and most recently used at a head of each bucket;
upon determination that the packet is part of the existing flow already tracked in one of the plurality of buckets, performing the following steps:
adding the packet to that existing flow; and
moving that existing flow to the head of a bucket of the plurality of buckets, the bucket for that existing flow is selected based on the predetermined criteria;
upon determination that the packet is not part of any existing flows already tracked in one of the plurality of buckets, performing the following steps:
determining whether a total number of flows tracked in the plurality of buckets of the forwarding plane has reached a maximum limit;
discarding a flow from the tail of the least-significant bucket only when the total number of flows in the plurality of buckets of the forwarding plane has reached a maximum limit, whereby the flow is discarded to allow a new flow to be created without causing the total number of flows to exceed the maximum limit and because the flow to be discarded is the least recently used flow in the least-significant bucket; and
creating a new flow for the packet at the head of the least-significant bucket;
reporting information on the flows tracked in the plurality of buckets upon the flows' expiration from the forwarding plane to a control plane of the network device,
wherein the information includes at least one of source IP address, destination IP address, source port number, destination port number, number of packets in the flow, number of bytes in the flow, time of first packet of the flow, or time of last packet of the flow, whereby reporting only the information on those flows that remain tracked in any of the plurality of buckets by not being discarded upon the flows' expiration ensures that information on the top talkers is reported and also guarantees that information on at least one flow is reported because all flows created by the network device are reported up to the maximum limit; and
exporting the information on the flows reported to the control plane from the control plane to an external collector for further analysis or processing by the external collector.
2. The method of claim 1 , wherein the predetermined criteria that identifies the top talkers is based on at least one of a total number of packets in the flow, a total number of bytes in the flow, a number of packets per second in the flow, a number of bytes per second in the flow, or a lifetime of the flow.
3. The method of claim 2 , wherein each of the plurality of buckets is designated for flows containing a different predetermined range of packets.
4. The method of claim 3 , wherein the least-significant bucket is designated to track flows containing a lowest number of packets and the most-significant bucket is designated to track flows containing a highest number of packets.
5. A network device for performing sampling-based monitoring of flows in a plurality of buckets, comprising:
a forwarding plane having memory and CPU including:
a packet receiving module to receive a packet;
a flow organizing module having the plurality of buckets, the flow organizing module to determine whether the packet is part of an existing flow already tracked in one of the plurality of buckets ranging from a least-significant bucket to a most-significant bucket in terms of a predetermined criteria that identifies top talkers, wherein the top talkers are the flows that most impact traffic of the network device, wherein each of the plurality of buckets sort the flows tracked therein by order of least recently used at a tail of each bucket and most recently used at a head of each bucket, wherein upon determination that the packet is part of the existing flow already tracked in one of the plurality of buckets, the flow organizing module to add the packet to that existing flow and move that existing flow to the head of a bucket of the plurality of buckets, the bucket for that existing flow is selected based on the predetermined criteria, and wherein upon determination that the packet is not part of any existing flows already tracked in one of the plurality of buckets, the flow organizing module to determine whether a total number of flows tracked in the plurality of buckets of the forwarding plane has reached a maximum limit, discard a flow from the tail of the least-significant bucket only when the total number of flows in the plurality of buckets of the forwarding plane has reached a maximum limit, whereby the flow is discarded to allow a new flow to be created without causing the total number of flows to exceed the maximum limit and because the flow to be discarded is the least recently used flow in the least-significant bucket, and create a new flow for the packet at the head of the least-significant bucket; and
a flow reporting module to report information on the flows tracked in the plurality of buckets upon the flows' expiration from the forwarding plane to a control plane, wherein the information includes at least one of source IP address, destination IP address, source port number, destination port number, number of packets in the flow, number of bytes in the flow, time of first packet of the flow, or time of last packet of the flow, whereby reporting only the information on those flows that remain tracked in any of the plurality of buckets by not being discarded upon the flows' expiration ensures that information on the top talkers is reported and also guarantees that information on at least one flow is reported because all flows created by the network device are reported up to the maximum limit; and
the control plane having memory and CPU including:
a flow exporter to export the information on the flows reported to the control plane from the control plane to an external collector for further analysis or processing by the external collector.
6. The network device of claim 5 , wherein the predetermined criteria that identifies the top talkers is based on at least one of a total number of packets in the flow, a total number of bytes in the flow, a number of packets per second in the flow, a number of bytes per second in the flow, or a lifetime of the flow.
7. The network device of claim 6 , wherein each of the plurality of buckets is designated for flows containing a different predetermined range of packets.
8. The network device of claim 7 , wherein the least-significant bucket is designated to track flows containing a lowest number of packets and the most-significant bucket is designated to track flows containing a highest number of packets.