System and method for identification of large-data flows
Apparatus, systems and methods may be used to monitor data flows and to select and track particularly large data flows. A method of tracking data flows and identifying large-data (“elephant”) flows comprises extracting fields from a packet of data to construct a flow key, computing a hash value on the flow key to provide a hashed flow signature, entering and/or comparing the hashed flow signature with entries in a flow hash table. Each hash table entry includes a byte count for a respective flow. When the byte count for a flow exceeds a threshold value, the flow is added to a large-data flow (“elephant”) table and the flow is then tracked in the large-data flow table.
1. A method of tracking data flows, comprising:
computing a hashed flow signature on values extracted from a data packet received on a network device, the hashed flow signature being used to locate a position for the hashed flow signature in a flow hash table;
adding a flow key of a respective entry in the flow hash table to a large-data flow table when a byte count for an entry in the flow hash table exceeds a predetermined byte count threshold, wherein the entry in the flow hash table includes the byte count for the respective flow; and
replacing the byte count in the flow hash table for the entry added to the large-data flow table with a pointer to the entry in the large-data flow table.
2. The method of claim 1 further comprising:
reading a row in the flow hash table indexed by the hashed index value; and
comparing the hashed flow signature with each signature in the row read from the flow hash table to determine if the hashed flow signature matches at least one existing flow signature in a column of the row to locate the hashed flow signature in the flow hash table.
3. The method of claim 2 wherein the comparing the hashed flow signature with
each signature in the row read from the hash table further comprises:
when the hashed flow signature matches the at least one existing flow signature in a column of the row of the flow hash table, locating the column in the flow hash table and adding a length of the packet to a byte counter in the flow hash table.
4. The method of claim 2 wherein the comparing the hashed flow signature with each signature in the row read from the hash table further comprises:
when the hashed flow signature does not match the at least one existing flow signature in the flow hash table, entering the hashed flow signature into a column in the row by:
locating an empty column in the row and inserting the hashed flow in the empty column.
5. The method of claim 2 wherein the comparing the hashed flow signature with each signature in the row read from the hash table further comprises:
when the hashed flow signature does not match the at least one existing flow signature in the flow hash table, entering the hashed flow signature into a column in the row by evicting an entry when there is not an empty column in the row, and inserting the hashed flow signature in the evicted column.
6. The method of claim 1 further comprising tracking a bandwidth of each flow entry in the large-data flow table based on the byte count, and aging out entries from the large-data flow table.
7. A non-transitory computer readable medium containing program instructions for causing a computer to perform operations, comprising:
computing a hashed flow signature on values extracted from a data packet received on a network device, the hashed flow signature being used to locate a position for the hashed flow signature in a flow hash table;
adding a flow key of a respective entry in the flow hash table to a large-data flow table when a byte count for an entry in the flow hash table exceeds a predetermined byte count threshold, wherein the entry in the flow hash table includes the byte count for the respective flow; and
replacing the byte count in the flow hash table for the entry added to the large-data flow table with a pointer to the entry in the large-data flow table.
8. The method of claim 7 the operations further comprising:
reading a row in the flow hash table indexed by the hashed index value; and
comparing the hashed flow signature with each signature in the row read from the flow hash table to determine if the hashed flow signature matches at least one existing flow signature in a column of the row to locate the hashed flow signature in the flow hash table.
9. The media of claim 8 wherein the comparing the hashed flow signature with each signature in the row read from the hash table further comprises:
when the hashed flow signature matches the at least one existing flow signature in a column of the row of the flow hash table, locating the column in the flow hash table and adding a length of the packet to a byte counter in the flow hash table.
10. The media of claim 8 wherein the comparing the hashed flow signature with each signature in the row read from the hash table further comprises:
when the hashed flow signature does not match the at least one existing flow signature in the flow hash table, entering the hashed flow signature into a column in the row by:
locating an empty column in the row and inserting the hashed flow in the empty column.
11. The media of claim 7 wherein the comparing the hashed flow signature with each signature in the row read from the hash table further comprises:
when the hashed flow signature does not match the at least one existing flow signature in the flow hash table, entering the hashed flow signature into a column in the row by evicting an entry when there is not an empty column in the row, and inserting the hashed flow signature in the evicted column.
12. The media of claim 7 , the operations further comprising tracking a bandwidth of each flow entry in the large-data flow table based on the byte count, and aging out entries from the large-data flow table.
13. A system of tracking data flows, comprising:
a memory;
a processor, the processor being configured to perform operations comprising:
computing a hashed flow signature on values extracted from a data packet received on a network device, the hashed flow signature being used to locate a position for the hashed flow signature in a flow hash table;
adding a flow key of a respective entry in the flow hash table to a large-data flow table when a byte count for an entry in the flow hash table exceeds a predetermined byte count threshold, wherein the entry in the flow hash table includes the byte count for the respective flow; and
replacing the byte count in the flow hash table for the entry added to the large-data flow table with a pointer to the entry in the large-data flow table.
14. The system of claim 13 the operations further comprising:
reading a row in the flow hash table indexed by the hashed index value; and
comparing the hashed flow signature with each signature in the row read from the flow hash table to determine if the hashed flow signature matches at least one existing flow signature in a column of the row to locate the hashed flow signature in the flow hash table.
15. The system of claim 14 wherein the comparing the hashed flow signature with each signature in the row read from the hash table further comprises:
when the hashed flow signature matches the at least one existing flow signature in a column of the row of the flow hash table, locating the column in the flow hash table and adding a length of the packet to a byte counter in the flow hash table.
16. The system of claim 14 wherein the comparing the hashed flow signature with each signature in the row read from the hash table further comprises:
when the hashed flow signature does not match the at least one existing flow signature in the flow hash table, entering the hashed flow signature into a column in the row by:
locating an empty column in the row and inserting the hashed flow in the empty column.
17. The system of claim 14 wherein the comparing the hashed flow signature with each signature in the row read from the hash table further comprises:
when the hashed flow signature does not match the at least one existing flow signature in the flow hash table, entering the hashed flow signature into a column in the row by evicting an entry when there is not an empty column in the row, and inserting the hashed flow signature in the evicted column.