Methods for intelligent load balancing and high speed intelligent network recorders
A high speed intelligent network recorder for recording a plurality of flows of network data packets into and out of a computer network over a relevant data time window is disclosed. The high speed intelligent network recorder includes a printed circuit board; a high speed network switching device mounted to the printed circuit board; and an X column by Y row array of a plurality of intelligent hard drives with micro-computers mounted to the printed circuit board and coupled in parallel with the high speed network switching device.
1. A method for recording and analytics of a plurality of packet records for network data packets entering a network over a relevant data time window, the method comprising:
receiving a plurality of incoming packets including ingress packets and egress packets, wherein each incoming packet belongs to one of a plurality of conversation flows;
forming a capture stream of packet records for the incoming packets, wherein each packet record includes metadata having a timestamp, a record length, and a flow hash; and
performing intelligent load balancing on the capture stream of packet records, the load balancing including
reading the metadata for each packet record,
determining a packet record is part of either a hot flow or a cold flow,
receiving a node status message from at least one destination node, wherein the node status message includes dynamic information including a level of fullness of a buffer for the at least one destination node, wherein the relevant data time window represents a usable storage capacity of the at least one destination node by using a plurality of units of time, wherein the plurality of units of time decreases according to an amount of failed storage capacity,
selecting a destination node for each packet record based on the flow hash and the node status message, and
steering the packet record to one of a plurality of encapsulation buffers based on the destination node, wherein a cold flow has a plurality of packets that are directed coherently to a single node.
2. The method of claim 1 , wherein:
each encapsulation buffer is configured to store a plurality of packet records temporarily for a particular active node.
3. The method of claim 1 , wherein:
each flow hash is a uniquely calculated tag for each conversation flow.
4. The method of claim 1 , wherein:
a hot flow is a conversation flow having an amount of traffic equal to or above a bandwidth threshold, and
a cold flow is a conversation flow having an amount of traffic below the bandwidth threshold.
5. The method of claim 4 , wherein:
selecting the destination node for each packet record is in response to determining the packet record is part of either a hot flow or a cold flow.
6. The method of claim 1 , wherein performing intelligent load balancing further comprises:
forming a plurality of record summaries for packet records based on the timestamp, the record length, and the flow hash;
adding the plurality of record summaries to a record summary queue;
applying a count-min sketch to the record length and the flow hash of each packet record, wherein the count-min sketch provides an estimate of a total length of records represented in the record summary queue for a particular conversation flow; and
based on the estimate of the total length of records, determining a packet record is part of either a hot flow or a cold flow.
7. The method of claim 1 , wherein performing intelligent load balancing further comprises:
forming a plurality of record summaries for packet records based on the timestamp, the record length, and the flow hash;
adding the plurality of record summaries to a record summary queue;
approximating a total recent bandwidth based on the plurality of record summaries in the record summary queue; and
based on the total recent bandwidth, determining a packet record is part of either a hot flow or a cold flow.
8. The method of claim 1 , wherein performing intelligent load balancing further comprises:
performing cold rebalancing including reassigning packet records for a cold flow to a different one of the plurality of encapsulation buffers.
9. The method of claim 1 , wherein performing intelligent load balancing further comprises:
receiving a node status message from at least one node, wherein the node status message includes static information about the destination node; and
performing node status and discovery by using the static information about the node.
10. The method of claim 1 , wherein performing intelligent load balancing further comprises:
performing node status and discovery by using the dynamic information about the destination node.
11. The method of claim 1 , wherein
a capture bandwidth of the destination node is less than one-tenth a total incoming bandwidth of the plurality of conversation flows.
12. The method of claim 1 , wherein
a capture bandwidth of the destination node is less than one-tenth of an admissible bandwidth of a single hot flow.
13. A computer-readable product for recording and analytics of a plurality of packet records for network data packets entering a network over a relevant data time window, the computer-readable product including a non-transitory computer-readable storage medium storing instructions that when executed perform the functions comprising:
receiving a plurality of incoming packets including ingress packets and egress packets, wherein each incoming packet belongs to one of a plurality of conversation flows;
forming a capture stream of packet records for the incoming packets, wherein each packet record includes metadata having a timestamp, a record length, and a flow hash; and
performing intelligent load balancing on the capture stream of packet records, the load balancing including
reading the metadata for each packet record,
determining a packet record is part of either a hot flow or a cold flow,
receiving a node status message from at least one destination node, wherein the node status message includes dynamic information including a level of fullness of a buffer for the at least one destination node, wherein the relevant data time window represents a usable storage capacity of the at least one destination node by using a plurality of units of time, wherein the plurality of units of time decreases according to an amount of failed storage capacity,
selecting a destination node for each packet record based on the flow hash and the node status message, and
steering the packet record to one of a plurality of encapsulation buffers based on the destination node, wherein a cold flow has a plurality of packets that are directed coherently to a single node.
14. The computer-readable medium of claim 13 , wherein:
each encapsulation buffer is configured to store a plurality of packet records temporarily for a particular active node.
15. The computer-readable medium of claim 13 , wherein:
each flow hash is a uniquely calculated tag for each conversation flow.
16. The computer-readable medium of claim 13 , wherein:
a hot flow is a conversation flow having an amount of traffic equal to or above a bandwidth threshold, and
a cold flow is a conversation flow having an amount of traffic below the bandwidth threshold.
17. The computer-readable medium of claim 16 , wherein:
selecting the destination node for each packet record is in response to determining the packet record is part of either a hot flow or a cold flow.
18. The computer-readable medium of claim 13 , wherein performing intelligent load balancing further comprises:
forming a plurality of record summaries for packet records based on the timestamp, the record length, and the flow hash;
adding the plurality of record summaries to a record summary queue;
applying a count-min sketch to the record length and the flow hash of each packet record, wherein the count-min sketch provides an estimate of a total length of records represented in the record summary queue for a particular conversation flow; and
based on the estimate of the total length of records, determining a packet record is part of either a hot flow or a cold flow.
19. The computer-readable medium of claim 13 , wherein performing intelligent load balancing further comprises:
forming a plurality of record summaries for packet records based on the timestamp, the record length, and the flow hash;
adding the plurality of record summaries to a record summary queue;
approximating a total recent bandwidth based on the plurality of record summaries in the record summary queue; and
based on the total recent bandwidth, determining a packet record is part of either a hot flow or a cold flow.
20. The computer-readable medium of claim 13 , wherein performing intelligent load balancing further comprises:
performing cold rebalancing including reassigning packet records for a cold flow to a different one of the plurality of encapsulation buffers.
21. The computer-readable medium of claim 13 , wherein performing intelligent load balancing further comprises:
receiving a node status message from at least one destination node, wherein the node status message includes static information about the destination node; and
performing node status and discovery by using the static information about the destination node.
22. The computer-readable medium of claim 13 , wherein performing intelligent load balancing further comprises:
performing node status and discovery by using the dynamic information about the destination node.
23. The computer-readable medium of claim 13 , wherein
a capture bandwidth of the destination node is less than one-tenth a total incoming bandwidth of the plurality of conversation flows.
24. The computer-readable medium of claim 13 , wherein
a capture bandwidth of the destination node is less than one-tenth of an admissible bandwidth of a single hot flow.