In-line tool performance monitoring and adaptive packet routing
Embodiments are disclosed for monitoring the performance of an in-line tool without adding data to network traffic routed through the in-line tool. In some embodiments, performance of the in-line tool is based on a measured latency introduced by the processing of packets through the in-line tool. In some embodiments, network traffic is adaptively routed based on the measured latency at the in-line tool.
1. A method comprising:
receiving a first packet, via a network port of a network switch appliance, from a source node on a computer network, the first packet destined for a destination node on the computer network;
forwarding the first packet, by the network switch appliance, to an in-line tool for processing via a first route;
storing, by the network switch appliance, information associated with the first packet as an entry in a key-value data structure, the entry including:
a source node identifier associated with the source node as a key; and
a payload data identifier and first timestamp as values associated with the key, the first timestamp based on a time of forwarding the first packet to the first packet to the in-line tool;
receiving the first packet, by the network switch appliance, from the in-line tool after the processing;
identifying, by the network switch appliance, the first packet received from the in-line tool as the same first packet forwarded to the in-line tool based on the entry in the key-value data structure; and
measuring, by a processor in the network switch appliance, a latency in network traffic through the in-line tool based on a difference between the first timestamp and a second timestamp, the second timestamp based on a time of receiving the first packet from the in-line tool.
2. The method of claim 1 , wherein the source node identifier is a media access control (MAC) address associated with the source node.
3. The method of claim 1 , wherein the payload data identifier is a computed cyclic redundancy check (CRC) value based on the payload data in the first packet.
4. The method of claim 1 , wherein the key-value data structure includes information associated with a plurality of packets received from the source node and other source nodes on the computer network.
5. The method of claim 1 , wherein the key value is associated with values for a plurality of packets received from the source node.
6. The method of claim 1 , wherein the identifying includes: determining the source node identifier associated with the first packet received from the in-line tool; identifying a key value stored in the key-value data structure matching the source node identifier associated with the first packet received from the in-line tool; and in response to identifying the key, identifying a payload data associated with the key matching the payload data identifier associated with the first packet received from the in-line tool.
7. The method of claim 1 , further comprising:
receiving a second packet, via the network port of the network switch appliance, from the source node, the second packet destined for the destination node;
in response to a determination that the measured latency satisfies a specified criterion, selecting, by the network switch appliance, a second route for the second packet different from the first route; and
forwarding, by the network switch appliance, the second packet for delivery to the destination node via the second route.
8. The method of claim 7 , wherein the second route bypasses the in-line tool.
9. The method of claim 7 , wherein the second route includes routing the second packet directly to the destination node.
10. The method of claim 7 , wherein the second route includes an alternative in-line tool.
11. The method of claim 7 , further comprising:
identifying, by the processor of the network switch appliance, a particular session out of a plurality of sessions with which the second packet is associated; and
forwarding, by the network switch appliance, other packets associated with the particular session via the second route.
12. The method of claim 7 , further comprising:
measuring, by the processor in the network switch appliance, a latency of the network traffic through the in-line tool over a time period; and
calculating, by the processor in the network switch appliance, a threshold latency value including one or more of a minimum, maximum, or average latency of the network traffic through the in-line tool over the time period;
wherein the specified criterion is based on the calculated threshold latency value.
13. The method of claim 12 , wherein specified criterion is further based on a priority level of the network traffic.
14. The method of claim 7 , further comprising:
tracking, by the processor in the network switch appliance, changes in the measured latency of the network traffic through the in-line tool over a time period;
defining, by the processor in the network switch appliance, a characteristic associated with the in-line tool based on the tracking; and
adjusting, by the processor in the network switch appliance, the specified criterion based on the characteristic associated with the in-line tool.
15. The method of claim 14 , wherein the characteristic associated with the in-line tool includes one or more of:
a statistical measure of latency when processing a particular type of network traffic,
a statistical measure of latency during particular time periods, or
a statistical measure of a rate of change in latency under particular network traffic conditions.
16. A network switch appliance comprising: a processor; and
a memory having instructions stored thereon, which when executed by the processor, cause the network switch appliance to:
receive a first packet from a source node on a computer network, the first packet destined for a destination node on the computer network;
forward the first packet to an in-line tool for processing via a first route;
store information associated with the first packet as an entry in a key-value data structure, the entry including:
a source node identifier associated with the source node as a key; and
a payload data identifier and first timestamp as values associated with the key, the first timestamp based on a time of forwarding the first packet to the in-line tool;
receive the first packet from the in-line tool after the processing;
identify the first packet received from the in-line tool as the same first packet forwarded to the in-line tool based on the entry in the key-value data structure; and
measure a latency in network traffic through the in-line tool based on a difference between the first timestamp and a second timestamp, the second timestamp based on a time of receiving the first packet from the in-line took.
17. The network switch appliance of claim 16 , wherein the memory has further instructions stored thereon, which when executed by the processor, cause the network switch appliance to further:
redirect network traffic to bypass the in-line tool if the measured latency satisfies a specified criterion.
18. The network switch appliance of claim 16 , wherein the memory has further instructions stored thereon, which when executed by the processor, cause the network switch appliance to further:
receive a second packet from the source node, the second packet destined for the destination node;
in response to a determination that the measured latency satisfies a specified criterion, select a second route for the second packet different from the first route and that bypasses the in-line tool; and
forward the second packet for delivery to the destination node via the second route.
19. A non-transitory machine-readable storage medium for use in a processing system, the non-transitory machine-readable storage medium storing instructions, execution of which in the processing system causes the processing system to perform operations comprising:
receiving a first packet from a source node on a computer network, the first packet destined for a destination node on the computer network;
forwarding the first packet to an in-line tool for processing via a first route;
storing information associated with the first packet as an entry in a key-value data structure, the entry including:
a source node identifier associated with the source node as a key; and
a payload data identifier and first timestamp as values associated with the key, the first timestamp based on a time of forwarding the first packet to the in-line tool;
receiving the first packet from the in-line tool after the processing;
identifying the first packet received from the in-line tool as the same first packet forwarded to the in-line tool based on the entry in the key-value data structure; and
measuring a latency in network traffic through the in-line tool based on a difference between the first timestamp and a second timestamp, the second timestamp based on a time of receiving the first packet from the in-line tool.
20. The non-transitory machine-readable storage medium of claim 19 storing further instructions, execution of which in the processing system causes the processing system to perform operations further comprising:
redirecting network traffic to bypass the in-line tool if the measured latency satisfies a specified criterion.
21. The non-transitory machine-readable storage medium of claim 19 storing further instructions, execution of which in the processing system causes the processing system to perform operations further comprising:
receiving a second packet from the source node, the second packet destined for the destination node;
in response to a determination that the measured latency satisfies a specified criterion, selecting a second route for the second packet different from the first route and that bypasses the in-line tool; and
forwarding the second packet for delivery to the destination node via the second route.