Methods, systems and computer readable media for evaluating load balancing (LB) mechanisms using packet in-flight time

1 . A method for evaluating a load balancing (LB) mechanism, the method comprising:

at a test system implemented using at least one processor:

sending, via one or more paths comprising network nodes, a first set of test packets comprising transmit (TX) timestamps for evaluating a first LB mechanism;

receiving, at a packet destination, the first set of test packets;

generating receive (RX) timestamps for the first set of test packets;

computing, by the packet destination or a packet analyzer, first in-flight metrics for the first set of test packets using the TX and RX timestamps, grouping the first set of test packets based on packet or flow characteristics and generating an average in-flight metric, a minimum in-flight metric, or a maximum in-flight metric for each of a plurality of groups, wherein grouping the first set of test packets based on the packet or flow characteristics includes assigning, to a first group, test packets in the first set of test packets that are part of packet flows of a first flow type and assigning, to a second group, test packets in the first set of test packets that are part of packet flows of a second flow type different from the first flow type and computing the first in-flight metrics includes computing the first in-flight metrics for the first and second flow types, wherein the first flow type includes packet flows of a first number of packets and a first packet size distribution and the second flow type includes packet flows of a second number of packets greater than the first number of packets and a second packet size distribution with packets of different sizes than the first packet size distribution; and

evaluating the first LB mechanism using the first in-flight metrics, wherein evaluating the first LB mechanism using the first in-flight metrics includes comparing the first in-flight metrics computed for first flow type with the first in-flight metrics computed for the second flow type, wherein comparing the in-flight metrics for the first flow type with the in-flight metrics for the second flow type includes comparing packet in-flight times for the packet flows with the first number of packets and the first packet size distribution with packet in-flight times for the packet flows with the second number of packets and the second packet size distribution.

2 . The method of claim 1 comprising:

at the test system:

sending, via one or more paths comprising network nodes, a second set of test packets comprising TX timestamps for evaluating a second LB mechanism;

receiving, at a packet destination, the second set of test packets;

generating RX timestamps for the second set of test packets;

computing, by the packet destination or a packet analyzer, second in-flight metrics for the second set of test packets using the TX and RX timestamps for the second set of test packets, wherein computing the second in-flight metrics includes grouping the second set of test packets based on packet or flow characteristics and generating an average in-flight metric, a minimum in-flight metric, or a maximum in-flight metric for each of a plurality of groups;

evaluating the second LB mechanism using the second in-flight metrics;

analyzing the evaluations of the first LB mechanism and the second LB mechanism; and

reporting, to a test operator, analysis information about the evaluations of the first LB mechanism and the second LB mechanism.

3 . The method of claim 1 wherein evaluating the first LB mechanism includes displaying graphically the in-flight metrics for each of the plurality of groups.

4 . The method of claim 1 wherein computing the first in-flight metrics includes generating the minimum in-flight metric for the first group by computing the packet in-flight times for the test packets of the first group and selecting a lowest packet in-flight time of the packet in-flight times computed for the test packets of the first group.

5 . The method of claim 1 wherein computing the first in-flight metrics includes generating the maximum in-flight metric for the first group by computing the packet in-flight times for the test packets of the first group and selecting a highest packet in-flight time of the packet in-flight times computed for the test packets of the first group.

6 . The method of claim 1 wherein computing the first in-flight metrics includes generating the average in-flight metric for the first group by computing the packet in-flight times for the test packets of the first group and averaging the packet in-flight times computed for the test packets of the first group.

7 . The method of claim 6 wherein computing the packet in-flight times for the test packets of the first group includes subtracting a first TX timestamp associated with a first test packet of the test packets of the first group from a first RX timestamp associated with the first test packet.

8 . The method of claim 7 wherein the first TX timestamp is stored in a packet payload or a packet header of the first test packet and the first RX timestamp is stored in the first test packet or a data store.

9 . A system for evaluating a load balancing (LB) mechanism, the system comprising:

at least one processor;

a memory; and

a test system implemented using the at least one processor and the memory, the test system configured for:

sending, via one or more paths comprising network nodes, a first set of test packets comprising transmit (TX) timestamps for evaluating a first LB mechanism;

receiving, at a packet destination, the first set of test packets;

generating receive (RX) timestamps for the first set of test packets;

computing, by the packet destination or a packet analyzer, first in-flight metrics for the first set of test packets using the TX and RX timestamps, grouping the first set of test packets based on packet or flow characteristics and generating an average in-flight metric, a minimum in-flight metric, or a maximum in-flight metric for each of a plurality of groups, wherein grouping the first set of test packets based on the packet or flow characteristics includes assigning, to a first group, test packets in the first set of test packets that are part of packet flows of a first flow type and assigning, to a second group, test packets in the first set of test packets that are part of packet flows of a second flow type different from the first flow type and computing the first in-flight metrics includes computing the first in-flight metrics for the first and second flow types, wherein the first flow type includes packet flows of a first number of packets and a first packet size distribution and the second flow type includes packet flows of a second number of packets greater than the first number of packets and a second packet size distribution with packets of different sizes than the first packet size distribution; and

evaluating the first LB mechanism using the first in-flight metrics, wherein evaluating the first LB mechanism using the first in-flight metrics includes comparing the first in-flight metrics computed for first flow type with the first in-flight metrics computed for the second flow type, wherein comparing the in-flight metrics for the first flow type with the in-flight metrics for the second flow type includes comparing in-flight times for the packet flows with the first number of packets and the first packet size distribution with in-flight times for the packet flows with the second number of packets and the second packet size distribution.

10 . The system of claim 9 wherein the test system is configured for:

sending, via one or more paths comprising network nodes, a second set of test packets comprising TX timestamps for evaluating a second LB mechanism;

receiving, at a packet destination, the second set of test packets;

generating RX timestamps for the second set of test packets;

computing, by the packet destination or a packet analyzer, second in-flight metrics for the second set of test packets using the TX and RX timestamps for the second set of test packets, wherein computing the second in-flight metrics includes grouping the second set of test packets based on packet or flow characteristics and generating an average in-flight metric, a minimum in-flight metric, or a maximum in-flight metric for each of a plurality of groups;

evaluating the second LB mechanism using the second in-flight metrics;

analyzing the evaluations of the first LB mechanism and the second LB mechanism; and

reporting, to a test operator, analysis information about the evaluations of the first LB mechanism and the second LB mechanism.

11 . The system of claim 9 wherein evaluating the LB mechanism includes displaying graphically the first in-flight metrics for each of the plurality of groups.

12 . The system of claim 9 wherein computing the first in-flight metrics includes generating the minimum in-flight metric for the first group by computing the packet in-flight times for the test packets of the first group and selecting a lowest packet in-flight time of the packet in-flight times computed for the test packets of the first group.

13 . The system of claim 9 wherein computing the first in-flight metrics includes generating the maximum in-flight metric for the first group by computing the packet in-flight times for the test packets of the first group and selecting a highest packet in-flight time of the packet in-flight times computed for the test packets of the first group.

14 . The system of claim 9 wherein computing the first in-flight metrics includes generating the average in-flight metric for the first group by computing the packet in-flight times for the test packets of the first group and averaging the packet in-flight times computed for the test packets of the first group.

15 . The system of claim 14 wherein computing the packet in-flight time for the test packets of the first group includes subtracting a first TX timestamp associated with a first test packet of the test packets of the first group from a first RX timestamp associated with the first test packet.

16 . The system of claim 15 wherein the first TX timestamp is stored in a packet payload or a packet header of the first test packet and the first RX timestamp is stored in the first test packet or a data store.

17 . The system of claim 9 wherein the first LB mechanism includes an implementation of a round robin algorithm, a least connections algorithm, a weighted round robin algorithm, a least response time algorithm, an internet protocol (IP) hash algorithm, a flow-based LB algorithm, a flowlet-based LB algorithm, a random packet spray (RPS) algorithm, or a randomized LB algorithm.

18 . A non-transitory computer readable medium having stored thereon executable instructions embodied in the non-transitory computer readable medium that when executed by at least one processor of a test system cause the test system to perform steps comprising:

sending, via one or more paths comprising network nodes, a first set of test packets comprising transmit (TX) timestamps for evaluating a first load balancing (LB) mechanism;

receiving, at a packet destination, the first set of test packets;

generating receive (RX) timestamps for the first set of test packets;

computing, by the packet destination or a packet analyzer, in-flight metrics for the first set of test packets using the TX and RX timestamps, grouping the first set of test packets based on packet or flow characteristics and generating an average in-flight metric, a minimum in-flight metric, or a maximum in-flight metric for each of a plurality of groups, wherein grouping the first set of test packets based on the packet or flow characteristics includes assigning, to a first group, test packets in the first set of test packets that are part of packet flows of a first flow type and assigning, to a second group, test packets in the first set of test packets that are part of packet flows of a second flow type different from the first flow type and computing the in-flight metrics includes computing the in-flight metrics for the first and second flow types, wherein the first flow type includes packet flows of a first number of packets and a first packet size distribution and the second flow type includes packet flows of a second number of packets greater than the first number of packets and a second packet size distribution with packets of different sizes than the first packet size distribution; and

evaluating the first LB mechanism using the in-flight metrics, wherein evaluating the first LB mechanism using the in-flight metrics includes comparing the in-flight metrics computed for first flow type with the in-flight metrics computed for the second flow type, wherein comparing the in-flight metrics for the first flow type with the in-flight metrics for the second flow type includes comparing in-flight times for the packet flows with the first number of packets and the first packet size distribution with in-flight times for the packet flows with the second number of packets and the second packet size distribution.