Capacity-aware layer-4 load balancer

1. A method of performing capacity-aware load balancing across a set of compute nodes, the method comprising:

from the set of compute nodes, identifying (i) a first subset of compute nodes comprising compute nodes that have a latency that is higher than an average latency computed for the set of compute nodes and (ii) a second subset of compute nodes comprising compute nodes that have a latency that is lower than the average latency computed for the set of compute nodes;

for each particular compute node in the first subset of compute nodes, assigning, to the particular compute node, a reduced weight value based on a target latency computed for the set of compute nodes, and reducing data traffic forwarded by a set of load balancers to the first subset of compute nodes;

increasing one or more weight values assigned to one or more of the compute nodes in the second subset of compute nodes and increasing data traffic forwarded by the set of load balancers to the second subset of compute nodes;

computing, based on the reduced weight values assigned to the compute nodes in the first subset of compute nodes, an excess weight value to be redistributed across the second subset of compute nodes, wherein said increasing comprises redistributing the computed excess weight value across the second subset of compute nodes; and

distributing the reduced and increased weight values to the set of load balancers.

2. The method of claim 1 , wherein the average latency computed for the set of compute nodes is computed by (i) computing, for each compute node in the set of compute nodes, a rolling average latency for the compute node based on a set of ten most recent recorded latencies for the compute node at a current weight value assigned to the compute node, and (ii) using the computed rolling average latencies for each compute node in the set of compute nodes to compute the average latency for the set of compute nodes.

3. The method of claim 1 , wherein the target latency is equal to the average latency computed for the set of compute nodes.

4. The method of claim 1 , wherein the excess weight value comprises a sum of excess weight values computed for each particular compute node in the first subset of compute nodes by subtracting the reduce weight value of the particular compute node from the weight value that was assigned to the particular compute node when the particular compute node had the latency higher than the average latency.

5. The method of claim 1 , wherein the weight value assigned to each compute node in the set of compute nodes corresponds to a percentage of traffic that is directed toward the compute node.

6. The method of claim 5 , wherein the weight value assigned to each compute node in the set of compute nodes further corresponds to a computing capacity of the compute node, wherein compute nodes having higher computing capacities are assigned higher weight values and compute nodes having lower computing capacities are assigned lower weight values.

7. The method of claim 1 , wherein each compute node in the set of compute nodes comprises one of a virtual machine (VM), a container, and a physical computer.

8. The method of claim 1 , wherein each compute node in the set of compute nodes is assigned a direct internet protocol (DIP) address, wherein the set of load balancers receives packets addressed to a virtual internet protocol (VIP) address associated with the set of compute nodes, and performs a load balancing operation to load balance [the] network traffic across the DIP addresses of the set of compute nodes based on the weight values assigned to the set of compute nodes.

9. The method of claim 8 , wherein the load balancing operation is a weighted round-robin operation.

10. A non-transitory machine readable medium storing a program for execution by a set of processing units to perform capacity-aware load balancing across a set of compute nodes, the program comprising sets of instructions for:

from the set of compute nodes, identifying (i) a first subset of compute nodes comprising compute nodes that have a latency that is higher than an average latency computed for the set of compute nodes and (ii) a second subset of compute nodes comprising compute nodes that have a latency that is lower than the average latency computed for the set of compute nodes;

for each particular compute node in the first subset of compute nodes, assigning, to the particular compute node, a reduced weight value based on a target latency computed for the set of compute nodes, and reducing data traffic forwarded by a set of load balancers to the first subset of compute nodes;

increasing one or more weight values assigned to one or more of the compute nodes in the second subset of compute nodes and increasing data traffic forwarded by the set of load balancers to the second subset of compute nodes;

computing, based on the reduced weight values assigned to the compute nodes in the first subset of compute nodes, an excess weight value to be redistributed across the second subset of compute nodes, wherein said increasing comprises redistributing the computed excess weight value across the second subset of compute nodes; and

distributing the reduced and increased weight values to the set of load balancers.

11. The non-transitory machine readable medium of claim 10 , wherein the average latency computed for the set of compute nodes is computed by (i) computing, for each compute node in the set of compute nodes, a rolling average latency for the compute node based on a set of ten most recent recorded latencies for the compute node at a current weight value assigned to the compute node, and (ii) using the computed rolling average latencies for each compute node in the set of compute nodes to compute the average latency for the set of compute nodes.

12. The non-transitory machine readable medium of claim 10 , wherein the target latency is equal to the average latency computed for the set of compute nodes.

13. The non-transitory machine readable medium of claim 10 , wherein the excess weight value comprises a sum of excess weight values computed for each particular compute node in the first subset of compute nodes by subtracting the reduce weight value of the particular compute node from the weight value that was assigned to the particular compute node when the particular compute node had the latency higher than the average latency.

14. The non-transitory machine readable medium of claim 10 , wherein the weight value assigned to each compute node in the set of compute nodes corresponds to a percentage of traffic that is directed toward the compute node.

15. The non-transitory machine readable medium of claim 14 , wherein the weight value assigned to each compute node in the set of compute nodes further corresponds to a computing capacity of the compute node, wherein compute nodes having higher computing capacities are assigned higher weight values and compute nodes having lower computing capacities are assigned lower weight values.

16. The non-transitory machine readable medium of claim 10 , wherein each compute node in the set of compute nodes comprises one of a virtual machine (VM), a container, and a physical computer.

17. The non-transitory machine readable medium of claim 10 , wherein each compute node in the set of compute nodes is assigned a direct internet protocol (DIP) address, wherein the set of load balancers receives packets addressed to a virtual internet protocol (VIP) address associated with the set of compute nodes, and performs a load balancing operation to load balance network traffic across the DIP addresses of the set of compute nodes based on the weight values assigned to the set of compute nodes.

18. The non-transitory machine readable medium of claim 17 , wherein the load balancing operation is a weighted round-robin operation.