Dynamic service rebalancing in network interface cards having processing units
An edge services controller may use a service scheduling algorithm to deploy services on Network Interface Cards (NICs) of a NIC fabric while incrementally scheduling services. The edge services controller may assign services to specific nodes depending on their available resources on these nodes. Available resources may include CPU compute, DPU compute, node bandwidth, etc. The edge services controller may also consider the distance between the services that communicate with each other (i.e., hop count between nodes if two communicating services are placed on separate nodes) and the weight of communication between the services. Two services that communicate heavily with each other may consume more bandwidth, and putting them further apart is more detrimental than keeping them closer to each other, i.e., reducing the hop count between each other depending on the bandwidth consumption due to their inter-service communications.
1. A method comprising:
receiving, at an edge services controller for a network interface card (NIC) fabric comprising NICs coupled by communication links in the NIC fabric, a list of services to be placed at the NICs of the NIC fabric;
determining an initial distribution of service instances of the services to be placed at the NICs considering at least service central processing unit (CPU) utilization of the NICs and service data processing unit (DPU) utilization of the NICs; and
distributing the service instances to the NICs of the NIC fabric according to the initial distribution.
2. The method of claim 1 , wherein determining the initial distribution of service instances comprises:
based on a plurality of candidate distributions satisfying a set of one or more constraints:
evaluating an objective function to determine scores for the candidate distributions based on one or more attributes of the candidate distributions, wherein for each of the candidate distributions:
the attributes of the candidate distributions include bandwidth weights of connections between service instances of the candidate distribution and a number of connection hops between nodes of the candidate distribution; wherein the initial distribution has a lowest score among the candidate distributions.
3. The method of claim 2 , wherein the objective function is a summation of multiplications of bandwidth weights for each pair of service instances by the number of connection hops that separate the pair of service instances in the candidate distributions.
4. The method of claim 1 , further comprising, based on a first set of one or more constraints not being satisfied and a second plurality of candidate distributions satisfying a second set of one or more constraints, evaluating a fallback objective function to determine scores for the second plurality of candidate distributions based on a number of the service instances placed at the NICs, wherein the initial distribution has a highest number of service instances placed at the NICs.
5. The method of claim 1 , further comprising:
receiving, at the edge services controller, an indication of an additional service;
determining, at the edge services controller, an incremental distribution for the additional service; and
distributing one or more instances of the additional service to at least one of the NICs of the NIC fabric according to the incremental distribution.
6. The method of claim 5 , further comprising:
based on a plurality of candidate distributions satisfying a set of one or more constraints, each of the plurality of candidate distributions assigning the additional service to one of the NICS, evaluating an objective function to determine scores for the plurality of candidate distributions based on one or more attributes of the candidate distributions,
wherein for each of the plurality of candidate distributions, the attributes of the candidate distributions of the plurality of candidate distributions include bandwidth weights of connections between pairs of service instances and a number of connection hops between the pairs of service instances, and
wherein the incremental distribution has a lowest score among the plurality of candidate distributions.
7. The method of claim 5 , further comprising, based on a set of one or more constraints not being satisfied by any candidate distribution and one or more candidate distributions satisfying a further set of one or more constraints, selecting one of the further set of one or more candidate distributions as the incremental distribution.
8. The method of claim 1 , further comprising:
determining, at the edge services controller, a rebalanced distribution for the service instances; and
redistributing the service instances to the NICs of the NIC fabric according to the rebalanced distribution.
9. The method of claim 8 , further comprising determining at least one pinned service instance at a specific NIC, wherein redistributing the service instances keeps the at least one pinned service instance at the specific NIC.
10. The method of claim 8 , further comprising, before distributing the service instances to the NICs of the NIC fabric according to the rebalanced distribution, determining that an improvement of the rebalanced distribution is above an objective threshold and determining that utilization of the NIC fabric is below a utilization threshold.
11. An edge services controller comprising:
a memory; and
a processor implemented in circuitry and configured to:
receive a list of services to be placed at network interface cards (NICs) of a NIC fabric;
determine an initial distribution of service instances of the services to be placed at the NICs considering at least service central processing unit (CPU) utilization of the NICs and service data processing unit (DPU) utilization of the NICs; and
distribute the service instances to the NICs of the NIC fabric according to the initial distribution.
12. The edge services controller of claim 11 , wherein to determine the initial distribution of the service instances the processor is further configured to:
based on a plurality of candidate distributions satisfying a set of one or more constraints:
evaluate an objective function to determine scores for the candidate distributions based on one or more attributes of the candidate distributions, wherein for each of the candidate distributions:
the attributes of the candidate distributions include bandwidth weights of connections between service instances of the candidate distribution and a number of connection hops between nodes of the candidate distribution; wherein the initial distribution has a lowest score among the candidate distributions.
13. The edge services controller of claim 12 , wherein the objective function is a summation of a multiplications of bandwidth weights for each pair of service instances by the number of connection hops that separate the pair of service instances in the candidate distributions.
14. The edge services controller of claim 11 , wherein the processing circuitry is further configured to, based on a first set of one or more constraints not being satisfied and a second plurality of candidate distributions satisfying a second set of one or more constraints, evaluate a fallback objective function to determine scores for the second plurality of candidate distributions based on a number of the service instances placed at the NICs, wherein the initial distribution has a highest number of service instances placed at the NICs.
15. The edge services controller of claim 11 , wherein the processing circuitry is further configured to:
receive an indication of an additional service;
determine, an incremental distribution for the additional service; and
distribute one or more instances of the additional service to at least one of the NICs of the NIC fabric according to the incremental distribution.
16. The edge services controller of claim 15 , wherein the processing circuitry is further configured to:
based on a plurality of candidate distributions satisfying a set of one or more constraints, each of the plurality of candidate distributions assigning the additional service to one of the NICS, evaluate an objective function to determine scores for the plurality of candidate distributions based on one or more attributes of the candidate distributions,
wherein for each of the plurality of candidate distributions, the attributes of the candidate distributions of the plurality of candidate distributions include bandwidth weights of connections between pairs of service instances and a number of connection hops between the pairs of service instances, and
wherein the incremental distribution has a lowest score among the plurality of candidate distributions.
17. The edge services controller of claim 15 , wherein the processor is further configured to, based on a set of one or more constraints not being satisfied by any candidate distribution and one or more candidate distributions satisfying a further set of one or more constraints, select one of the further set of one or more candidate distributions as the incremental distribution.
18. The edge services controller of claim 11 , wherein the processing circuitry is further configured to:
determine a rebalanced distribution for the service instances; and
redistribute the service instances to the NICs of the NIC fabric according to the rebalanced distribution.
19. The edge services controller of claim 18 , wherein the processor is further configured to determine at least one pinned service instance at a specific NIC, wherein the processor is configured to redistribute the service instances but keep the at least one pinned service instance at the specific NIC.
20. A system comprising:
a network interface card (NIC) fabric comprising NICs coupled by communication links in a NIC fabric; and
an edge services controller comprising processing circuitry having access to storage media, the edge services controller configured to:
receive a list of services to be placed at the NICs of the NIC fabric;
determine an initial distribution of service instances of the services to be placed at the NICs considering at least service central processing unit (CPU) utilization of the NICs and service data processing unit (DPU) utilization of the NICs; and
distribute the service instances to the NICs of the NIC fabric according to the initial distribution.