Dynamically adjusting performance tuning parameters in a network system
A container-based orchestration system includes a master node and a plurality of worker nodes. The master node can receive, from each agent executing on a corresponding worker node, node characteristics associated with the worker node. The master node can determine, for each worker node, one or more parameters corresponding to the node characteristics associated with the corresponding worker node and a node profile of the worker node and provide the parameters to the agent executing on the corresponding worker node. The agent configures the worker node in accordance with the parameters. In response to receiving a request to deploy a pod to a worker node, the master node can select a worker node to receive the pod based on the node characteristics and the pod characteristics. The agent can configure the selected worker node to execute workloads of the pod in accordance with the one or more parameters.
1. A method comprising:
receiving, by a master node of a cluster and from each agent executing on a corresponding worker node of a plurality of worker nodes of the cluster, a plurality of node characteristics associated with the corresponding worker node, wherein the cluster is a container-based orchestration system cluster, and wherein the master node manages orchestration of workloads on the worker nodes of the cluster;
for each worker node of the plurality of worker nodes, determining, by the master node, one or more configuration parameters corresponding to the plurality of node characteristics associated with the corresponding worker node and a node profile of the corresponding worker node, and providing, by the master node to the corresponding agent executing on the worker node, the one or more configuration parameters, to enable the corresponding agent to configure the worker node in accordance with the one or more configuration parameters;
receiving, by the master node, a request to deploy a pod having a pod specification describing one or more pod characteristics;
subsequent to each of the plurality of worker nodes being configured by the corresponding agent in accordance with the configuration parameters, selecting, by the master node and based on the plurality of node characteristics and the one or more pod characteristics of the pod specification, a selected worker node of the plurality of worker nodes on which to deploy the pod; and
scheduling, by the master node, the pod on the selected worker node.
2. The method of claim 1 , wherein selecting the selected worker node comprises selecting the worker node based on that worker node being configured with a virtual router that is operating in a virtual router mode indicated by the one or more pod characteristics of the pod specification.
3. The method of claim 2 , wherein the virtual router mode comprises a data plane development kit (DPDK) mode.
4. The method of claim 1 , wherein the one or more configuration parameters include one or more of central processing unit (CPU) pinning parameters, non-uniform memory access (NUMA) binding parameters, and virtual router configuration parameters.
5. The method of claim 1 , further comprising:
providing, by the master node to the corresponding agent of the selected worker node, one or more central processing unit (CPU) assignments for the pod, the CPU assignments based, at least in part, on one or more of the node characteristics of the selected worker node.
6. The method of claim 1 , wherein selecting the selected worker node to receive the pod comprises selecting the selected worker node based on a best match between the plurality of node characteristics and the one or more pod characteristics of the pod specification.
7. The method of claim 1 , further comprising:
receiving, by the master node and from the agent after initialization of the worker node, resource usage information associated with the worker node,
wherein selecting the selected worker node on which to deploy the pod is further based on the resource usage information.
8. The method of claim 7 , further comprising:
determining, by the master node and based on the resource usage information, a number of currently available CPUs on the worker node; and
determining, based on the one or more pod characteristics of the pod specification, a number of desired CPUs for the pod,
wherein selecting the selected worker node on which to deploy the pod is further based on the currently available CPUs on the selected worker node and the desired number of CPUs for the pod.
9. The method of claim 8 , further comprising:
determining, by the master node and based on the plurality of node characteristics, a first pool of reserved CPUs and a second pool of non-reserved CPUs,
wherein the currently available CPUs comprise available CPUs from the first pool of reserved CPUs.
10. The method of claim 1 , wherein the node profile comprises a first node profile and wherein the worker node comprises a first worker node, wherein the method further comprises:
receiving, by the master node via a user interface device, an indication of a profile change for the worker node from the first node profile to a second node profile;
migrating, by the master node and based on receiving the indication of the profile change, workloads from the first worker node to a second worker node associated with the first node profile; and
configuring, by the master node and subsequent to migrating the workloads, the first worker node in accordance with the second node profile.
11. A worker node comprising:
one or more processors;
memory coupled to the one or more processors;
an orchestration agent; and
a performance agent executable by the one or more processors, the performance agent configured to:
monitor resource usage on the worker node,
report the resource usage to a master node configured to orchestrate workloads on a plurality of worker nodes including the worker node, the master node being in a same cluster of nodes as the worker node, wherein the cluster is a container-based orchestration system cluster,
receive one or more configuration parameters from the master node, the one or more configuration parameters determined by the master node based on one or more node characteristics corresponding to the worker node of the plurality of worker nodes,
configure the worker node in accordance with the one or more configuration parameters from the master node,
subsequent to configuration of the worker node in accordance with the one or more configuration parameters, receive an indication from the orchestration agent that the worker node has been selected, by the master node based on the one or more configuration parameters and the resource usage, for deploying a pod having one or more pod characteristics defined by a pod specification, and
configure, based on the indication, the worker node to execute workloads of the pod such that the worker node executes the workloads of the pod in accordance with the one or more configuration parameters.
12. The worker node of claim 11 , wherein the one or more processors comprises a plurality of processors, the worker node further comprising:
a virtual router configured to execute on a first subset of the plurality of processors;
wherein the performance agent is further configured to:
based on a determination that one or more configuration parameters indicate that the virtual router is configured as a data plane development kit (DPDK) virtual router, configure the workloads to execute on a same non-uniform memory access (NUMA) node of the worker node as the DPDK virtual router.
13. The worker node of claim 12 , wherein the performance agent is further configured to execute the workloads on a second subset of the plurality of processors, wherein the first subset does not have any processors in common with the second subset.
14. The worker node of claim 11 , wherein the performance agent is configured to, at an initialization time of the worker node, collect the one or more node characteristics of the worker node and provide the one or more node characteristics to the master node.
15. A container-based orchestration system comprising:
a master node; and
a plurality of worker nodes communicatively coupled to the master node, wherein the master node manages orchestration of workloads on each of the plurality of worker nodes,
wherein the master node comprises memory storing instructions that, when executed by processing circuitry of the master node, causes the master node to:
receive, from each agent executing on a corresponding worker node of the plurality of worker nodes, a plurality of node characteristics associated with the corresponding worker node,
for each worker node of the plurality of worker nodes, determine one or more configuration parameters corresponding to the plurality of node characteristics associated with the corresponding worker node and a node profile of the corresponding worker node, and provide, to the corresponding agent executing on the worker node, the one or more configuration parameters, to enable the corresponding agent to configure the worker node in accordance with the one or more configuration parameters;
receive a request to deploy a pod having a pod specification describing one or more pod characteristics,
subsequent to each of the plurality of worker nodes being configured by the corresponding agent in accordance with the configuration parameters, select a selected worker node of the plurality of worker nodes on which to deploy the pod based on the plurality of node characteristics and the one or more pod characteristics of the pod specification, and
schedule the pod on the selected worker node, and wherein the agent of the selected worker node is configured to:
receive an indication that the pod is scheduled to the selected worker node, and
configure, based on the indication, the selected worker node to execute workloads of the pod such that the worker node executes the workloads of the pod in accordance with the one or more configuration parameters.
16. The container-based orchestration system of claim 15 , wherein the selected worker node further comprises:
a virtual router,
wherein the agent of the selected worker node is further configured to:
based on to a determination that one or more configuration parameters indicate that the virtual router is configured as a data plane development kit (DPDK) virtual router, configure the workloads to execute on a same non-uniform memory access (NUMA) node of the selected worker node as the DPDK virtual router.
17. The container-based orchestration system of claim 16 , wherein the virtual router executes on a first subset of a plurality of processors of the selected worker node, wherein the agent is further configured to execute the workloads on a second subset of the plurality of processors, wherein the first subset does not have any processors in common with the second subset.
18. The container-based orchestration system of claim 15 ,
wherein the agent is further configured to provide resource usage information associated with the selected worker node; and
wherein the master node selects the selected worker node on which to deploy the pod further based on the resource usage information.
19. The container-based orchestration system of claim 18 , wherein the master node is further configured to:
determine, based on the resource usage information, a number of currently available CPUs on the selected worker node; and
determine, based on the one or more pod characteristics, a number of desired CPUs for the pod,
wherein the master node selects the selected worker node on which to deploy the pod further based on the currently available CPUs on the selected worker node and the desired number of CPUs for the pod.
20. The container-based orchestration system of claim 15 , wherein the worker node comprises a first worker node and wherein the agent is further configured to:
based on a determination that a workload of the first worker node is not complying with a NUMA node assignment, report the non-compliance to the master node, wherein based on receiving the report of the non-compliance, the master node reschedules the pod having the workload to a second worker node having a same node profile as the first worker node.