Dynamically adaptive, resource aware system and method for scheduling
View Patent ↗The following relates generally to computer system efficiency improvements. Broadly, systems and methods are disclosed that improve efficiency in a cluster of nodes by efficient processing of tasks among nodes in the cluster of nodes. Assignment of tasks to compute nodes may be based on learned CPU capabilities and I/O bandwidth capabilities of the compute nodes in the cluster.
1. A system for scheduling jobs, said system comprising:
a plurality of interconnected compute nodes defining a cluster of compute nodes, the cluster including a NameNode and a multitude of DataNodes;
the NameNode including at least one processor programmed to:
learn central processing unit (CPU) capabilities and disk input/output (I/O) bandwidth capabilities of compute nodes in the cluster; and
schedule execution of a plurality of tasks on compute nodes of the cluster based on the learned CPU capabilities and I/O bandwidth capabilities of compute nodes of the cluster; and
wherein the system further includes an optimal number of containers determined for compute nodes in the cluster;
wherein the scheduling is further based on the determined optimal number of containers; and
wherein the optimal number of containers is at least in part determined by use of the equation:
X
k
*
=
argmax
x
k
∑
i
x
ki
ρ
(
θ
j
,
π
k
,
L
k
)
where X k * denotes a vector of containers, ρ denotes a predicted task completion time, θ j denotes a resource requirement, π k denotes a resource capability, and L k denotes a load on a node.
2. The system of claim 1 , wherein the learning is performed during an offline training phase.
3. The system of claim 2 , wherein the learning is completed before bringing the cluster online.
4. The system of claim 1 , wherein:
the execution of a plurality of tasks is scheduled such that a throughput of the cluster is increased; and
the throughput is determined based on sizes of tasks of the plurality of tasks.
5. A method for scheduling jobs in a cluster of compute nodes including a NameNode and a multitude of DataNodes, said method comprising:
learning central processing unit (CPU) capabilities and disk input/output (I/O) bandwidth capabilities of compute nodes in the cluster;
scheduling execution of a plurality of tasks on compute nodes of the cluster based on the learned CPU capabilities and I/O bandwidth capabilities of compute nodes of the cluster;
determining an optimal number of containers for compute nodes in the cluster;
wherein the scheduling is further based on the determined optimal number of containers; and
wherein the optimal number of containers is at least in part determined by use of the equation:
X
k
*
=
argmax
x
k
∑
i
x
ki
ρ
(
θ
j
,
π
k
,
L
k
)
where X k * denotes a vector of containers, ρ denotes a predicted task completion time, θ j denotes a resource requirement, π k denotes a resource capability, and L k denotes a load on a node.
6. The method of claim 5 , wherein the learning is performed during an offline training phase.
7. The method of claim 6 , wherein the learning is completed before bringing the cluster online.
8. The method of claim 5 , wherein:
the execution of a plurality of tasks is scheduled such that a throughput of the cluster is increased; and
the throughput is determined based on sizes of tasks of the plurality of tasks.
9. A system for scheduling jobs, said system comprising:
a plurality of interconnected compute nodes defining a cluster of compute nodes, the cluster including a NameNode and a multitude of DataNodes;
at least one processor programmed to:
predict a relative resource usage of map tasks by observing a resource usage of a first map task;
automatically train an adjustable system of task completion time;
learn central processing unit (CPU) capabilities and disk input/output (I/O) bandwidth capabilities of compute nodes in the cluster; and
schedule execution of a plurality of tasks on compute nodes of the cluster based on the learned CPU capabilities and I/O bandwidth capabilities of compute nodes of the cluster; and
wherein the system further includes an optimal number of containers determined for compute nodes in the cluster;
wherein the scheduling is further based on the determined optimal number of containers; and
wherein the optimal number of containers is at least in part determined by use of the equation:
X
k
*
=
argmax
x
k
∑
i
x
ki
ρ
(
θ
j
,
π
k
,
L
k
)
where X k * denotes a vector of containers, ρ denotes a predicted task completion time, θ j denotes a resource requirement, π k denotes a resource capability, and L k denotes a load on a node.
10. The system of claim 9 , wherein the learning is performed during an offline training phase.
11. The system of claim 10 , wherein the learning is completed before bringing the cluster online.
12. The system of claim 9 , wherein:
the execution of a plurality of tasks is scheduled such that a throughput of the cluster is increased; and
the throughput is determined based on sizes of tasks of the plurality of tasks.