Job scheduler for distributed systems using pervasive state estimation with modeling of capabilities of compute nodes
The following relates generally to computer system efficiency improvements. Broadly, systems and methods are disclosed that improve efficiency in a cluster of nodes by efficient processing of tasks among nodes in a cluster of nodes. Initially, tasks may be scheduled on the nodes in the cluster of nodes. Following that, state information may be received, and a determination may be made as to if tasks should be rescheduled.
1. A system for scheduling jobs, said system comprising:
a plurality of interconnected compute nodes defining a cluster of compute nodes, the cluster including a NameNode and a multitude of DataNodes;
the NameNode including at least one processor programmed to:
schedule execution of a plurality of tasks on compute nodes of the cluster;
receive state information variables of the compute nodes;
determine if the state variables indicate that a current model of the capabilities of the compute nodes are no longer correct; and
if the state variables show that the current model is no longer correct, schedule a specific plurality of tasks onto the compute nodes for which a model discrepancy has been detected in order to obtain specific observations that allow for efficiently updating the model of that compute node;
wherein the NameNode has no offline learning phase prior to the scheduling execution of the plurality of tasks;
wherein the at least one processor is further programmed to:
if the plurality of interconnected compute nodes are in a steady state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that maximizes throughput; and
if the plurality of interconnected compute nodes are in an uncertain state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that gives information about the system to maximize future production.
2. The system of claim 1 , wherein the scheduling of the specific plurality of tasks onto the compute nodes for which the model discrepancy has been detected is based on:
E [ (Θ t ,Π)]= E Θ t [U (Θ t ,Π)]+β· I (Θ t+1 ;O |Π)
wherein:
E is an objective function;
( ) is a cost function;
Θ t represents the variable capabilities of the compute node state at time t;
Π is a policy for job allocation;
U( ) is a production utility function;
β is a linear multiple;
I( ) is a state of the cluster; and
O|Π is an observation for a given workload.
3. The system of claim 1 , wherein the at least one processor is further programmed to:
if the state variables show that the current model is correct, schedule tasks on the compute nodes so that throughput is maximized.
4. The system of claim 1 , wherein:
each DataNode of the multitude of DataNodes has a base capability component that is fixed over time, and dynamic component that changes over time; and
the base capability component and the dynamic component are used in the scheduling execution of the plurality of tasks on the compute nodes of the cluster.
5. The system of claim 1 , wherein the scheduling of execution of the plurality of tasks on compute nodes of the cluster uses a model that has a computation component and a disk input/output (I/O) component.
6. A server, comprising:
a memory with executable instructions stored thereon; and
a processor configured to access said memory to execute said instructions and to:
schedule execution of a plurality of tasks on compute nodes of a cluster of compute nodes;
receive state information variables of the compute nodes;
determine if the state variables indicate that a current model of the capabilities of the compute nodes are no longer correct; and
if the state variables show that the current model is no longer correct, schedule a specific plurality of tasks onto the compute nodes for which a model discrepancy has been detected in order to obtain specific observations that allow for efficiently updating the model of that compute node;
wherein there is no offline learning phase prior to the scheduling execution of the plurality of tasks;
wherein the processor is further programmed to:
if the plurality of interconnected compute nodes are in a steady state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that maximizes throughput; and
if the plurality of interconnected compute nodes are in an uncertain state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that gives information about the system to maximize future production.
7. The server of claim 6 , wherein the scheduling of the specific plurality of tasks onto the compute nodes for which the model discrepancy has been detected is based on:
E [ (Θ t ,Π)]= E Θ t [U (Θ t ,Π)]+β· I (Θ t+1 ,O |Π)
wherein:
E is an objective function;
( ) is a cost function;
Θ t is a represents the variable capabilities of the compute node state at time t;
Π is a policy for job allocation;
U( ) is a production utility function;
β is a linear multiple;
I( ) is a state of the cluster; and
O|Π is an observation for a given workload.
8. The server of claim 6 , wherein the processor is further programmed to:
if the state variables show that the current model is correct, schedule tasks on the compute nodes so that throughput is maximized.
9. The server of claim 6 , wherein:
each DataNode of the multitude of DataNodes has a base capability component that is fixed over time, and dynamic component that changes over time; and
the base capability component and the dynamic component are used in the scheduling execution of the plurality of tasks on the compute nodes of the cluster.
10. The server of claim 6 , wherein the scheduling of execution of the plurality of tasks on compute nodes of the cluster uses a model that has a computation component and a disk input/output (I/O) component.
11. A method for scheduling jobs in a cluster of compute nodes including a NameNode and a multitude of DataNodes, said method performed by one or more processors, said method comprising:
schedule execution of a plurality of tasks on compute nodes of the cluster;
receive state information variables of the compute nodes;
determine if the state variables indicate that a current model of the capabilities of the compute nodes are no longer correct; and
in response to the state variables showing that the current model is no longer correct, schedule a specific plurality of tasks onto the compute nodes for which a model discrepancy has been detected in order to obtain specific observations that allow for efficiently updating the model of that compute node;
wherein there is no offline learning phase prior to the scheduling execution of the plurality of tasks;
wherein the method further comprises:
in response to the plurality of interconnected compute nodes being in a steady state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that maximizes throughput; and
in response to the plurality of interconnected compute nodes being in an uncertain state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that gives information about the system to maximize future production.
12. The method of claim 11 , wherein the scheduling of the specific plurality of tasks onto the compute nodes for which the model discrepancy has been detected is based on:
E [ (Θ t ,Π)]= E Θ t [U (Θ t ,Π)]+β· I (Θ t+1 ,O |Π)
wherein:
E is an objective function;
( ) is a cost function;
Θ t represents the variable capabilities of the compute node state at time t;
Π is a policy for job allocation;
U( ) is a production utility function;
β is a linear multiple;
I( ) is a state of the cluster; and
O|Π is an observation for a given workload.
13. The method of claim 11 , wherein the method further comprises:
in response to the state variables showing that the current model is correct, schedule tasks on the compute nodes so that throughput is maximized.
14. The method of claim 11 , wherein:
each DataNode of the multitude of DataNodes has a base capability component that is fixed over time, and dynamic component that changes over time; and
the base capability component and the dynamic component are used in the scheduling execution of the plurality of tasks on the compute nodes of the cluster.
15. A system for scheduling jobs, said system comprising:
a plurality of interconnected compute nodes defining a cluster of compute nodes, the cluster including a NameNode and a multitude of DataNodes;
the NameNode including at least one processor programmed to:
schedule execution of a plurality of tasks on compute nodes of the cluster;
receive state information variables of the compute nodes;
determine if the state variables indicate that a current model of the capabilities of the compute nodes are no longer correct; and
if the state variables show that the current model is no longer correct, schedule a specific plurality of tasks onto the compute nodes for which a model discrepancy has been detected in order to obtain specific observations that allow for efficiently updating the model of that compute node;
wherein:
the NameNode has no offline learning phase prior to the scheduling execution of the plurality of tasks;
each DataNode of the multitude of DataNodes has a base capability component that is fixed over time, and dynamic component that changes over time; and
the base capability component and the dynamic component are used in the scheduling execution of the plurality of tasks on the compute nodes of the cluster.
16. A system for scheduling jobs, said system comprising:
a plurality of interconnected compute nodes defining a cluster of compute nodes, the cluster including a NameNode and a multitude of DataNodes;
the NameNode including at least one processor programmed to:
schedule execution of a plurality of tasks on compute nodes of the cluster;
receive state information variables of the compute nodes;
determine if the state variables indicate that a current model of the capabilities of the compute nodes are no longer correct;
if the state variables show that the current model is no longer correct, schedule a specific plurality of tasks onto the compute nodes for which a model discrepancy has been detected in order to obtain specific observations that allow for efficiently updating the model of that compute node;
if the plurality of interconnected compute nodes are in a steady state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that maximizes throughput; and
if the plurality of interconnected compute nodes are in an uncertain state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that gives information about the system to maximize future production;
wherein:
each DataNode of the multitude of DataNodes has a base capability component that is fixed over time, and dynamic component that changes over time; and
the base capability component and the dynamic component are used in the scheduling execution of the plurality of tasks on the compute nodes of the cluster.
17. The system of claim 16 , wherein the NameNode has no offline learning phase prior to the scheduling execution of the plurality of tasks.
18. A system for scheduling jobs, said system comprising:
a plurality of interconnected compute nodes defining a cluster of compute nodes, the cluster including a NameNode and a multitude of DataNodes;
the NameNode including at least one processor programmed to:
schedule execution of a plurality of tasks on compute nodes of the cluster;
receive state information variables of the compute nodes;
determine if the state variables indicate that a current model of the capabilities of the compute nodes are no longer correct; and
if the state variables show that the current model is no longer correct, schedule a specific plurality of tasks onto the compute nodes for which a model discrepancy has been detected in order to obtain specific observations that allow for efficiently updating the model of that compute node;
wherein:
each DataNode of the multitude of DataNodes has a base capability component that is fixed over time, and dynamic component that changes over time;
the base capability component and the dynamic component are used in the scheduling execution of the plurality of tasks on the compute nodes of the cluster; and
the scheduling of execution of the plurality of tasks on compute nodes of the cluster uses a model that has a computation component and a disk input/output (I/O) component.
19. The system of claim 18 , wherein the at least one processor is further programmed to:
if the plurality of interconnected compute nodes are in a steady state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that maximizes throughput; and
if the plurality of interconnected compute nodes are in an uncertain state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that gives information about the system to maximize future production.
20. A server, comprising:
a memory with executable instructions stored thereon; and
a processor configured to access said memory to execute said instructions and to:
schedule execution of a plurality of tasks on compute nodes of a cluster of compute nodes;
receive state information variables of the compute nodes;
determine if the state variables indicate that a current model of the capabilities of the compute nodes are no longer correct; and
if the state variables show that the current model is no longer correct, schedule a specific plurality of tasks onto the compute nodes for which a model discrepancy has been detected in order to obtain specific observations that allow for efficiently updating the model of that compute node;
wherein:
there is no offline learning phase prior to the scheduling execution of the plurality of tasks;
each DataNode of the multitude of DataNodes has a base capability component that is fixed over time, and dynamic component that changes over time; and
the base capability component and the dynamic component are used in the scheduling execution of the plurality of tasks on the compute nodes of the cluster.
21. A server, comprising:
a memory with executable instructions stored thereon; and
a processor configured to access said memory to execute said instructions and to:
schedule execution of a plurality of tasks on compute nodes of a cluster of compute nodes;
receive state information variables of the compute nodes;
determine if the state variables indicate that a current model of the capabilities of the compute nodes are no longer correct; and
if the state variables show that the current model is no longer correct, schedule a specific plurality of tasks onto the compute nodes for which a model discrepancy has been detected in order to obtain specific observations that allow for efficiently updating the model of that compute node;
if the plurality of interconnected compute nodes are in a steady state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that maximizes throughput; and
if the plurality of interconnected compute nodes are in an uncertain state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that gives information about the system to maximize future production;
wherein:
each DataNode of the multitude of DataNodes has a base capability component that is fixed over time, and dynamic component that changes over time; and
the base capability component and the dynamic component are used in the scheduling execution of the plurality of tasks on the compute nodes of the cluster.
22. The server of claim 21 , wherein there is no offline learning phase prior to the scheduling execution of the plurality of tasks.
23. A server, comprising:
a memory with executable instructions stored thereon; and
a processor configured to access said memory to execute said instructions and to:
schedule execution of a plurality of tasks on compute nodes of a cluster of compute nodes;
receive state information variables of the compute nodes;
determine if the state variables indicate that a current model of the capabilities of the compute nodes are no longer correct; and
if the state variables show that the current model is no longer correct, schedule a specific plurality of tasks onto the compute nodes for which a model discrepancy has been detected in order to obtain specific observations that allow for efficiently updating the model of that compute node;
wherein:
each DataNode of the multitude of DataNodes has a base capability component that is fixed over time, and dynamic component that changes over time;
the base capability component and the dynamic component are used in the scheduling execution of the plurality of tasks on the compute nodes of the cluster; and
the scheduling of execution of the plurality of tasks on compute nodes of the cluster uses a model that has a computation component and a disk input/output (I/O) component.
24. The server of claim 23 , wherein the processor is further programmed to:
if the plurality of interconnected compute nodes are in a steady state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that maximizes throughput; and
if the plurality of interconnected compute nodes are in an uncertain state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that gives information about the system to maximize future production.
25. A method for scheduling jobs in a cluster of compute nodes including a NameNode and a multitude of DataNodes, said method performed by one or more processors, said method comprising:
schedule execution of a plurality of tasks on compute nodes of the cluster;
receive state information variables of the compute nodes;
determine if the state variables indicate that a current model of the capabilities of the compute nodes are no longer correct; and
in response to the state variables showing that the current model is no longer correct, schedule a specific plurality of tasks onto the compute nodes for which a model discrepancy has been detected in order to obtain specific observations that allow for efficiently updating the model of that compute node;
wherein:
there is no offline learning phase prior to the scheduling execution of the plurality of tasks;
each DataNode of the multitude of DataNodes has a base capability component that is fixed over time, and dynamic component that changes over time; and
the base capability component and the dynamic component are used in the scheduling execution of the plurality of tasks on the compute nodes of the cluster.
26. A method for scheduling jobs in a cluster of compute nodes including a NameNode and a multitude of DataNodes, said method performed by one or more processors, said method comprising:
schedule execution of a plurality of tasks on compute nodes of the cluster;
receive state information variables of the compute nodes;
determine if the state variables indicate that a current model of the capabilities of the compute nodes are no longer correct; and
in response to the state variables showing that the current model is no longer correct, schedule a specific plurality of tasks onto the compute nodes for which a model discrepancy has been detected in order to obtain specific observations that allow for efficiently updating the model of that compute node;
in response to the plurality of interconnected compute nodes being in a steady state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that maximizes throughput; and
in response to the plurality of interconnected compute nodes being in an uncertain state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that gives information about the system to maximize future production;
wherein:
each DataNode of the multitude of DataNodes has a base capability component that is fixed over time, and dynamic component that changes over time; and
the base capability component and the dynamic component are used in the scheduling execution of the plurality of tasks on the compute nodes of the cluster.
27. The method of claim 26 , wherein there is no offline learning phase prior to the scheduling execution of the plurality of tasks.
28. A method for scheduling jobs in a cluster of compute nodes including a NameNode and a multitude of DataNodes, said method performed by one or more processors, said method comprising:
schedule execution of a plurality of tasks on compute nodes of the cluster;
receive state information variables of the compute nodes;
determine if the state variables indicate that a current model of the capabilities of the compute nodes are no longer correct; and
in response to the state variables showing that the current model is no longer correct, schedule a specific plurality of tasks onto the compute nodes for which a model discrepancy has been detected in order to obtain specific observations that allow for efficiently updating the model of that compute node;
wherein:
each DataNode of the multitude of DataNodes has a base capability component that is fixed over time, and dynamic component that changes over time;
the base capability component and the dynamic component are used in the scheduling execution of the plurality of tasks on the compute nodes of the cluster; and
the scheduling of execution of the plurality of tasks on compute nodes of the cluster uses a model that has a computation component and a disk input/output (I/O) component.
29. The method of claim 28 , wherein the method further comprises:
in response to the plurality of interconnected compute nodes being in a steady state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that maximizes throughput; and
in response to the plurality of interconnected compute nodes being in an uncertain state, schedule the execution of a plurality of tasks on compute nodes of the cluster using a policy that gives information about the system to maximize future production.