Systems, methods, and devices for dynamic resource monitoring and allocation in a cluster system
In an embodiment, the systems, methods, and devices disclosed herein comprise a computer resource monitoring and allocation system. In an embodiment, the resource monitoring and allocation system can be configured to allocate computer resources that are available on various nodes of a cluster to specific jobs and/or sub-jobs and/or tasks and/or processes.
1. A hadoop computer cluster comprising:
one or more processors of a master node, wherein the master node comprises a supervisor controller; and
one or more processors of a plurality of computing system nodes, the one or more processors of the plurality of computing system nodes configured to perform computing processes on received tasks, wherein each computing system node comprises an agent controller,
wherein each agent controller is configured to monitor system resources utilization by one or more tasks of the respective computing system node and to transmit information describing the monitored system resources utilization to the supervisor controller,
wherein the supervisor controller is configured to assign an additional task to a first computing system node based on determining that the utilization of electronic random access memory of the first computing system node is below a first threshold level, the determination based at least in part on the information describing the monitored system resources utilization, transmitted from the agent controller of the first computing system node,
wherein the supervisor controller is further configured to reduce the use of the electronic random access memory of the second computing system node by a task assigned to the second computing system node based on determining that utilization of the electronic random access memory of the second computing system node is above a second threshold level, the determination based at least in part on the information describing the monitored system resources utilization, transmitted from the agent controller of the second computing system node, and
wherein the master node and the plurality of computing system nodes include a computer processor and an electronic storage medium.
2. The hadoop computer cluster of claim 1 , wherein the transmission of information from an agent controller to the supervisor controller describing the monitored system resource utilization is periodic.
3. The hadoop computer cluster of claim 2 , wherein the periodic transmission of information from an agent controller to the supervisor controller describing the monitored system resource utilization comprises transmitting in substantially real-time.
4. The hadoop computer cluster of claim 2 , wherein each agent controller periodically transmits the monitored system resources utilization of the respective computing system node to the supervisor controller once every 1 to 5 seconds.
5. The hadoop computer cluster of claim 1 , wherein reducing the use of the electronic random access memory of the second computing system node by a task assigned to the second computing system node comprises killing that task assigned to the second computing system node.
6. The hadoop computer cluster of claim 1 , wherein each computing system node is configured to perform computing processes on a received task corresponding to a container associated with the respective computing system node, wherein each container corresponds to set of resources on the respective computing system node for processing one or more tasks.
7. The hadoop computer cluster of claim 6 , wherein the supervisor controller is configured to assign an additional task to the first computing system node through allocating one or multiple additional containers to the first computing system node.
8. The hadoop computer cluster of claim 6 , wherein reducing the use of the electronic random access memory of the second computing system node by a task assigned to the second computing system node comprises killing the container associated with that task assigned to the second computing system node.
9. The hadoop computer cluster of claim 6 , wherein the master node further comprises a job tracker configured to assign tasks to the plurality of computing system nodes based on the number of containers available at a particular computing system node.
10. The hadoop computer cluster of claim 6 , wherein the supervisor controller assigns the additional task to the first computing system node further after having determined that no containers are available at the first computing system node.
11. A supervisor controller configured to dynamically manage assignment of tasks in a hadoop computer cluster, the supervisor controller comprising:
a management controller interface configured to communicate with a management controller to access data representing an assignment of a plurality of tasks across a plurality of computer system nodes in the computer cluster;
an agent controller interface configured to communicate with an agent controller operating on a first computing system node to receive information describing utilization of system resources on the first computing system node, wherein the agent controller interface is further configured to communicate with an agent controller operating on a second computing system node to receive information describing utilization of system resources on the second computing system node; and
a system resource allocation engine configured to assign an additional task to the first computing system node based on determining that the utilization of electronic random access memory of the first computing system node is below a first threshold level, the determination based at least in part on the information describing utilization of system resources on the first computing system node,
wherein one or more computer processors and one or more electronic storage medium are configured to operate the supervisor controller, and
wherein the system resource allocation engine is further configured to reduce the use of electronic random access memory of the second computing system node by a task assigned to the second computing system node based on determining that the utilization of the electronic random access memory of the second computing system node is above a second threshold level, the determination based at least in part on the information describing utilization of system resources on the second computing system node.
12. The supervisor controller of claim 11 , wherein each agent controller transmits information describing utilization of system resources of the respective computing system node to the supervisor controller periodically.
13. The supervisor controller of claim 12 , wherein the periodic transmission of information from an agent controller to the supervisor controller describing utilization of system resources comprises transmitting in substantially real-time.
14. The supervisor controller of claim 12 , wherein each agent controller periodically transmits information describing utilization of system resources of the respective computing system node to the supervisor controller once every 1 to 5 seconds.
15. The supervisor controller of claim 11 , wherein reducing the use of the electronic random access memory of the second computing system node by a task assigned to the second computing system node comprises killing that task assigned to the second computing system node.
16. The supervisor controller of claim 11 , wherein each computing system node is configured to perform computing processes on a received task within a container on the respective computing system node, wherein each container corresponds to set of resources on the respective computing system node for processing one or more tasks.
17. The supervisor controller of claim 16 , wherein the supervisor controller is configured to assign an additional task to the first computing system node through allocating one or multiple additional containers to the first computing system node.
18. The supervisor controller of claim 16 , wherein reducing the use of the electronic random access memory of the second computing system node by a task assigned to the second computing system node comprises killing the container associated with that task assigned to the second computing system node.
19. The supervisor controller of claim 16 , wherein the master node further comprises a job tracker configured to assign tasks to the plurality of computing system nodes based on the number of containers available at a particular computing system node.
20. The supervisor controller of claim 16 , wherein the supervisor controller assigns the additional task to the first computing system node further after having determined that no containers are available at the first computing system node.
21. A supervisor controller configured to dynamically manage assignment of tasks in a hadoop computer cluster, the supervisor controller comprising:
a management controller interface configured to communicate with a management controller to access data representing an assignment of a plurality of tasks across a plurality of computer system nodes in the computer cluster;
an agent controller interface configured to communicate with an agent controller operating on a first computing system node to receive information describing utilization of system resources on the first computing system node, wherein the agent controller interface is further configured to communicate with an agent controller operating on a second computing system node to receive information describing utilization of system resources on the second computing system node; and
a system resource allocation engine configured to assign an additional task to the first computing system node based on determining that the utilization of a CPU of the first computing system node is below a first threshold level, the determination based at least in part on the information describing utilization of system resources on the first computing system node,
wherein one or more computer processors and one or more electronic storage medium are configured to operate the supervisor controller, and
wherein the system resource allocation engine is further configured to reduce the use of a CPU of the second computing system node by a task assigned to the second computing system node based on determining that the utilization of the CPU of the second computing system node is above a second threshold level, the determination based at least in part on the information describing utilization of system resources on the second computing system node.
22. The supervisor controller of claim 21 , wherein the transmission of information from an agent controller to the supervisor controller describing the monitored system resource utilization is periodic.
23. The supervisor controller of claim 22 , wherein the periodic transmission of information from an agent controller to the supervisor controller describing the monitored system resource utilization comprises transmitting in substantially real-time.
24. The supervisor controller of claim 22 , wherein each agent controller periodically transmits the monitored system resources utilization of the respective computing system node to the supervisor controller once every 1 to 5 seconds.
25. The supervisor controller of claim 21 , wherein reducing the use of the CPU of the second computing system node by a task assigned to the second computing system node comprises killing that task assigned to the second computing system node.
26. The supervisor controller of claim 21 , wherein each computing system node is configured to perform computing processes on a received task corresponding to a container associated with the respective computing system node, wherein each container corresponds to set of resources on the respective computing system node for processing one or more tasks.
27. The supervisor controller of claim 26 , wherein the supervisor controller is configured to assign an additional task to the first computing system node through allocating one or multiple additional containers to the first computing system node.
28. The supervisor controller of claim 26 , wherein reducing the use of the CPU of the second computing system node by a task assigned to the second computing system node comprises killing the container associated with that task assigned to the second computing system node.
29. The supervisor controller of claim 26 , wherein the master node further comprises a job tracker configured to assign tasks to the plurality of computing system nodes based on the number of containers available at a particular computing system node.
30. The supervisor controller of claim 26 , wherein the supervisor controller assigns the additional task to the first computing system node further after having determined that no containers are available at the first computing system node.