Distributed computing task management system and method
View Patent ↗A distributed computing task management system includes an application for monitoring a processing load of multiple resources that are used to execute a distributed computing task. The resources include at least one hardware resource and one or more virtual objects executed on the hardware resources in a virtual computing environment. Using the monitored processing load, the application adjusts a performance of the resources to compensate for changes in the processing load incurred by the resources due to execution of the distributed computing task.
1. A distributed computing task management method comprising:
monitoring, using an application stored in a non-transitory medium and executed on at least one processor, a processing load of a plurality of resources executing a map-reduce process comprising a plurality of splitter tasks, a plurality of mapper tasks, and a plurality of reducer tasks, the plurality of resources comprising at least one of a plurality of hardware resources and a plurality of virtual objects executed on the hardware resources in a virtual computing environment, wherein each of the plurality of splitter tasks, the plurality of mapper tasks, and the plurality of reducer tasks are executed on individual ones of the virtual objects in the virtual computing environment;
determining, while the map-reduce process continues to execute on the plurality of resources, performance characteristics of the plurality of resources, the performance characteristics comprising one or more of:
a type of processing load being executed by the plurality of resources; and
a change in the processing load being incurred by the plurality of resources; and
adjusting, using the application executed by the processor and while the map-reduce process continues to execute, a quantity of the plurality of resources used to execute the map-reduce process, the adjusting based on the performance characteristics of the plurality of resources.
2. The method of claim 1 , wherein the type of processing load being executed comprises at least one of a memory usage of the plurality of resources, a computational loading level of the plurality of resources, and an inter-resource communication loading level of the plurality of resources.
3. The method of claim 1 , further comprising monitoring at least one of a job completion time of the map-reduce process executed on each resource, a processing load level of a processor unit used by the plurality of resources, and a memory usage level of a memory unit used by each of the plurality of resources.
4. The method of claim 1 , further comprising adjusting a performance of the plurality of resources by instantiating one or more additional virtual objects when the processing load meets a specified first level or deleting one or more of the plurality of virtual objects when the processing load meets a specified second level, the virtual objects comprising the at least one or more of the plurality of resources.
5. The method of claim 1 , further comprising adjusting the quantity of the plurality of resources by enhancing one or more performance characteristics of at least one of the plurality of resources when the processing load of the at least one resource meets a specified first level or reducing the one or more performance characteristics of the at least one resource when the processing load meets a specified second level.
6. The method of claim 5 , wherein the one or more performance characteristics comprise at least one of a number of processors allocated to at least one of the plurality of resources and an amount of memory allocated to at least one of the plurality of resources.
7. The method of claim 1 , further comprising adjusting the quantity of the plurality of resources by allocating one or more additional hardware resources when the processing load meets a specified first level and de-allocating one or more previously allocated hardware resources when the processing load meets a specified second level, the hardware resources comprising the plurality of resources.
8. The method of claim 1 , wherein the virtual computing environment includes at least one of a converged infrastructure and a cloud computing environment.
9. A distributed computing task management system comprising:
a computing system comprising at least one processor and at least one memory to store an application that is executed by the at least one processor to:
monitor a processing load of a plurality of resources executing a map-reduce process comprising a plurality of splitter tasks, a plurality of mapper tasks, and a plurality of reducer tasks, the resources comprising at least one of a plurality of hardware resources and a plurality of virtual objects executed on the hardware resources in a virtual computing environment, wherein each of the plurality of splitter tasks, the plurality of mapper tasks, and the plurality of reducer tasks are executed on individual ones of the virtual objects in the virtual computing environment;
determine, while the map-reduce process continues to execute on the plurality of resources, performance characteristics of the plurality of resources, the performance characteristics comprising one or more of:
a type of processing load being executed by the plurality of resources; and
a change in the processing load being incurred by the plurality of resources; and
adjust, while the map-reduce process continues to execute, a quantity of the resources used to execute the map-reduce process, the adjusting based on the performance characteristics of the plurality of resources.
10. The distributed computing task management system of claim 9 , wherein the type of processing load being executed comprises at least one of a memory usage of the plurality of resources, a computational loading level of the plurality of resources, and an inter-resource communication loading level of the plurality of resources.
11. The distributed computing task management system of claim 9 , wherein the application is further executed to monitor at least one of a job completion time of the map-reduce process executed on each of the plurality of resources, a processing load level of a processor unit used by the plurality of resources, and a memory usage level of a memory unit used by each of the plurality of resources.
12. The distributed computing task management system of claim 9 , wherein the application is further executed to adjust a performance of the plurality of resources by instantiating one or more additional virtual objects when the processing load meets a specified first level or deleting one or more of the existing virtual objects when the processing load meets a specified second level, the virtual objects comprising the plurality of resources.
13. The distributed computing task management system of claim 9 , wherein the application is further executed to adjust a performance of the plurality of resources by enhancing one or more performance characteristics of at least one of the plurality of resources when the processing load of the at least one resource meets a specified first level or reducing the one or more performance characteristics of the at least one resource when the processing load meets a specified second level.
14. The distributed computing task management system of claim 13 , wherein the one or more performance characteristics comprise at least one of a number of processors allocated to at least one of the plurality of resources and an amount of memory allocated to the at least one resource.
15. The distributed computing task management system of claim 9 , wherein the application is further executed to adjust the quantity of the plurality of resources by allocating one or more additional hardware resources when the processing load meets a specified first level and de-allocating one or more previously allocated hardware resources when the processing load meets a specified second level, the hardware resources comprising the plurality of resources.
16. The distributed computing task management system of claim 9 , wherein the virtual computing environment includes at least one of a converged infrastructure and a cloud computing environment.
17. A non-transitory, computer readable medium including computer executable instructions that when executed by at least one processor are operable to:
monitor a processing load of a plurality of resources executing a map-reduce process comprising a plurality of splitter tasks, a plurality of mapper tasks, and a plurality of reducer tasks, the plurality of resources comprising at least one of a plurality of hardware resources and a plurality of virtual objects executed on the hardware resources in a virtual computing environment, wherein each of the plurality of splitter tasks, the plurality of mapper tasks, and the plurality of reducer tasks are executed on individual ones of the virtual objects in the virtual computing environment;
determine, while the map-reduce process continues to execute on the plurality of resources, performance characteristics of the plurality of resources, the performance characteristics comprising one or more of:
a type of processing load being executed by the plurality of resources; and
a change in the processing load being incurred by the plurality of resources; and
adjust, while the map-reduce process continues to execute, a quantity of the resources used to execute the map-reduce process, the adjusting based on the performance characteristics of the plurality of resources.
18. The non-transitory computer readable medium of claim 17 , the computer executable instructions further operable to perform monitoring at least one of a job completion time of the map-reduce process executed on each resource of the plurality of resources, a processing load level of a processor unit used by the resource, and a memory usage level of a memory unit used by the resource.
19. The non-transitory computer readable medium of claim 17 , the computer executable instructions further operable to perform adjusting a performance of the plurality of resources by instantiating one or more additional virtual objects when the processing load meets a specified first level or deleting one or more of the existing virtual objects when the processing load meets a specified second level, the virtual objects comprising the resources.
20. The non-transitory computer readable medium of claim 17 , the computer executable instructions further operable to perform adjusting a performance of the plurality of resources by enhancing one or more performance characteristics of at least one of the plurality of resources when the processing load of the at least one resource meets a specified first level or reducing the one or more performance characteristics of the at least one resource when the processing load meets a specified second level.