Dynamically allocating resources for interdependent portions of distributed data processing programs
Portions of distributed data processing programs may be dynamically allocated computing resources. When execution of a distributed data processing program is requested, interdependent portions of the resource may be dynamically computing resources according to requirements determined for the portions. A resource manager may, in some embodiments, respond to individual allocation requests for the different portions received from a program manager for the distributed data processing program. The allocated resources may be obtained from pools of computing resources that can be utilized by different distributed data processing programs.
1. A system, comprising:
a memory to store program instructions which, if performed by at least one processor, cause the at least one processor to perform a method to at least:
receive a request to cause performance of a distributed data processing program;
in response to the receipt of the request, launch a program manager of the distributed data processing program at a host;
during the performance of the distributed data processing program:
determine two or more interdependent portions of the distributed data processing program to be performed;
receive a first request to allocate a first computing resource to perform a first of the two or more interdependent portions at a resource manager;
allocate, by the resource manager, the first computing resource from one or more pools of computing resources according one or more requirements of the first of the two or more interdependent portions responsive to the first request;
subsequent to initiation of performance of the first of the two or more independent portions, receive a second request to allocate a second computing resource to perform a second of the two or more independent portions of the distributed data processing program at the resource manager; and
allocate, by the resource manager, the second computing resource from the one or more pools of computing resources according one or more requirements of the second of the two or more interdependent portions of the distributed data processing program responsive to the second request; and
cause a result of the request to cause the performance of the distributed data processing program to be provided to a user.
2. The system of claim 1 , wherein the requests to allocate respective resources indicate the one or more requirements for the two or more interdependent portions.
3. The system of claim 1 , wherein the program instructions cause the at least one processor to perform the method to further:
direct, by the resource manager, one or more cleanup operations at the allocated resources for one of the interdependent portions after performance the one interdependent portion at the allocated computing resources; and
return, by the resource manager, the allocated resources of the one interdependent portion to the one or more pools.
4. The system of claim 1 , wherein the at least one processor and the memory are implemented as part of a program execution service of a provider network and wherein the request to cause the performance of the distributed data processing program is received from another service of the provider network.
5. A method, comprising:
receiving a request to cause performance of a distributed data processing program;
during the performance of the distributed data processing program:
determine two or more interdependent portions of the distributed data processing program to be performed; and
dynamically allocating a first computing resource to perform a first of the two or more interdependent portions of the distributed data processing program according to one or more requirements of the first of the two or more interdependent portions, the one or more requirements the first of the two or more interdependent portions indicated in a first allocation request for the first of the two or more interdependent portions;
subsequent to initiation of performance of the first of the two or more independent portions, allocating a second computing resource to perform a second of the two or more interdependent portions of the distributed data processing program according to one or more requirements of the second of the two or more interdependent portions, the one or more requirements the second of the two or more interdependent portions indicated in a second allocation request for the second of the two or more interdependent portions; and
causing a result of the request to be provided to a user.
6. The method of claim 5 , further comprising launching a program manager at a host to direct performance of the interdependent portions of the distributed data processing program at the allocated computing resources.
7. The method of claim 6 , wherein dynamically allocating the computing resources to perform the two or more interdependent portions of the distributed data processing program according to the one or more requirements of each of the two or more interdependent portions comprises:
receiving the respective allocation requests for the two or more interdependent portions at a resource manager from the program manager for the distributed data processing program; and
evaluating, by the resource manager the respective allocation requests according to the indicated requirements to select the resources to allocate to the two or more interdependent portions of the distributed data processing program.
8. The method of claim 7 , wherein dynamically allocating the computing resources to perform the two or more interdependent portions of the distributed data processing program according to the one or more requirements of each of the two or more interdependent portions comprises causing the allocated resources for the two or more interdependent portions to register the with the program manager to perform the two or more interdependent portions.
9. The method of claim 6 , wherein the computing resources are allocated from one or more pools of preconfigured computing resources and wherein the method further comprises returning, by the resource manager the allocated resources of one of the interdependent portions to the one or more pools for allocation to another distributed data processing program after the cleanup operations after performance of the one interdependent portion at the allocated computing resources.
10. The method of claim 6 , wherein dynamically allocating the computing resources to perform the two or more interdependent portions of the distributed data processing program is performed by a resource manager and wherein the method further comprises determining, by the resource manager the one or more requirements for performing each of the two or more interdependent portions of the distributed data processing program.
11. The method of claim 5 , wherein at least one of the computing resources allocated to a first one of the two or more interdependent portions of the distributed data processing program is different than the computing resources allocated to a second one of the two or more interdependent portions of the distributed data processing program.
12. The method of claim 5 , wherein the distributed data processing program is a query processing program form performing a query to a distributed data set, wherein the request to cause the performance of the distributed data processing program is received from a managed query service at a program execution service, wherein the managed query service and the program execution service are implemented as part of a provider network.
13. The method of claim 5 , wherein the computing resources are allocated from one or more pools of preconfigured computing resources hosted in one or more network-based service of a provider network, wherein the receiving, the dynamically allocating, and the causing are performed by a program execution service implemented as part of the provider network.
14. A non-transitory, computer-readable storage medium, storing program instructions that when executed by one or more computing devices cause the one or more computing devices to implement:
receiving a request to cause performance of a distributed data processing program;
during the performance of the distributed data processing program:
determine two or more interdependent portions of the distributed data processing program to be performed;
determining one or more requirements for performing a first of the two or more interdependent portions of the distributed data processing program, the one or more requirements indicated in a first allocation request for the first of the two or more interdependent portions;
dynamically allocating a first computing resource to perform the first of the two or more interdependent portions of the distributed data processing program according to the determined one or more requirements of the first of the two or more interdependent portions;
subsequent to initiation of performance of the first of the two or more independent portions, determining one or more requirements for performing a second of the two or more interdependent portions of the distributed data processing program, the one or more requirements indicated in a second allocation request for the second of the two or more interdependent portions; and
dynamically allocating a second computing resource to perform the second of the two or more interdependent portions of the distributed data processing program according to the determined one or more requirements of the second of the two or more interdependent portions; and
causing a result of the request to be provided to a user.
15. The non-transitory, computer-readable storage medium of claim 14 , wherein the program instructions cause the one or more computing devices to further implement launching a program manager at a host to direct performance of the interdependent portions of the distributed data processing program at the allocated computing resources.
16. The non-transitory, computer-readable storage medium of claim 15 ,
wherein the program instructions cause the one or more computing devices to further implement receiving the respective allocation requests for the two or more interdependent portions at a resource manager from the program manager for the distributed data processing program;
wherein, in determining the one or more requirements for performing each of two or more interdependent portions of the distributed data processing program, the program instructions cause the one or more computing devices to implement identifying the requirements for the interdependent portions of the distributed data processing program indicated in the requests; and
wherein, in dynamically allocating computing resources to perform the two or more interdependent portions of the distributed data processing program according to the determined requirements of each of the two or more interdependent portions, the program instructions cause the one or more computing devices to implement evaluating, by the resource manager the respective allocation requests according to the indicated requirements to select the resources to allocate to the two or more interdependent portions of the distributed data processing program.
17. The non-transitory, computer-readable storage medium of claim 15 , wherein the program instructions further cause the one or more computing devices to implement directing, by the resource manager, one or more cleanup operations at the allocated resources for one of the interdependent portions after performance the one interdependent portion at the allocated computing resources.
18. The non-transitory, computer-readable storage medium of claim 17 , wherein the computing resources are allocated from one or more pools of preconfigured computing resources and wherein the program instructions further cause the one or more computing devices to implement returning, by the resource manager the allocated resources of the one interdependent portion to the one or more pools for allocation to another distributed data processing program after the cleanup operations.
19. The non-transitory, computer-readable storage medium of claim 14 , wherein at least one of the computing resources allocated to a first one of the two or more interdependent portions of the distributed data processing program is also allocated to a second one of the two or more interdependent portions of the distributed data processing program.
20. The non-transitory, computer-readable storage medium of claim 14 , wherein the distributed data processing program is an Extract Transform Load (ETL) job, wherein the request to cause the performance of the distributed data processing program is received from an ETL service at a program execution service, wherein the ETL service and the program execution service are implemented as part of a provider network.