Generational management of compute resource pools
Provided is a system for managing a pool of compute resources usable to execute user codes such as containerized software applications. The pool may include a plurality of sub-pools of compute instances that are each suited to handle a specific type of code execution requests and/or a specific set of capacity or resource configurations. For example, when a code execution request is received, the system may determine compute capacity from which sub-pool would be best suited to handle the request, and direct the request to that sub-pool (or execute the user code using compute capacity from that sub-pool). Each sub-pool may be assigned a generational indicator that indicates its level of use (e.g., rarely used, moderately use, frequently used, etc.) and/or its age (e.g., brand new, young, old, etc.). As a sub-pool experiences continued use, the sub-pool may be promoted and its retention/replenishment policy may be updated accordingly.
1. A cloud provider system comprising:
a pool of virtual machines, wherein each virtual machine in the pool is usable to execute a task in response to an incoming request to execute the task, and the pool is divided into a plurality of sub-pools, wherein the plurality of sub-pools include at least a first sub-pool and a second sub-pool; and
a container service comprising computer hardware, wherein the container service is configured to manage and utilize the pool of virtual machines and is further configured to at least:
receive a request to execute a user code on a virtual machine in the pool on behalf of a user, wherein the request includes capacity configuration information indicating a type of computing resource and an amount of computing resource corresponding to the type of computing resource requested for executing the user code;
determine, based at least partly on (i) the amount of computing resource indicated by the capacity configuration information exceeding a first amount or range of amounts of computing resource provided by one or more individual compute instances within the first sub-pool and also exceeding a second amount or range of amounts of computing resource provided by one or more individual compute instances within the second sub-pool and different from the first amount or range of amounts provided by the one or more individual compute instances within the first sub-pool, or (ii) the type of computing resource indicated by the capacity configuration information not being supported by one or more individual compute instances within the first sub-pool and one or more individual compute instances within the second sub-pool, that a condition for adding an additional sub-pool to the pool of compute instances such that a total amount of computing resource in the pool is increased is satisfied;
create an additional sub-pool having a generational indicator that indicates that the additional sub-pool is new, wherein the generational indicator of the additional sub-pool is different from a generational indicator of at least one of the plurality of sub-pools;
update capacity configuration mapping information associated with the pool such that at least one code execution request that would have been directed to one of the plurality of sub-pools prior to the creation of the additional sub-pool would now be directed to the additional sub-pool;
initialize an additional virtual machine having a capacity configuration that is different from at least one of the virtual machines in the pool; and
cause the user code to be executed on the additional virtual machine.
2. The cloud provider system of claim 1 , wherein the container service is further configured to:
determine that a threshold number of code execution requests have been processed using virtual machines from the additional sub-pool; and
update the generational indicator associated with the additional sub-pool to another value that indicates that the additional sub-pool is no longer new.
3. The cloud provider system of claim 2 , wherein the container service is further configured to update a replenishment policy associated with the additional sub-pool to correspond to the updated generational indicator, wherein the updated replenishment policy is different from that associated with at least one other sub-pool in the pool.
4. The cloud provider system of claim 3 , wherein the container service is further configured to control the number of virtual machines in the additional sub-pool according to the updated replenishment policy associated with the additional sub-pool such that an increased number of pre-initialized virtual machines are maintained in the additional sub-pool than prior to the updating of the replenishment policy.
5. A computer-implemented method comprising:
providing a pool of compute instances, wherein the compute instances in the pool are divided into a plurality of sub-pools, wherein the plurality of sub-pools include at least a first sub-pool and a second sub-pool;
receiving a request to execute a user code on behalf of a user, wherein the request includes capacity configuration information indicating a type of computing resource and an amount of computing resource corresponding to the type of computing resource requested for executing the user code;
determining, based at least partly on (i) the amount of computing resource indicated by the capacity configuration information exceeding a first amount or range of amounts of computing resource provided by one or more individual compute instances within the first sub-pool and also exceeding a second amount or range of amounts of computing resource provided by one or more individual compute instances within the second sub-pool and different from the first amount or range of amounts provided by the one r more individual compute instances within the first sub-pool, or (ii) the type of computing resource indicated by the capacity configuration information not being supported by one or more individual compute instances within the first sub-pool and one or more individual compute instances within the second sub-pool, that a condition for adding an additional sub-pool to the pool of compute instances such that a total amount of computing resource in the pool is increased is satisfied;
creating an additional sub-pool having a generational indicator that indicates that the additional sub-pool is in a first generational state, wherein the generational indicator of the additional sub-pool is different from a generational indicator of at least one other sub-pool in the pool of compute instances;
initializing an additional compute instance having a capacity configuration that is different from at least one of the compute instances in the pool; and
executing the user code on the additional compute instance.
6. The computer-implemented method of claim 5 , further comprising:
determining that a condition for updating the generational indicator associated with the additional sub-pool is satisfied; and
updating the generational indicator associated with the additional sub-pool to
another value that indicates that the additional sub-pool is in a second generational state different from the first generational state.
7. The computer-implemented method of claim 5 , wherein the additional sub-pool is associated with a replenishment policy corresponding to the first generational state, wherein the replenishment policy indicates a number of compute instances to be maintained in the additional sub-pool.
8. The computer-implemented method of claim 7 , further comprising:
determining that the replenishment policy associated with the additional sub-pool has been updated; and
causing the number of compute instances maintained in the additional sub-pool according to the updated replenishment policy associated with the additional sub-pool.
9. The computer-implemented method of claim 5 , further comprising updating capacity configuration mapping information associated with the pool of compute instances such that at least one code execution request that would have been directed to one of the plurality of sub-pools prior to the creation of the additional sub-pool would now be directed to the additional sub-pool.
10. The computer-implemented method of claim 9 , further comprising directing an incoming code execution request to the additional sub-pool based on the updated capacity configuration mapping information associated with the pool of compute instances.
11. The computer-implemented method of claim 5 , wherein the additional compute instance is one of a virtual machine instance, a bare-metal instance, a physical machine, a container, a node, an offload card, an IoT device, or on-premises compute capacity.
12. The computer-implemented method of claim 5 , wherein the additional compute instance (i) provides an amount or range of amounts of computing resource greater than or equal to the amount of computing resource indicated by the capacity configuration information or (ii) supports the type of computing resource indicated by the capacity configuration information.
13. A non-transitory computer-readable medium storing instructions that, when executed by a computing system within a cloud provider network, cause the computing system to perform operations comprising:
providing a pool of compute instances, wherein the compute instances in the pool are divided into a plurality of sub-pools, wherein the plurality of sub-pools include at least a first sub-pool and a second sub-pool;
receiving a request to execute a user code on behalf of a user, wherein the request includes capacity configuration information indicating a type of computing resource and an amount of computing resource corresponding to the type of computing resource requested for executing the user code;
determining, based at least partly on (i) the amount of computing resource indicated by the capacity configuration information exceeding a first amount or range of amounts of computing resource provided by one or more individual compute instances within the first sub-pool and a second amount or range of amounts of computing resource provided by one or more individual compute instances within the second sub-pool and different from the first amount or range of amounts provided by the one or more individual compute instances within the first sub-pool, or (ii) the type of computing resource indicated by the capacity configuration information not being supported by one or more individual compute instances within the first sub-pool and one or more individual compute instances within the second sub-pool, that a condition for adding an additional sub-pool to the pool of compute instances such that a total amount of computing resource in the pool is increased is satisfied;
creating an additional sub-pool having a generational indicator that indicates that the additional sub-pool is in a first generational state, wherein the generational indicator of the additional sub-pool is different from a generational indicator of at least one other sub-pool in the pool of compute instances;
initializing an additional compute instance having a capacity configuration that is different from at least one of the compute instances in the pool; and
executing the user code on the additional compute instance.
14. The non-transitory computer-readable medium of claim 13 , storing further instructions that, when executed by the computing system, cause the computing system to perform operations comprising:
determining that a condition for updating the generational indicator associated with the additional sub-pool is satisfied; and
updating the generational indicator associated with the additional sub-pool to
another value that indicates that the additional sub-pool is in a second generational state different from the first generational state.
15. The non-transitory computer-readable medium of claim 13 , wherein the additional sub-pool is associated with a replenishment policy corresponding to the first generational state, wherein the replenishment policy indicates a number of compute instances to be maintained in the additional sub-pool.
16. The non-transitory computer-readable medium of claim 15 , storing further instructions that, when executed by the computing system, cause the computing system to perform operations comprising: determining that the replenishment policy associated with the additional sub-pool has been updated; and
causing the number of compute instances maintained in the additional sub-pool according to the updated replenishment policy associated with the additional sub-pool.
17. The non-transitory computer-readable medium of claim 13 , storing further instructions that, when executed by the computing system, cause the computing system to perform operations comprising updating capacity configuration mapping information associated with the pool of compute instances such that at least one code execution request that would have been directed to one of the plurality of sub-pools prior to the creation of the additional sub-pool would now be directed to the additional sub-pool.
18. The non-transitory computer-readable medium of claim 17 , storing further instructions that, when executed by the computing system, cause the computing system to perform operations comprising directing an incoming code execution request to the additional sub-pool based on the updated capacity configuration mapping information associated with the pool of compute instances.
19. The non-transitory computer-readable medium of claim 13 , wherein the additional compute instance is one of a virtual machine instance, a bare-metal instance, a physical machine, a container, a node, an offload card, an IoT device, or on-premises compute capacity.
20. The non-transitory computer-readable medium of claim 13 , wherein the additional compute instance (i) provides an amount or range of amounts of computing resource greater than or equal to the amount of computing resource indicated by the capacity configuration information or (ii) supports the type of computing resource indicated by the capacity configuration information.