Cloud bursting technologies
Technologies are provided for a multi-cloud bursting service. An example method can include receiving, via a cloud bursting service associated with different clouds, a cloud bursting configuration enabling the cloud bursting service for a local compute environment; based on the cloud bursting configuration, determining a number of jobs in a jobs queue associated with one or more cloud environments from the different clouds; determining a number of nodes available to process the number of jobs in the jobs queue; based on the number of jobs in the jobs queue and number of nodes available, determining whether to spin up a new node, take offline an existing node, or shutdown the existing node to yield a determination; and based on the determination and cloud bursting configuration, performing a cloud bursting action including spinning up the new node, taking offline the existing node, or shutting down the existing node.
1. A method comprising:
evaluating, via a bursting service, a plurality of jobs received from a plurality of users, the plurality of jobs being in a job queue to yield an evaluation, wherein the evaluation determines whether there are enough active nodes in a combination of nodes in the current cluster and first nodes in an on-demand cluster to process the plurality of jobs in the job queue;
determining, via the bursting service and based on the evaluation, a bursting mode from a plurality of bursting modes;
bursting from the current cluster into the on-demand cluster based on the bursting mode determined via the bursting service to yield a new cluster comprising second nodes from the on-demand cluster in addition to the nodes in the current cluster and first nodes of the on-demand cluster; and
processing the plurality of jobs via at least one of the current cluster and the new cluster.
2. The method of claim 1 , wherein the evaluation covers all jobs within the job queue that have requirements to finish by a certain time in the job queue.
3. The method of claim 1 , wherein the plurality of bursting modes comprises at least one of a minimum burst mode, a maximum burst mode and a burst all mode.
4. The method of claim 3 , wherein in the minimum burst mode, the method comprises spinning up a minimum number of nodes required to complete all the plurality of jobs.
5. The method of claim 3 , wherein in the maximum burst mode, the method comprises spinning up enough nodes required to complete all the plurality of jobs immediately.
6. The method of claim 3 , wherein in the burst all mode, the method comprises spinning up all available nodes at one time.
7. The method of claim 6 , further comprising, for the burst all mode, shutting down all nodes upon completing all jobs.
8. The method of claim 1 , wherein bursting from the current cluster into the on-demand cluster further comprises:
in a first mode, maintaining a head node as active and destroying compute nodes after a job of the plurality of jobs requiring the compute nodes is complete;
in a second mode, maintaining the head node as active and setting off-line the compute nodes after the job requiring the compute nodes is complete; and
in a third mode, upon completion of the job, destroying the new cluster including the head node.
9. The method of claim 1 , wherein the bursting service can be set to on or off.
10. The method of claim 9 , wherein when the bursting service is set to be off, the method includes spinning up all or a portion of licensed instances in a cluster that remain persistent.
11. A system comprising:
one or more processors; and
at least one non-transitory computer-readable storage medium storing instructions which, when executed by the one or more processors, cause the one or more processors to perform operations comprising:
evaluating a plurality of jobs received from a plurality of users, the plurality of jobs being in a job queue and to be processed in a current cluster to yield an evaluation, wherein the evaluation determines whether there are enough active nodes in a combination of nodes in the current cluster and first nodes in an on-demand cluster to process the plurality of jobs in the job queue;
determining, based on the evaluation, a bursting mode from a plurality of bursting modes;
bursting from the current cluster into the on-demand cluster based on the bursting mode determined to yield a new cluster comprising second nodes from the on-demand cluster in addition to the nodes in the current cluster and first nodes of the on-demand cluster; and
processing the plurality of jobs via at least one of the current cluster and the new cluster.
12. The system of claim 11 , wherein the evaluation covers all jobs within the job queue that have requirements to finish by a certain time in the job queue.
13. The system of claim 11 , wherein the plurality of bursting modes comprises at least one of a minimum burst mode, a maximum burst mode and a burst all mode.
14. The system of claim 13 , wherein in the minimum burst mode, the operations further comprise spinning up a minimum number of nodes required to complete all the plurality of jobs.
15. The system of claim 13 , wherein in the maximum burst mode, the operations further comprise spinning up enough nodes required to complete all the plurality of jobs immediately.
16. The system of claim 13 , wherein in the burst all mode, the operations further comprise spinning up all available nodes at one time.
17. The system of claim 16 , further comprising, for the burst all mode, shutting down all nodes upon completing all jobs.
18. The system of claim 11 , wherein bursting from the current cluster into the on-demand cluster further comprises:
in a first mode, maintaining a head node as active and destroying compute nodes after a job of the plurality of jobs requiring the compute nodes is complete;
in a second mode, maintaining the head node as active and setting off-line the compute nodes after the job requiring the compute nodes is complete; and
in a third mode, upon completion of the job, destroying the new cluster including the head node.
19. The system of claim 11 , wherein the operations comprise a bursting service that can be set to on or off.
20. The system of claim 19 , wherein when the bursting service is set to be off, the operations further comprise spinning up all or a portion of licensed instances in a cluster that remain persistent.