Cloud bursting technologies
In some examples, a multi-cloud bursting service generates cloud agnostic burst templates for bursting workload environments on different clouds, each of the cloud agnostic burst templates defining a stack for a workload environment and tasks for provisioning cloud resources and deploying, on the cloud resources, the workload environment associated with the stack, the stack including applications, libraries, services, data, and/or an operating system. The multi-cloud bursting service can receive, from a local compute environment, a request to burst the workload environment onto a selected cloud and, based on a cloud agnostic burst template identified for the request, provision the cloud resources from the selected cloud and deploy the workload environment on the cloud resources from the selected cloud.
1. A method comprising:
generating one or more cloud agnostic burst templates for bursting one or more workload environments on different cloud environments, each of the one or more cloud agnostic burst templates defining:
a respective stack for a respective workload environment, the respective stack comprising at least one of one or more applications, one or more libraries, one or more services, and an operating system, the respective stack further comprising data to be processed by the one or more applications; and
one or more tasks for provisioning one or more cloud resources and deploying, on the one or more cloud resources, the respective workload environment associated with the respective stack;
receiving, at a multi-cloud bursting service, from a local compute environment, a request to burst the respective workload environment onto a particular cloud environment from the different cloud environments; and
based on the one or more cloud agnostic burst templates:
provisioning, via the multi-cloud bursting service, the one or more cloud resources from the particular cloud environment; and
deploying, via the multi-cloud bursting service, the respective workload environment on the one or more cloud resources from the particular cloud environment.
2. The method of claim 1 , further comprising adding the one or more cloud resources to a cluster of resources on the local compute environment, the cluster of resources comprising the one or more cloud resources from the particular cloud environment and one or more resources from the local compute environment.
3. The method of claim 2 , wherein the one or more cloud resources from the particular cloud environment and one or more resources from the local compute environment comprise a set of nodes from the particular cloud environment and the local compute environment, the set of nodes comprising at least one of a storage node, a compute node, and a network node.
4. The method of claim 3 , further comprising generating a graphical user interface identifying the set of nodes from the particular cloud environment and the local compute environment, jobs assigned to the set of nodes, and a respective status of each job.
5. The method of claim 4 , wherein the graphical user interface identifies a respective queue status of each of a plurality of jobs on the local compute environment and a respective job identifier associated with each of the plurality of jobs, the plurality of jobs comprising at least a portion of the jobs assigned to the set of nodes.
6. The method of claim 1 , wherein provisioning the one or more cloud resources and deploying the respective workload environment on the one or more cloud resources are based on the respective stack and the one or more tasks, the method further comprising:
launching, on the one or more cloud resources, at least one instance of the respective workload environment; and
assigning one or more jobs to the at least one instance of the respective workload environment on the one or more cloud resources.
7. The method of claim 1 , wherein provisioning the one or more cloud resources and deploying the respective workload environment on the one or more cloud resources comprise translating the one or more tasks into one or more commands defined for the particular cloud environment based at least partly on at least one of a configuration or a characteristic of the particular cloud environment, the one or more commands being executable for provisioning the one or more cloud resources and deploying the respective workload environment at the particular cloud environment.
8. The method of claim 1 , wherein the one or more tasks comprise executable instructions for at least one of uploading one or more files associated with the respective stack to one or more locations, requesting the one or more cloud resources for provisioning, installing at least one of the one or more applications and libraries, and configuring at least one of the one or more applications, the one or more services, the one or more cloud resources, and the respective workload environment.
9. The method of claim 8 , wherein the one or more cloud agnostic templates are associated with one or more job definitions associated with one or more jobs, the one or more job definitions specifying, for each job, at least one of a respective job identifier, a stack identifier corresponding to a stack for the job, and one or more parameters associated with the job.
10. The method of claim 9 , wherein the one or more parameters specify at least one of a trigger for provisioning one or more cloud nodes for the job, a type of cloud instance for the job, a number of cloud instances for the job, a number of cloud nodes to request for the job, a number of processors for each cloud node, and one or more job policies.
11. The method of claim 10 , wherein the one or more job policies comprise at least one of a quality of service, a node or resource usage limit, a cloud bursting limit, and a purge condition defining at least one of a time-to-live and a node idle purge time, the trigger comprising at least one of a threshold backlog, a threshold resource availability, a policy violation, and a threshold condition.
12. A system comprising:
one or more processors; and
at least one non-transitory computer-readable storage medium storing instructions which, when executed by the one or more processors, cause the system to:
generate one or more cloud agnostic burst templates for bursting one or more workload environments on different cloud environments, each of the one or more cloud agnostic burst templates defining:
a respective stack for a respective workload environment, the respective stack comprising at least one of one or more applications, one or more libraries, one or more services, and an operating system, the respective stack further comprising data to be processed by the one or more applications; and
one or more tasks for provisioning one or more cloud resources and deploying, on the one or more cloud resources, the respective workload environment associated with the respective stack;
receive, from a local compute environment, a request to burst the respective workload environment onto a particular cloud environment from the different cloud environments; and
based on the one or more cloud agnostic burst templates:
provision the one or more cloud resources from the particular cloud environment; and
deploy the respective workload environment on the one or more cloud resources from the particular cloud environment.
13. The system of claim 12 , the non-transitory computer-readable storage medium storing instructions which, when executed by the one or more processors, cause the system to:
add the one or more cloud resources to a cluster of resources on the local compute environment, the cluster of resources comprising the one or more cloud resources from the particular cloud environment and one or more resources from the local compute environment.
14. The system of claim 13 , wherein the one or more cloud resources from the particular cloud environment and one or more resources from the local compute environment comprise a set of nodes from the particular cloud environment and the local compute environment, the set of nodes comprising at least one of a storage node, a compute node, and a network node.
15. The system of claim 14 , the non-transitory computer-readable storage medium storing instructions which, when executed by the one or more processors, cause the system to:
generate a graphical user interface identifying the set of nodes from the particular cloud environment and the local compute environment, jobs assigned to the set of nodes, and a respective status of each job.
16. The system of claim 12 , wherein provisioning the one or more cloud resources and deploying the respective workload environment on the one or more cloud resources are based on the respective stack and the one or more tasks, the non-transitory computer-readable storage medium storing instructions which, when executed by the one or more processors, cause the system to:
launch, on the one or more cloud resources, at least one instance of the respective workload environment; and
assign one or more jobs to the at least one instance of the respective workload environment on the one or more cloud resources.
17. A non-transitory computer-readable storage medium comprising:
instructions stored thereon which, when executed by one or more processors, cause the one or more processors to:
generate one or more cloud agnostic burst templates for bursting one or more workload environments on different cloud environments, each of the one or more cloud agnostic burst templates defining:
a respective stack for a respective workload environment, the respective stack comprising at least one of one or more applications, one or more libraries, one or more services, and an operating system, the respective stack further comprising data to be processed by the one or more applications; and
one or more tasks for provisioning one or more cloud resources and deploying, on the one or more cloud resources, the respective workload environment associated with the respective stack;
receive, from a local compute environment, a request to burst the respective workload environment onto a particular cloud environment from the different cloud environments; and
based on the one or more cloud agnostic burst templates:
provision the one or more cloud resources from the particular cloud environment; and
deploy the respective workload environment on the one or more cloud resources from the particular cloud environment.
18. The non-transitory computer-readable storage medium of claim 17 , wherein provisioning the one or more cloud resources and deploying the respective workload environment on the one or more cloud resources comprise translating the one or more tasks into one or more commands specifically for the particular cloud environment based on at least one of a configuration or a characteristic of the particular cloud environment, the one or more commands being executable for provisioning the one or more cloud resources and deploying the respective workload environment at the particular cloud environment.
19. The non-transitory computer-readable storage medium of claim 18 , wherein the one or more tasks defined in the one or more cloud agnostic templates comprise executable instructions for at least one of uploading one or more files associated with the respective stack to one or more locations, requesting the one or more cloud resources, installing at least one of the one or more applications and libraries, and configuring at least one of the one or more applications, the one or more services, the one or more cloud resources, and the respective workload environment.
20. The non-transitory computer-readable storage medium of claim 19 , wherein the one or more cloud agnostic templates comprises one or more job templates associated with one or more respective jobs, the one or more job templates defining, for each job, at least one of a respective job identifier, a stack identifier associated with a stack for the job, and one or more parameters, wherein the one or more parameters specify at least one of a trigger for provisioning one or more cloud nodes for the job, a type of cloud instance to burst for the job, a number of cloud instances to burst for the job, a number of cloud nodes to request for the job, a number of processors for each cloud node, and one or more job policies comprising at least one of usage policies and purge policies.