Job execution with managed compute environments

1 . A system, comprising:

one or more computing devices configured to implement a compute environment management system, wherein the compute environment management system is configured to:

provide an interface to receive compute environment service requests;

receive, from a client via the interface, a compute environment service request indicating a type of compute environment, wherein the compute environment management system supports managed and unmanaged compute environments;

receive, from the client, a specification to allow compute instances of a spot resources type to be included in the managed compute environment during batch processing;

determine whether the type of compute environment indicated in the compute environment service request is for an unmanaged compute environment or a managed compute environment, wherein the managed compute environment automatically adds or removes one or more compute instances within the managed compute environment as workload for the managed compute environment changes during the batch processing;

based on a determination that the type of compute environment is for a managed compute environment, establish the managed compute environment according to specified constraints for compute instances, wherein the specified constraints are received from the client and include: the specification to allow compute instances of a spot resources type to be included in the managed compute environment and indicate a desired total number of compute instances to be allocated in the managed compute environment, a specified computational capacity of the one or more compute instances, and a specified software stack for the one or more compute instances;

subsequently, receive a batch of jobs to be executed in the managed compute environment, wherein a given job of the batch of jobs comprises an indication that:

the given job is allowed to be executed on a compute instance of a spot resource type, and

the given job is to be terminated after a maximum time has elapsed;

select, based on the indication that the given job is allowed to be executed on the compute instance of the spot resource type and is to be terminated according to the maximum time, one or more compute instances for the managed compute environment, including a given compute instance of the spot resource type, to be used to execute the batch of jobs, wherein the selection is according to requirements for the given job and the specified constraints;

provision the selected one or more compute instances in the managed compute environment;

initiate execution of the batch of jobs on or across the one or more compute instances in parallel in the managed compute environment for the batch processing; and

automatically scale, during the execution of the batch of jobs, based on number of pending jobs of the batch of jobs and the maximum time, a number of compute instances allocated to the managed compute environment.

2 . The system of claim 1 , wherein the compute environment management system is further configured to:

determine a resource utilization of the one or more compute instances based on usage metrics for the one or more compute instances.

3 . The system of claim 2 , wherein the compute environment management system is further configured to:

increase a quantity of the one or more compute instances based on the resource utilization indicating that the one or more compute instances are overutilized.

4 . The system of claim 2 , wherein the compute environment management system is further configured to:

decrease a quantity of the one or more compute instances based on the resource utilization indicating that the one or more compute instances are underutilized.

5 . The system of claim 1 , wherein the managed compute environment comprises a job queue for the batch of jobs.

6 . The system of claim 5 , wherein the compute environment management system is further configured to:

monitor execution of the batch of jobs to collect metrics for the batch of jobs.

7 . A computer-implemented method, comprising:

providing an interface to receive compute environment service requests;

receiving, from a client via the interface, a compute environment service request indicating a type of compute environment, wherein the compute environment management system supports managed and unmanaged compute environments;

receiving, from the client, a specification to allow compute instances of a spot resources type to be included in the managed compute environment during batch processing;

determining whether the type of compute environment is for an unmanaged compute environment or a managed compute environment, wherein the managed compute environment automatically adds or removes one or more compute instances within the managed compute environment as workload for the managed compute environment changes during the batch processing;

based on a determination that the type of compute environment is for a managed compute environment, establishing the managed compute environment according to specified constraints for compute instances, wherein the specified constraints are received from the client and include: the specification to allow compute instances of a spot resources type to be included in the managed compute environment and indicate a desired total number of compute instances to be allocated in the managed compute environment, a specified computational capacity of the one or more compute instances, and a specified software stack for the one or more compute instances;

subsequently, receiving a batch of jobs to be executed in the managed compute environment, wherein a given job of the batch of jobs comprises an indication that:

the given job is allowed to be executed on a compute instance of a spot resource type, and

the given job is to be terminated after a maximum time has elapsed;

selecting, based on the indication that the given job is allowed to be executed on the compute instance of the spot resource type, one or more compute instances for the managed compute environment, including a given compute instance of the spot resource type, to be used to execute the batch of jobs, wherein the selection is according to requirements for the given job and the specified constraints;

provisioning the selected one or more compute instances in the managed compute environment;

initiating execution of the batch of jobs on or across the one or more compute instances in parallel in the managed compute environment for the batch processing; and

automatically scaling, during the execution of the batch of jobs, based on number of pending jobs of the batch of jobs and the maximum time, a number of compute instances allocated to the managed compute environment.

8 . The method of claim 7 , further comprising:

determining a resource utilization of the one or more compute instances based on usage metrics for the one or more compute instances.

9 . The method of claim 8 , further comprising:

increasing a quantity of the one or more compute instances based on the resource utilization indicating that the one or more compute instances are overutilized.

10 . The method of claim 8 , further comprising:

decreasing a quantity of the one or more compute instances based on the resource utilization indicating that the one or more compute instances are underutilized.

11 . The method of claim 7 , further comprising:

monitoring execution of the batch of jobs to collect metrics for the batch of jobs.

12 . One or more non-transitory computer-readable storage media storing instructions that, when executed on or across one or more processors, cause the one or more processors to:

provide an interface to receive compute environment service requests;

receive, from a client via the interface, a compute environment service request indicating a type of compute environment, wherein the compute environment management system supports managed and unmanaged compute environments;

receive, from the client, a specification to allow compute instances of a spot resources type to be included in the managed compute environment during batch processing;

determine whether the type of compute environment is for an unmanaged compute environment or a managed compute environment, wherein the managed compute environment automatically adds and removes one or more compute instances within the managed compute environment as workload for the managed compute environment changes during the batch processing;

based on a determination that the type of compute environment is for a managed compute environment, establish the managed compute environment according to specified constraints for compute instances, wherein the specified constraints are received from the client and include: the specification to allow compute instances of a spot resources type to be included in the managed compute environment and indicate a desired total number of compute instances to be allocated in the managed compute environment, a specified computational capacity of the one or more compute instances, and a specified software stack for the one or more compute instances;

subsequently, receive a batch of jobs to be executed in the managed compute environment, wherein a given job of the batch of jobs comprises an indication that:

the given job is allowed to be executed on a compute instance of a spot resource type, and

the given job is to be terminated after a maximum time has elapsed;

select, based on the indication that the given job is allowed to be executed on the compute instance of the spot resource type, one or more compute instances for the managed compute environment, including a given compute instance of the spot resource type, to be used to execute the batch of jobs, wherein the selection is according to requirements for the given job and the specified constraints;

provision the selected one or more compute instances in the managed compute environment;

initiate execution of the batch of jobs on or across the one or more compute instances in parallel in the managed compute environment for the batch processing; and

automatically scale, during the execution of the batch of jobs, based on number of pending jobs of the batch of jobs and the maximum time, a number of compute instances allocated to the managed compute environment.

13 . The one or more non-transitory computer-readable storage media of claim 12 , further comprising instructions that, when executed on or across the one or more processors, cause the one or more processors to:

determine a resource utilization of the one or more compute instances based on usage metrics for the one or more compute instances.

14 . The one or more non-transitory computer-readable storage media of claim 13 , further comprising instructions that, when executed on or cross the one or more processors, cause the one or more processors to:

increase a quantity of the one or more compute instances based on the resource utilization indicating that the one or more compute instances are overutilized.

15 . The one or more non-transitory computer-readable storage media of claim 13 , further comprising instructions that, when executed on or across the one or more processors, cause the one or more processors to:

decrease a quantity of the one or more compute instances based on the resource utilization indicating that the one or more compute instances are underutilized.

16 . The one or more non-transitory computer-readable storage media of claim 13 , wherein the managed compute environment comprises a job queue for the batch of jobs.