Framework for scheduling accelerator-enabled workloads

1 . A method comprising:

receiving, by a computer system acting as a control plane of a workload orchestration platform, one or more accelerator requirements for a workload to be scheduled on an accelerator in a cluster, the cluster comprising a plurality of accelerators, the one or more accelerator requirements including:

an accelerator type, architecture, or model on which the workload should be scheduled;

an amount of accelerator compute or memory resources that should be allocated to the workload; and/or

an accelerator service level agreement (SLA) that should be met for the workload;

creating, by the computer system, an accelerator snapshot identifying each accelerator in the cluster and including static and dynamic metadata pertaining to said each accelerator;

determining, by the computer system based on the accelerator snapshot and the one or more accelerator requirements, a target accelerator for the workload; and

scheduling, by the computer system, the workload on the target accelerator;

wherein:

the one or more accelerator requirements further include a prioritized accelerator list; and

determining the target accelerator comprises:

identifying a first subset of accelerators in the cluster that satisfy all static requirements in the one or more accelerator requirements;

providing the accelerator snapshot, the first subset of accelerators, and all accelerator SLAs in the one or more accelerator requirements to an expert system, which outputs a second subset of the first subset that the expert system has determined can meet said all accelerator SLAs; and

identifying an accelerator in the second subset with a highest user-defined priority as defined in the prioritized accelerator list.

2 . The method of claim 1 further comprising:

determining that the workload has failed due to a functional failure;

identifying an alternative accelerator in the cluster based on a current accelerator snapshot and the one or more accelerator requirements; and

re-scheduling the workload on the alternative accelerator.

3 . The method of claim 1 further comprising:

determining that the workload has failed due to a breach of the SLA;

determining one or more constraints at the target accelerator that caused the breach to occur;

identifying an alternative accelerator in the cluster that conforms to the one or more accelerator requirements but is not subject to the one or more constraints; and

re-scheduling the workload on the alternative accelerator.

4 . The method of claim 1 further comprising, prior to determining the target accelerator:

for each accelerator in the cluster:

initializing a sandbox testing instance on the accelerator;

deploying the workload in the sandbox testing instance;

generating synthetic requests against the deployed workload; and

collecting performance data pertaining to the SLA while the synthetic requests are processed; and

modifying the amount of accelerator compute or memory resources that should be allocated to the workload based on the performance data.

5 . The method of claim 1 further comprising:

receiving runtime performance data for the workload;

generating adjusted versions of the one or more accelerator requirements based on the runtime performance data; and

saving the adjusted versions for use in re-scheduling the workload at a future time.

6 . A non-transitory computer readable storage medium comprising program code executable by a computer system acting as a control plane of a workload orchestration platform, the program code causing the computer system to execute a method comprising:

receiving one or more accelerator requirements for a workload to be scheduled on an accelerator in a cluster, the cluster comprising a plurality of accelerators, the one or more accelerator requirements including:

an accelerator type, architecture, or model on which the workload should be scheduled;

an amount of accelerator compute or memory resources that should be allocated to the workload; and/or

an accelerator service level agreement (SLA) that should be met for the workload;

creating an accelerator snapshot identifying each accelerator in the cluster and including static and dynamic metadata pertaining to said each accelerator;

determining, based on the accelerator snapshot and the one or more accelerator requirements, a target accelerator for the workload; and

scheduling the workload on the target accelerator;

wherein:

the one or more accelerator requirements further include a prioritized accelerator list; and

determining the target accelerator comprises:

identifying a first subset of accelerators in the cluster that satisfy all static requirements in the one or more accelerator requirements;

providing the accelerator snapshot, the first subset of accelerators, and all accelerator SLAs in the one or more accelerator requirements to an expert system, which outputs a second subset of the first subset that the expert system has determined can meet said all accelerator SLAs; and

identifying an accelerator in the second subset with a highest user-defined priority as defined in the prioritized accelerator list.

7 . The non-transitory computer readable storage medium of claim 6 wherein the method further comprises:

determining that the workload has failed due to a functional failure;

identifying an alternative accelerator in the cluster based on a current accelerator snapshot and the one or more accelerator requirements; and

re-scheduling the workload on the alternative accelerator.

8 . The non-transitory computer readable storage medium of claim 6 wherein the method further comprises:

determining that the workload has failed due to a breach of the SLA;

determining one or more constraints at the target accelerator that caused the breach to occur;

identifying an alternative accelerator in the cluster that conforms to the one or more accelerator requirements but is not subject to the one or more constraints; and

re-scheduling the workload on the alternative accelerator.

9 . The non-transitory computer readable storage medium of claim 6 wherein the method further comprises, prior to determining the target accelerator:

for each accelerator in the cluster:

initializing a sandbox testing instance on the accelerator;

deploying the workload in the sandbox testing instance;

generating synthetic requests against the deployed workload; and

collecting performance data pertaining to the SLA while the synthetic requests are processed; and

modifying the amount of accelerator compute or memory resources that should be allocated to the workload based on the performance data.

10 . The non-transitory computer readable storage medium of claim 6 wherein the method further comprises:

receiving runtime performance data for the workload;

generating adjusted versions of the one or more accelerator requirements based on the runtime performance data; and

saving the adjusted versions for use in re-scheduling the workload at a future time.

11 . A computer system acting as a control plane of a workload orchestration platform, the computer system comprising:

a processor; and

a non-transitory computer readable medium comprising program code that, when executed, causes the processor to:

receive one or more accelerator requirements for a workload to be scheduled on an accelerator in a cluster, the cluster comprising a plurality of accelerators, the one or more accelerator requirements including:

an accelerator type, architecture, or model on which the workload should be scheduled;

an amount of accelerator compute or memory resources that should be allocated to the workload; and/or

an accelerator service level agreement (SLA) that should be met for the workload;

create an accelerator snapshot identifying each accelerator in the cluster and including static and dynamic metadata pertaining to said each accelerator;

determine, based on the accelerator snapshot and the one or more accelerator requirements, a target accelerator for the workload; and

schedule the workload on the target accelerator;

wherein:

the one or more accelerator requirements further include a prioritized accelerator list; and

determining the target accelerator comprises:

identifying a first subset of accelerators in the cluster that satisfy all static requirements in the one or more accelerator requirements;

providing the accelerator snapshot, the first subset of accelerators, and all accelerator SLAs in the one or more accelerator requirements to an expert system, which outputs a second subset of the first subset that the expert system has determined can meet said all accelerator SLAs; and

identifying an accelerator in the second subset with a highest user-defined priority as defined in the prioritized accelerator list.

12 . The computer system of claim 11 wherein the program code further causes the processor to:

determine that the workload has failed due to a functional failure;

identify an alternative accelerator in the cluster based on a current accelerator snapshot and the one or more accelerator requirements; and

re-schedule the workload on the alternative accelerator.

13 . The computer system of claim 11 wherein the program code further causes the processor to:

determine that the workload has failed due to a breach of the SLA;

determine one or more constraints at the target accelerator that caused the breach to occur;

identify an alternative accelerator in the cluster that conforms to the one or more accelerator requirements but is not subject to the one or more constraints; and

re-schedule the workload on the alternative accelerator.

14 . The computer system of claim 11 wherein the program code further causes the processor to, prior to determining the target accelerator:

for each accelerator in the cluster:

initialize a sandbox testing instance on the accelerator;

deploy the workload in the sandbox testing instance;

generate synthetic requests against the deployed workload; and

collect performance data pertaining to the SLA while the synthetic requests are processed; and

modify the amount of accelerator compute or memory resources that should be allocated to the workload based on the performance data.

15 . The computer system of claim 11 wherein the program code further causes the processor to:

receive runtime performance data for the workload;

generate adjusted versions of the one or more accelerator requirements based on the runtime performance data; and

save the adjusted versions for use in re-scheduling the workload at a future time.