Attached accelerator based inference service

1. A computer-implemented method, comprising:

receiving, in a multi-tenant web services provider, an application instance configuration, an application of the application instance to utilize a portion of an accelerator, having a plurality of accelerator slots, during execution of a machine learning model, the application instance configuration including:

an indication of a central processing unit (CPU) capability to be used,

an arithmetic precision of the machine learning model to be used,

an indication of an accelerator capability to be used,

a storage location of the application, and

an indication of an amount of random access memory to use;

based on the received application instance configuration, provisioning the application instance, wherein provisioning includes:

provisioning a physical compute instance including a configuration of a CPU, memory, storage, and networking capacity to execute the application in a first location;

provisioning an accelerator appliance including a configuration of a CPU, memory, storage, and networking capacity to execute a machine learning model of the application in a second location, wherein the accelerator appliance comprises one or more physical accelerators; and

provisioning the plurality of accelerator slots of the accelerator appliance, wherein the application instance is implemented using the physical compute instance, wherein each accelerator slot is implemented using one of the physical accelerators, and wherein the one or more physical accelerators is accessible to the physical compute instance over a network;

attaching the plurality of accelerator slots to the application instance;

loading the machine learning model onto the attached plurality of accelerator slots; and

performing inference using the loaded machine learning model of the application using the attached plurality of accelerator slots, by:

receiving an inference request by the application instance;

transmitting inference request data to the attached plurality of accelerator slots;

receiving and using in the application an initial response from one of the attached plurality of accelerator slots;

processing subsequent responses that are received by discarding one or more of the subsequent responses; and

tracking timing of the responses to determine if migration in any attached accelerator slot from the attached plurality of accelerator slots should occur, wherein:

if a timing of one or more responses is greater than a threshold, performing a migration to a different accelerator slot from the attached plurality of accelerator slots, wherein the migration includes replacing one or more underperforming accelerator slots with new one or more accelerator slots to assume operation in place of the replaced one or more accelerator slots, and

if a timing of one or more responses is less than or equal to a threshold, not perform a migration.

2. The method of claim 1 , wherein the machine learning model includes a description of a computation graph for inference and weights obtained from training.

3. The method of claim 1 , wherein components of each accelerator slot are isolated in terms of resources including CPU, RAM, GPU compute, GPU memory, disk, and network.

4. The method of claim 1 , wherein the application instance and the one or more physical accelerators are within different virtual networks.

5. The method of claim 1 , wherein the accelerator appliance includes a plurality of accelerators of different capabilities.

6. The method of claim 5 , wherein a central processing unit of the accelerator appliance is shared proportional to capabilities of the plurality of accelerators.

7. The method of claim 1 , further comprising:

detaching a portion of the one or more physical accelerators; and

migrating the machine learning model to a different portion of the one or more physical accelerators.

8. The method of claim 1 , further comprising:

detaching a portion of the one or more physical accelerators; and

migrating the machine learning model to a portion of a different accelerator.

9. The method of claim 6 , further comprising:

prior to performing inference using the loaded machine learning model of the application using the attached plurality of accelerator slots, determining an inference engine to use based on the loaded machine learning model.

10. The method of claim 9 , wherein the inference engine is compatible with a version number of a format of the machine learning model.

11. A system comprising:

storage to store an application, the application including a machine learning model; and

an elastic inference service implemented by one or more electronic devices, the elastic inference service including an application instance and an accelerator appliance, the elastic inference service to:

receive a configuration for the application instance, the application of the application instance to utilize a portion of an accelerator of the accelerator appliance, the accelerator including a plurality of accelerator slots during execution of the machine learning model, the application instance configuration including:

an indication of a central processing unit (CPU) capability to be used,

an arithmetic precision of the machine learning model to be used,

an indication of an accelerator capability to be used,

a storage location of the application, and

an indication of an amount of random access memory to use;

based on the received configuration for the application instance, provision the application instance, wherein provisioning includes:

provisioning a physical compute instance including a configuration of a CPU, memory, storage, and networking capacity to execute the application in a first location;

provisioning an accelerator appliance including a configuration of a CPU, memory, storage, and networking capacity to execute a machine learning model of the application in a second location, wherein the accelerator appliance comprises one or more physical accelerators; and

provisioning the plurality of accelerator slots of the accelerator appliance, wherein the application instance is implemented using the physical compute instance, wherein each accelerator slot is implemented using the physical accelerators, and wherein the one or more physical accelerators is accessible to the physical compute instance over a network;

attach the plurality of accelerator slots to the application instance;

load the machine learning model onto the attached plurality of accelerator slots; and

perform inference using the loaded machine learning model of the application using the attached plurality of accelerator slots, by:

receiving an inference request by the application instance;

transmitting inference request data to the attached plurality of accelerator slots;

receiving and using in the application an initial response from one of the attached plurality of accelerator slots;

processing subsequent responses that are received by discarding one or more of the subsequent responses; and

tracking timing of the responses to determine if migration in any attached accelerator slot from the attached plurality of accelerator slots should occur, wherein:

if a timing of one or more responses is greater than a threshold, performing a migration to a different accelerator slot from the attached plurality of accelerator slots, wherein the migration includes replacing one or more underperforming accelerator slots with new one or more accelerator slots to assume operation in place of the replaced one or more accelerator slots, and

if a timing of one or more responses is less than or equal to a threshold, not perform a migration.

12. The system of claim 11 , wherein the elastic inference service is to detach a portion of the one or more physical accelerators and migrate the machine learning model to a different portion of the one or more physical accelerators.

13. The system of claim 11 , wherein the elastic inference service is to detach a portion of the one or more physical accelerators and migrate the machine learning model to a different accelerator.

14. The system of claim 11 , wherein the machine learning model includes a description of a computation graph for inference and weights obtained from training.

15. The system of claim 11 , wherein components of each accelerator slot are isolated in terms of resources including CPU, RAM, GPU compute, GPU memory, disk, and network.

16. The system of claim 11 , wherein the application instance and the one or more physical accelerators are within different virtual networks.

17. The system of claim 16 , wherein the accelerator appliance includes a plurality of accelerators of different capabilities.

18. The system of claim 17 , wherein a central processing unit of the accelerator appliance is shared proportional to capabilities of the plurality of accelerators.