SCHEDULING SYSTEM FOR COMPUTATIONAL WORK ON HETEROGENEOUS HARDWARE

1 . A computer-implemented method, comprising:

under the control of one or more computer systems configured with executable instructions,

managing an amount of Graphics Processing Unit (GPU) memory used by one or more processes, wherein the one or more processes directly or indirectly impact one or more other processes running on the GPU;

organizing a host machine with access to the GPU according to one or more request slots configured to specify a number of processes that are available to be processed by the GPU; and

scheduling the one or more processes based at least in part on the GPU memory.

2 . The computer-implemented method of claim 1 , wherein the computer-implemented method further includes Application Programming Interface (API) requests, complex-interaction calculations, neural networks, artificial intelligence, or other computation-intensive application.

3 . The computer-implemented method of claim 1 , wherein the computer-implemented method further includes:

optimizing a queue, in response to receiving two or more requests at or around a same time;

determining an amount of time to process a first request of the two or more requests;

determining an amount of time to process a second request of the two or more requests; and

ordering the first request and the second request in a queue, wherein the queue is configured to store the two or more requests.

4 . The computer-implemented method of claim 1 , wherein the computer-implemented method further includes:

receiving, from a client device, a request to add a persistent slot;

scheduling one process of the one or more processes in the persistent slot; and

determining if the one process executes one or more child processes.

5 . A system, comprising:

at least one computing device configured to implement one or more services, wherein the one or more services are configured to:

manage an amount of Graphics Processing Unit (GPU) memory used by one or more processes, wherein the one or more processes directly or indirectly impact one or more other processes running on the GPU;

organize a host machine with access to the GPU according to one or more request slots configured to specify a number of processes that are available to be processed by the GPU; and

to schedule the one or more processes based at least in part on the GPU memory.

6 . The system of claim 5 , wherein the one or more processes are received from one or more client devices.

7 . The system of claim 5 , wherein the at least one computing device is further configured to:

receive, from a client device, a request to add a persistent slot;

schedule one process of the one or more processes in the persistent slot; and

determine if the one process executes one or more child processes.

8 . The system of claim 5 , wherein the at least one computing device is further configured to:

optimize a queue, in response to receiving two or more requests at or around a same time;

determine an amount of time to process a first request of the two or more requests;

determine an amount of time to process a second request of the two or more requests; and

order the first request and the second request in a queue, wherein the queue is configured to store the two or more requests.

9 . The system of claim 8 , wherein the at least one computing device is further configured to:

receive at least one of information, input, and data associated with the one or more processes; and

store the at least one of information, input, and data in a database operably connected to the queue.

10 . The system of claim 5 , wherein the host machine is a virtual machine operably connected to a GPU.

11 . The system of claim 5 , wherein one or more processes include Application Programming Interface (API) requests, complex-interaction calculations, neural networks, artificial intelligence, or other computation-intensive applications.

12 . The system of claim 5 , wherein the at least one computing device is further configured to:

launch a new host machine;

create a new slot on an existing host machine; and

designate additional resources to the host machine from a pool of host machines operably connected to the GPU.

13 . A non-transitory computer-readable storage medium having stored thereon executable instructions that, when executed by one or more processors of a computer system, cause the computer system to at least:

receive, from a client device, a request to process data associated with the request;

schedule the request to one or more resources of a Graphics Processing Unit (GPU);

identify an amount of GPU resources being available to process the request using at least the data associated with the request; and

assign the request to the GPU.

14 . The non-transitory computer-readable storage medium of claim 13 , wherein the instructions further comprise instructions that, when executed by the one or more processors, cause the computer system to provide, to the client device, the processed request.

15 . The non-transitory computer-readable storage medium of claim 14 , wherein the instructions that cause the computer system to provide the processed request further include instructions that cause the computer system to maintain the request assigned to the GPU after the processed request is provided to the client device.

16 . The non-transitory computer-readable storage medium of claim 13 , wherein the instructions further comprise instructions that, when executed by the one or more processors, cause the computer system to maintain at least one of the request, the data associated with the request, and the processed request.

17 . The non-transitory computer-readable storage medium of claim 16 , wherein the instructions further comprise instructions that, when executed by the one or more processors, cause the computer system to:

receive one or more additional requests, the one or more additional requests including new data associated with the request;

retrieve the request maintained by the computer system; and

process the one or more additional requests based on the request.

18 . The non-transitory computer-readable storage medium of claim 13 , wherein the instructions that cause the computer system to schedule the request further include instructions that, when executed by the one or more processors, cause the computer system to schedule the request in a slot of the GPU, wherein the slot of the GPU is initialized based on the amount of GPU resources being available.

19 . The non-transitory computer-readable storage medium of claim 13 , wherein the instructions that cause the computer system to identify the amount of GPU resources being available to process the request further include instructions that, when executed by the one or more processors, cause the computer system to determine a status of one or more slots of the GPU, wherein the one or more slots of the GPU are configured to receive at least the request and the data associated with the request in order to process the request.

20 . The non-transitory computer-readable storage medium of claim 13 , wherein the instructions further comprise instructions that, when executed by the one or more processors, cause the computer system to:

determine if one or more other requests are assigned to the GPU;

identify the one or more other requests assigned to the GPU to be evicted from the GPU; and

evict the one or more other requests from the GPU.