Queue-based GPU virtualization and management system
A method implemented by a server enables sharing of GPU resources by multiple clients. The server receives a request from a first client for GPU services. The request includes a first block of GPU code of an application executing on the first client. A first task corresponding to the first block of GPU code is enqueued in a task queue. The task queue includes a second task that corresponds to a second block of GPU code of an application executing on a second client. The server schedules a time for executing the first task using a GPU device that is assigned to the first client, and dispatches the first task to a GPU worker process to execute the first task at the scheduled time using the GPU device. The GPU device is shared, either temporally or spatially, by the first and second clients for executing the first and second tasks.
1. A method, comprising:
receiving, by a computer server, a first service request from a graphics processing unit (GPU) application programming interface (API) executing on a first client system for GPU processing services provided by the computer server, wherein the first service request comprises a first block of GPU program code of a GPU-accelerated application executing on a processor of the first client system, and wherein the first service request is generated by the GPU API intercepting an API call of the GPU-accelerated application executing on the processor of the first client system and seamlessly connecting with the computer server to redirect execution of the API call by the computer server in a manner transparent to the GPU-accelerated application;
enqueuing, by the computer server, a plurality of pending tasks in a front-end task queue, wherein the plurality of pending tasks correspond to a plurality of GPU service requests received from a plurality of client systems, wherein the plurality of pending tasks comprise a first task that corresponds to the first block of GPU program code received from the first client system, and a second task that corresponds to a second block of GPU program code of a GPU-accelerated application executing on a processor of a second client system;
scheduling, by the computer server, times for executing the pending tasks in the front-end queue using one or more GPU devices;
dispatching, by the computer server, the first task to a server backend GPU worker process to handle execution of the first task at a scheduled time for the first task using an allocated GPU device; and
returning processing results of the execution of the first task to the GPU API executing on the first client system to enable the GPU API to pass the processing results to the GPU-accelerated application executing on the processor of the first client system.
2. The method of claim 1 , wherein the first service request further comprises a block of data which is to be processed using the first block of GPU program code.
3. The method of claim 1 , further comprising:
wherein in response to intercepting the API call of the GPU-accelerated application executing on the processor of the first client system, the GPU API performing a method comprising:
receiving GPU-related code corresponding to the intercepted API call;
determining if the GPU-related code corresponding to the intercepted API call can be processed locally on the first client system;
processing the GPU-related code locally by the GPU API, in response to determining that the GPU-related code corresponding to the intercepted API call can be processed locally on the first client system; and
generating the first service request, in response to determining that the GPU-related code corresponding to the intercepted API call cannot be processed locally on the first client system.
4. The method of claim 1 , wherein the computer server is part of a server cluster that provides GPU-as-a-Service.
5. The method of claim 1 , further comprising dispatching, by the computer server, the second task to a server backend GPU worker process to handle execution of the second task at a scheduled time for the second task using at least the same GPU device allocated for executing the first task.
6. The method of claim 5 , wherein the allocated GPU device is shared temporally, wherein the first and second tasks are executed using the allocated GPU device at different times.
7. The method of claim 5 , wherein the allocated GPU device is shared spatially, wherein both the first and second tasks are concurrently executed using the allocated GPU device.
8. An article of manufacture comprising a processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code is executable by one or more processors to implement a process comprising:
receiving, by a computer server, a first service request from a graphics processing unit (GPU) application programming interface (API) executing on a first client system for GPU processing services provided by the computer server, wherein the first service request comprises a first block of GPU program code of a GPU-accelerated application executing on a processor of the first client system, and wherein the first service request is generated by the GPU API intercepting an API call of the GPU-accelerated application executing on the processor of the first client system and seamlessly connecting with the computer server to redirect execution of the API call by the computer server in a manner transparent to the GPU-accelerated application;
enqueuing, by the computer server, a plurality of pending tasks in a front-end task queue, wherein the plurality of pending tasks correspond to a plurality of GPU service requests received from a plurality of client systems, wherein the plurality of pending tasks comprise a first task that corresponds to the first block of GPU program code received from the first client system, and a second task that corresponds to a second block of GPU program code of a GPU-accelerated application executing on a processor of a second client system;
scheduling, by the computer server, times for executing the pending tasks in the front-end queue using one or more GPU devices;
dispatching, by the computer server, the first task to a server backend GPU worker process to handle execution of the first task at a scheduled time for the first task using an allocated GPU device; and
returning processing results of the execution of the first task to the GPU API executing on the first client system to enable the GPU API to pass the processing results to the GPU-accelerated application executing on the processor of the first client system.
9. The article of manufacture of claim 8 , wherein the first service request further comprises a block of data which is to be processed using the first block of GPU program code.
10. The article of manufacture of claim 8 , wherein the program code is executable by the one or more processors to implement a process comprising dispatching, by the computer server, the second task to a server backend GPU worker process to handle execution of the second task at a scheduled time for the second task using at least the same GPU device allocated for executing the first task.
11. The article of manufacture of claim 10 , wherein the allocated GPU device is shared temporally, wherein the first and second tasks are executed using the allocated GPU device at different times.
12. The article of manufacture of claim 10 , wherein the allocated GPU device is shared spatially, wherein both the first and second tasks are concurrently executed using the allocated GPU device.
13. The article of manufacture of claim 8 , wherein the program code is executable by the one or more processors to implement a process comprising:
wherein in response to intercepting the API call of the GPU-accelerated application executing on the processor of the first client system, the GPU API performing a method comprising:
receiving GPU-related code corresponding to the intercepted API call;
determining if the GPU-related code corresponding to the intercepted API call can be processed locally on the first client system;
processing the GPU-related code locally by the GPU API, in response to determining that the GPU-related code corresponding to the intercepted API call can be processed locally on the first client system; and
generating the first service request, in response to determining that the GPU-related code corresponding to the intercepted API call cannot be processed locally on the first client system.
14. The article of manufacture of claim 8 , wherein the computer server is part of a server cluster that provides GPU-as-a-Service.
15. A system, comprising:
a computer server comprising at least one processor, and a memory to store program code that is executable by the at least one processor to implement a process comprising:
receiving, by the computer server, a first service request from a graphics processing unit (GPU) application programming interface (API) executing on a first client system for GPU processing services provided by the computer server, wherein the first service request comprises a first block of GPU program code of a GPU-accelerated application executing on a processor of the first client system, and wherein the first service request is generated by the GPU API intercepting an API call of the GPU-accelerated application executing on the processor of the first client system and seamlessly connecting with the computer server to redirect execution of the API call by the computer server in a manner transparent to the GPU-accelerated application;
enqueuing, by the computer server, a plurality of pending tasks in a front-end task queue, wherein the plurality of pending tasks correspond to a plurality of GPU service requests received from a plurality of client systems, wherein the plurality of pending tasks comprise a first task that corresponds to the first block of GPU program code received from the first client system, and a second task that corresponds to a second block of GPU program code of a GPU-accelerated application executing on a processor of a second client system;
scheduling, by the computer server, times for executing the pending tasks in the front-end queue using one or more GPU devices;
dispatching, by the computer server, the first task to a server backend GPU worker process to handle execution of the first task at a scheduled time for the first task using an allocated GPU device; and
returning processing results of the execution of the first task to the GPU API executing on the first client system to enable the GPU API to pass the processing results to the GPU-accelerated application executing on the processor of the first client system.
16. The system of claim 15 , wherein the first service request further comprises a block of data which is to be processed using the first block of GPU program code.
17. The system of claim 15 , further comprising dispatching, by the computer server, the second task to a server backend GPU worker process to handle execution of the second task at a scheduled time for the second task using at least the same GPU device allocated for executing the first task.
18. The system of claim 17 , wherein the allocated GPU device is shared temporally, wherein the first and second tasks are executed using the allocated GPU device at different times.
19. The system of claim 17 , wherein the allocated GPU device is shared spatially, wherein both the first and second tasks are concurrently executed using the allocated GPU device.
20. The system of claim 15 , wherein the computer server is part of a server cluster that provides GPU-as-a-Service.