Managing access to a resource pool of graphics processing units under fine grain control
A graphics processing unit (GPU) service platform includes a control server, and a cluster of GPU servers each having one or more GPU devices. The control server receives a service request from a client system for GPU processing services, allocates multiple GPU servers nodes within the cluster to handle GPU processing tasks specified by the service request by logically binding the allocated GPU server nodes, and designating one of the at least two GPU servers as a master server, and send connection information to the client system to enable the client system to connect to the master server. The master GPU server node receives a block of GPU program code transmitted from the client system, which is associated with the GPU processing tasks specified by the service request, processes the block of GPU program code using the GPU devices of the logically bound GPU servers, and returns processing results to the client system.
1. A method, comprising:
receiving, by a graphics processing unit (GPU) service platform, a service request from a client system for GPU processing services provided by the GPU service platform, wherein the GPU service platform comprises a server cluster of GPU server nodes, wherein each GPU server node within the server cluster comprises one or more GPU devices;
allocating, by the GPU service platform, at least two GPU server nodes within the cluster of GPU server nodes to handle GPU processing tasks specified by the service request, wherein allocating comprises logically binding the at least two GPU server nodes, and designating one of the at least two GPU server nodes as a master GPU server node;
sending, by the GPU service platform, connection information to the client system to enable the client system to establish a network connection to the master GPU server node;
receiving, by the master GPU server node, a block of GPU program code transmitted from the client system over the network connection, wherein the block of GPU program code is associated with the GPU processing tasks specified by the service request;
processing the block of GPU program code using the GPU devices of the logically bound GPU server nodes under the control of the master GPU server node; and
returning, by the master GPU server node, processing results to the client system over the network connection;
wherein each GPU server node within the server cluster comprises a queue-based GPU virtualization and management system;
wherein the queue-based GPU virtualization and management system of each GPU server node within the server cluster comprises (i) a task queue service module that is configured to enqueue GPU processing tasks in a task queue, and (ii) a task scheduler/dispatcher module that is configured to schedule execution times for GPU processing tasks that are enqueued in the task queue, and to dispatch the GPU processing tasks enqueued in the task queue to a server backend GPU worker process to handle execution of the GPU processing tasks at their scheduled times using the GPU devices of the GPU server node;
wherein logically binding the at least two GPU server nodes comprises logically binding the queue-based GPU virtualization and management systems of the at least two GPU server nodes to generate a single logical queue-based GPU virtualization and management system which is managed by the master server node to control queueing, dispatching and processing of the GPU processing tasks associated with the service request of the client system; and
wherein logically binding the queue-based GPU virtualization and management systems of the at least two GPU server nodes comprises logically establishing a single task queue, a single task queue service module, and a single task scheduler/dispatcher module, which are managed by the master server node to control queueing and dispatching of the GPU processing tasks associated with the service request of the client system, and to control the processing of the block of GPU program code using the GPU devices of the logically bound GPU server nodes.
2. The method of claim 1 , wherein allocating further comprises establishing peer-to-peer communication links between the at least two GPU server nodes that are allocated to handle the service request received from the client system.
3. The method of claim 1 , wherein the service request received from the client system specifies a number of GPU devices for handling the processing tasks specified by the service request, and wherein allocating comprises determining at least two GPU server nodes within the cluster of GPU server nodes having a combined number of available GPU devices that meet the specified number of GPU devices in the service request.
4. The method of claim 1 , wherein the service request received from the client system specifies one or more attributes associated with the GPU processing tasks specified by the service request, wherein the one or more attributes specify at least one of a quality of service (QoS) and a priority level for executing the GPU processing tasks, and wherein allocating comprises determining at least two GPU server nodes within the cluster of GPU server nodes having sufficient processing resources to satisfy the specified one or more attributes.
5. The method of claim 1 , wherein processing the block of GPU program code using the GPU devices of the logically bound GPU server nodes comprises coordinating, by the master server node, access to the GPU devices across the logically bound GPU server nodes.
6. The method of claim 1 , wherein allocating further comprises:
accessing a database of GPU server registration information which comprises registration information for each GPU server node within the server cluster which is registered with the GPU service platform, wherein the registration information for a given GPU server node specifies a number of available GPU devices residing on the given GPU server node; and
selecting two or more registered GPU server nodes which collectively comprise a total number of available GPU devices which is sufficient to handle the service request.
7. The method of claim 1 , wherein allocating further comprises:
accessing a database of policies maintained by the GPU service platform, wherein the policies are associated with different client systems or GPU processing tasks, wherein the policies specify GPU server nodes within the server cluster which can be allocated for handling service requests associated with the different client systems or GPU processing tasks based on locations of the GPU server nodes; and
selecting two or more GPU server nodes which reside in locations that correspond to a policy in the database of policies, which is associated with the client system or the GPU processing tasks specified by the service request received from the client system.
8. The method of claim 1 , wherein the cluster of GPU server nodes resides within a single data center.
9. The method of claim 1 , wherein the cluster of GPU server nodes is distributed over a plurality of different data centers or computing platforms.
10. An article of manufacture comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code is executable by one or more processors to implement a process comprising:
receiving, by a graphics processing unit (GPU) service platform, a service request from a client system for GPU processing services provided by the GPU service platform, wherein the GPU service platform comprises a server cluster of GPU server nodes, wherein each GPU server node within the server cluster comprises one or more GPU devices;
allocating, by the GPU service platform, at least two GPU server nodes within the cluster of GPU server nodes to handle GPU processing tasks specified by the service request, wherein allocating comprises logically binding the at least two GPU server nodes, and designating one of the at least two GPU server nodes as a master GPU server node;
sending, by the GPU service platform, connection information to the client system to enable the client system to establish a network connection to the master GPU server node;
receiving, by the master GPU server node, a block of GPU program code transmitted from the client system over the network connection, wherein the block of GPU program code is associated with the GPU processing tasks specified by the service request;
processing the block of GPU program code using the GPU devices of the logically bound GPU server nodes; and
returning, by the master GPU server node, processing results to the client system over the network connection;
wherein each GPU server node within the server cluster comprises a queue-based GPU virtualization and management system;
wherein the queue-based GPU virtualization and management system of each GPU server node within the server cluster comprises (i) a task queue service module that is configured to enqueue GPU processing tasks in a task queue, and (ii) a task scheduler/dispatcher module that is configured to schedule execution times for GPU processing tasks that are enqueued in the task queue, and to dispatch the GPU processing tasks enqueued in the task queue to a server backend GPU worker process to handle execution of the GPU processing tasks at their scheduled times using the GPU devices of the GPU server node;
wherein logically binding the at least two GPU server nodes comprises logically binding the queue-based GPU virtualization and management systems of the at least two GPU server nodes to generate a single logical queue-based GPU virtualization and management system which is managed by the master server node to control queueing, dispatching and processing of the GPU processing tasks associated with the service request of the client system; and
wherein logically binding the queue-based GPU virtualization and management systems of the at least two GPU server nodes comprises logically establishing a single task queue, a single task queue service module, and a single task scheduler/dispatcher module, which are managed by the master server node to control queueing and dispatching of the GPU processing tasks associated with the service request of the client system, and to control the processing of the block of GPU program code using the GPU devices of the logically bound GPU server nodes.
11. The article of manufacture of claim 10 , wherein allocating further comprises establishing peer-to-peer communication links between the at least two GPU server nodes that are allocated to handle the service request received from the client system.
12. The article of manufacture of claim 10 , wherein the service request received from the client system specifies a number of GPU devices for handling the processing tasks specified by the service request, and wherein allocating comprises determining at least two GPU server nodes within the cluster of GPU server nodes having a combined number of available GPU devices that meet the specified number of GPU devices in the service request.
13. The article of manufacture of claim 10 , wherein the cluster of GPU server nodes is distributed over a plurality of different data centers or computing platforms.
14. The article of manufacture of claim 10 , wherein the service request received from the client system specifies one or more attributes associated with the GPU processing tasks specified by the service request, wherein the one or more attributes specify at least one of a quality of service (QoS) and a priority level for executing the GPU processing tasks, and wherein allocating comprises determining at least two GPU server nodes within the cluster of GPU server nodes having sufficient processing resources to satisfy the specified one or more attributes.
15. A graphics processing unit (GPU) service platform, comprising:
a server cluster of GPU server nodes wherein each GPU server node within the server cluster comprises one or more GPU devices; and
a control server configured to:
receive a service request from a client system for GPU processing services provided by the GPU service platform;
allocate at least two GPU server nodes within the cluster of GPU server nodes to handle GPU processing tasks specified by the service request, wherein allocating comprises logically binding the at least two GPU server nodes, and designating one of the at least two GPU server nodes as a master GPU server node; and
send connection information to the client system to enable the client system to establish a network connection to the master GPU server node; and
wherein the master GPU server node is configured to:
receive a block of GPU program code transmitted from the client system over the network connection, wherein the block of GPU program code is associated with the GPU processing tasks specified by the service request;
process the block of GPU program code using the GPU devices of the logically bound GPU server nodes; and
return processing results to the client system over the network connection;
wherein each GPU server node within the server cluster comprises a queue-based GPU virtualization and management system;
wherein the queue-based GPU virtualization and management system of each GPU server node within the server cluster comprises (i) a task queue service module that is configured to enqueue GPU processing tasks in a task queue, and (ii) a task scheduler/dispatcher module that is configured to schedule execution times for GPU processing tasks that are enqueued in the task queue, and to dispatch the GPU processing tasks enqueued in the task queue to a server backend GPU worker process to handle execution of the GPU processing tasks at their scheduled times using the GPU devices of the GPU server node;
wherein logically binding the at least two GPU server nodes comprises logically binding the queue-based GPU virtualization and management systems of the at least two GPU server nodes to generate a single logical queue-based GPU virtualization and management system which is managed by the master server node to control queueing, dispatching and processing of the GPU processing tasks associated with the service request of the client system; and
wherein logically binding the queue-based GPU virtualization and management systems of the at least two GPU server nodes comprises logically establishing a single task queue, a single task queue service module, and a single task scheduler/dispatcher module, which are managed by the master server node to control queueing and dispatching of the GPU processing tasks associated with the service request of the client system, and to control the processing of the block of GPU program code using the GPU devices of the logically bound GPU server nodes.
16. The GPU service platform of claim 15 , wherein the control server is further configured cause the at least two GPU server nodes, which are allocated to handing the service request, to establish peer-to-peer communication links between the at least two GPU server nodes.
17. The GPU service platform of claim 15 , wherein the service request received from the client system specifies a number of GPU devices for handling the processing tasks specified by the service request, and wherein the control server is further configured to determine at least two GPU server nodes within the cluster of GPU server nodes having a combined number of available GPU devices that meet the specified number of GPU devices in the service request.
18. The GPU service platform of claim 15 , wherein the cluster of GPU server nodes resides within a single data center.
19. The GPU service platform of claim 15 , wherein the cluster of GPU server nodes is distributed over a plurality of different data centers or computing platforms.
20. The GPU service platform of claim 15 , wherein the service request received from the client system specifies one or more attributes associated with the GPU processing tasks specified by the service request, wherein the one or more attributes specify at least one of a quality of service (QoS) and a priority level for executing the GPU processing tasks, and wherein allocating comprises determining at least two GPU server nodes within the cluster of GPU server nodes having sufficient processing resources to satisfy the specified one or more attributes.