Scalable graphic processing unit allocation infrastructure

1 . A system comprising:

a non-transitory data store storing computer-executable instructions; and

one or more processors in communication with the non-transitory data store, wherein the computer-executable instructions, when executed by the one or more processors, configure the one or more processors to:

obtain a plurality of requests to access pooled graphic processing unit (GPU) resources for GPU usage jobs, wherein a first request of the plurality of requests includes a time estimate for completing a first GPU usage job and a number of GPUs requested for the first GPU usage job;

identify usage classifications according to the plurality of requests and the GPU usage jobs, wherein a first usage classifications comprises, for the first GPU usage job, at least one of a first subset of GPU usage jobs, a second subset of GPU usage jobs, and a third subset of GPU usage jobs,

wherein the first subset of GPU usage jobs is scheduled to use GPU resources to train large language models, wherein GPU resources for the first subset of GPU usage jobs are associated with a proximity parameter, wherein the proximity parameter corresponds to a plurality of GPUs adjacent to perform the first subset of GPU usage jobs, wherein the plurality of GPUs are physically adjacent within a networked environment,

wherein the second subset of GPU usage jobs is scheduled to use the GPU resources for interactive use, wherein the GPU resources for the second subset of GPU usage jobs are associated with a timing constraint for the second subset of GPU usage jobs to be performed during an N-hour reservation after a start time,

wherein the third subset of GPU usage jobs is scheduled to use remaining GPU resources of the pooled GPU resources, wherein the GPU resources for the third subset of GPU usage jobs include executing offline inferencing models on available GPU resources;

allocate available GPU resources to perform the GPU usage jobs;

control the available GPU resources to perform the GPU usage jobs;

identify unused GPU resources from the pooled GPU resources; and

schedule the GPU usage jobs from the third subset of GPU usage jobs to achieve a high utilization rate.

2 . The system of claim 1 , wherein to allocate the available GPU resources, the one or more processors are configured to use a greedy algorithm, wherein the greedy algorithm allocates resources to the GPU usage jobs according to the usage classifications.

3 . The system of claim 1 , wherein the one or more processors are further configured to, in response to identifying the unused GPU resources, schedule additional jobs when the pooled GPU resources are below an average expected utilization rate.

4 . The system of claim 1 , wherein allocation of resources is computed at least in part by solving for a bin packing problem corresponding to the usage classifications.

5 . A computer-implemented method comprising:

under control of a computing system comprising memory and one or more computer processors configured to execute specific instructions:

classifying a first subset of graphics processing unit (GPU) usage jobs in a first class associated with a proximity parameter;

classifying a second subset of GPU usage jobs in a second class associated with a timing constraint;

classifying a third subset of GPU usage jobs in a third class;

executing, using a first subset of GPUs from a GPU pool, a first set of individual usage jobs classified in the first class;

executing, using a second subset of GPUs from the GPU pool, a second set of individual usage jobs classified in the second class; and

executing, using a third subset of GPUs remaining from the GPU pool after excluding the first subset of GPUs and the second subset of GPUs, a third set of individual usage jobs classified in the third class.

6 . The computer-implemented method of claim 5 , wherein the proximity parameter corresponds to a preference for each of a plurality of GPUs for performing the first subset of GPU usage jobs being located, within a networked environment, adjacent to each other GPU of the plurality of GPUs within the networked environment.

7 . The computer-implemented method of claim 5 , wherein the first subset of GPU usage jobs includes large language model training.

8 . The computer-implemented method of claim 5 , wherein the timing constraint corresponds to performing the second subset of GPU usage jobs during a scheduled reservation after a start time and before a completion time.

9 . The computer-implemented method of claim 5 , wherein the second subset of GPU usage jobs includes interactive experimentation.

10 . The computer-implemented method of claim 5 , wherein the third subset of GPU usage jobs includes jobs allocated a particular number of GPU hours by a particular deadline, wherein the third subset of GPU usage jobs is performed without being associated with the proximity parameter or the timing constraint, wherein the third subset of GPU usage jobs is performed with GPUs remaining after assigning the first subset of GPUs to perform the first subset of GPU usage jobs and the second subset of GPUs to perform the second subset of GPU usage jobs.

11 . The computer-implemented method of claim 5 , wherein the third subset of GPU usage jobs includes offline inferencing.

12 . The computer-implemented method of claim 5 , further comprising:

classifying a fourth subset of GPU usage jobs in a fourth class; and

executing, using a fourth subset of GPUs from a GPU pool, a fourth set of individual usage jobs classified in the fourth class, wherein the fourth subset of GPU usage jobs includes customer access.

13 . The computer-implemented method of claim 5 , wherein executing the GPU usage jobs is based at least in part on an account associated to each of the GPU usage jobs, wherein the account corresponds to a resource usage budget used to allocate GPU resources from the GPU pool.

14 . A system comprising:

a non-transitory data store storing computer-executable instructions; and

a processor in communication with the non-transitory data store, wherein the computer-executable instructions, when executed by the processor, configure the processor to:

classify a first subset of resource usage jobs in a first class associated with a proximity parameter;

classify a second subset of resource usage jobs in a second class associated with a timing constraint;

classify a third subset of resource usage jobs in a third class;

execute, using a first subset of computational resources from a computational resource pool, a first set of individual usage jobs classified in the first class;

execute, using a second subset of computational resources from the computational resource pool, a second set of individual usage jobs classified in the second class; and

execute, using a third subset of computational resources remaining from the computational resource pool after excluding the first subset of computational resources and the second subset of computational resources, a third set of individual usage jobs classified in the third class.

15 . The system of claim 14 , wherein the proximity parameter corresponds to a preference for each a plurality of computational resources for performing the first subset of resource usage jobs being located, within a networked environment, adjacent to each other computational resource of the plurality of computational resources within the networked environment.

16 . The system of claim 14 , wherein the first subset of resource usage jobs includes large language model training.

17 . The system of claim 14 , wherein the timing constraint corresponds to performing the second subset of resource usage jobs during a scheduled reservation after a start time and before a completion time.

18 . The system of claim 14 , wherein the second subset of resource usage jobs includes interactive experimentation.

19 . The system of claim 14 , wherein the third subset of resource usage jobs includes jobs allocated a particular number of computational resource hours by a particular deadline, wherein the third subset of resource usage jobs is performed without being associated with the proximity parameter or the timing constraint, wherein the third subset of resource usage jobs is performed with computational resources remaining after assigning the first subset of computational resources to perform the first subset of resource usage jobs and the second subset of computational resources to perform the second subset of resource usage jobs.

20 . The system of claim 14 , wherein the third subset of resource usage jobs includes offline inferencing.