Graphics processing unit training job allocation
View Patent ↗A computer-implemented method for training a machine learning model includes receiving a first training job at a processing device of a computer system having a plurality of graphics processing unit (GPU) resources, the first training job being part of a set of training jobs, and determining an amount of available memory in each GPU resource of the plurality of GPU resources. The method also includes loading the training job into one or more GPU resources with at least one second training job. The loading includes determining a cost model indicating an efficiency cost of each of a plurality of packing patterns, and packing the first training job and the second training job into the one or more GPU resources according to a packing pattern associated with a lowest efficiency cost.
1 . A computer-implemented method for training a machine learning model, the method comprising:
receiving a first training job at a processing device of a computer system having a plurality of graphics processing unit (GPU) resources, the first training job being part of a set of training jobs;
determining an amount of available memory in each GPU resource of the plurality of GPU resources; and
loading the training job into one or more GPU resources with at least one second training job, wherein the loading includes:
determining a cost model indicating an efficiency cost of each of a plurality of packing patterns, and packing the first training job and the second training job into the one or more GPU resources according to a packing pattern associated with a lowest efficiency cost.
2 . The method of claim 1 , wherein determining the cost model includes performing a test run for the first training job, the test run including a selected number of iterations, and collecting runtime metrics for the first training job.
3 . The method of claim 2 , wherein determining the cost model includes generating a packing pair for each iteration, each packing pair including an iteration for the first training job and an iteration for the second training job.
4 . The method of claim 3 , wherein determining the cost model includes calculating the efficiency cost for each packing pair, and selecting the packing pattern associated with a packing pair having the lowest efficiency cost.
5 . The method of claim 2 , further comprising scaling in an existing job to reduce a number of GPU resources executing the existing job, to provide sufficient memory to perform the test run.
6 . The method of claim 1 , wherein determining the cost model includes splitting a training model associated with the first training job.
7 . The method of claim 1 , wherein the loading includes adaptively packing the first training job into the one or more GPU resources based on the packing pattern associated with the lowest efficiency cost, wherein adaptively packing includes scaling up the first training job based on there being insufficient memory to hold the first training job and the second training job in a single GPU resource.
8 . The method of claim 1 , wherein the first training job is configured for elastic distributed training.
9 . A system for training a machine learning model, the system comprising:
a processor in communication with a computer system having a plurality of graphics processing unit (GPU) resources, the processor configured to perform:
receiving a first training job, the first training job being part of a set of training jobs;
determining an amount of available memory in each GPU resource of the plurality of GPU resources; and
loading the training job into one or more GPU resources with at least one second training job, wherein the loading includes:
determining a cost model indicating an efficiency cost of each of a plurality of packing patterns, and packing the first training job and the second training job into the one or more GPU resources according to a packing pattern associated with a lowest efficiency cost.
10 . The system of claim 9 , wherein determining the cost model includes performing a test run for the first training job, the test run including a selected number of iterations, and collecting runtime metrics for the first training job.
11 . The system of claim 10 , wherein determining the cost model includes generating a packing pair for each iteration, each packing pair including an iteration for the first training job and an iteration for the second training job.
12 . The system of claim 11 , wherein determining the cost model includes calculating the efficiency cost for each packing pair, and selecting the packing pattern associated with a packing pair having the lowest efficiency cost.
13 . The system of claim 10 , wherein the processor is further configured to perform scaling in an existing job to reduce a number of GPU resources executing the existing job, to provide sufficient memory to perform the test run.
14 . The system of claim 9 , wherein the loading includes adaptively packing the first training job into the one or more GPU resources based on the packing pattern associated with the lowest efficiency cost, wherein adaptively packing includes scaling up the first training job based on there being insufficient memory to hold the first training job and the second training job in a single GPU resource.
15 . A computer program product for training a machine learning model, the computer program product comprising:
a non-transitory storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising:
receiving a first training job at a processing device of a computer system having a plurality of graphics processing unit (GPU) resources, the first training job being part of a set of training jobs;
determining an amount of available memory in each GPU resource of the plurality of GPU resources; and
loading the training job into one or more GPU resources with at least one second training job, wherein the loading includes:
determining a cost model indicating an efficiency cost of each of a plurality of packing patterns, and packing the first training job and the second training job into the one or more GPU resources according to a packing pattern associated with a lowest efficiency cost.
16 . The computer program product of claim 15 , wherein determining the cost model includes performing a test run for the first training job, the test run including a selected number of iterations, and collecting runtime metrics for the first training job.
17 . The computer program product of claim 16 , wherein determining the cost model includes generating a packing pair for each iteration, each packing pair including an iteration for the first training job and an iteration for the second training job.
18 . The computer program product of claim 17 , wherein determining the cost model includes calculating the efficiency cost for each packing pair, and selecting the packing pattern associated with a packing pair having the lowest efficiency cost.
19 . The computer program product of claim 16 , wherein the method further includes scaling in an existing job to reduce a number of GPU resources executing the existing job, to provide sufficient memory to perform the test run.
20 . The computer program product of claim 15 , wherein the loading includes adaptively packing the first training job into the one or more GPU resources based on the packing pattern associated with the lowest efficiency cost, wherein adaptively packing includes scaling up the first training job based on there being insufficient memory to hold the first training job and the second training job in a single GPU resource.