Granted Patent B2

US 12,572,847 · App. 17/704,176 · Granted Mar 10, 2026

Inventors: Balasubramanian Chandrasekaran (Austin, TX); Lucas Avery Wilson (Cedar Park, TX); Dharmesh M. Patel (Round Rock, TX)

Assignee: Dell Products L.P.

G06N20/00

Abstract

Claims (97)

1 . A model training recalibration method comprising:

initiating a model training routine on a first set of virtual machines (VMs) hosted across a plurality of servers to produce an existing model training routine, wherein each server of the plurality of servers has insufficient computing resources to execute the model training routine;

receiving a notification comprising a recommendation for recalibrating the existing model training routine, wherein the notification comprises network resource information, wherein the network resource information further comprises:

specifying a change in available resources within a network,

creating a second set of VMs to execute the model training routine with a reduced processing time, and

deleting a first VM of the first set of VMs to execute the model training routine with the reduced processing time;

making a first determination, in response to receiving the notification, that the existing model training routine should be recalibrated; and

in response to the first determination, instantiating a recalibration of the existing model training routine based on the recommendation, wherein the recalibration comprises:

migrating a dataset associated with the model training routine to a second VM of the second set of VMs, wherein the recalibration enables execution of the model training routine on a single VM, wherein the execution is associated with the reduced processing time.

2 . The model training recalibration method of claim 1 ,

wherein the recommendation is based on the network resource information, and

wherein the first determination that the existing model training routine should be recalibrated is based on policies and priorities associated with the existing model training routine.

3 . The model training recalibration method of claim 2 ,

wherein the change in available resources comprises additional resources becoming available within the network,

wherein the recalibration of the existing model training routine comprises using the additional resources to execute the existing model training routine, and

wherein the additional resources comprise a graphical processing unit (GPU) of a server that is, prior to the recalibration of the existing model training routine, not being used to execute the existing model training routine.

4 . The model training recalibration method of claim 2 , further comprising:

waiting a predetermined waiting period before instantiating the recalibration, wherein the predetermined waiting period is based on the policies and priorities associated with the existing model training routine; and

transmitting, to an entity that transmitted the notification, a response specifying the predetermined waiting period, wherein

the recalibration of the existing model training routine is instantiated after the predetermined waiting period expires.

5 . The model training recalibration method of claim 2 , further comprising:

after recalibrating the existing model training routine, receiving a second notification comprising a second recommendation for further recalibrating the existing model training routine;

making a second determination, in response to receiving the second notification and based on the policies and priorities associated with the existing model training routine, that the second recommendation should be ignored; and

in response to the second determination, not further recalibrating the existing model training by ignoring the second recommendation.

6 . The model training recalibration method of claim 1 ,

wherein the model training is a synchronous data parallel training, and

wherein the recalibration is instantiated after an all reduce operation of the synchronous data parallel training is completed.

7 . The model training recalibration method of claim 1 ,

wherein the existing model training routine is executing on a first set of network resources of the network,

wherein the recommendation is associated with a second set of network resources of the network, wherein the second set of network resources are distinct and independent from the first set of network resources, and

wherein instantiating the recalibration of the existing model training routine based on the recommendation comprises:

stopping the execution of the existing model training routine on the first set of network resources;

in response to stopping the execution of the existing model training routine on the first set of network resources, instantiating of a copy of the existing model training routine, wherein the copy comprises data reflecting a training progress of the existing model training routine up to when the execution of the existing model training routine is stopped;

transmitting the copy of the existing model training routine to the second set of network resources to resume the execution of the existing model training routine using the second set of network resources.

8 . A non-transitory computer readable medium (CRM) comprising computer readable program code, which when executed by a computer processor enables the computer processor to perform a model training recalibration method, the model training recalibration method comprising:

initiating a model training routine on a first set of virtual machines (VMs) hosted across a plurality of servers to produce an existing model training routine, wherein each server of the plurality of servers has insufficient computing resources to execute the model training routine;

receiving a notification comprising a recommendation for recalibrating the existing model training routine, wherein the notification comprises network resource information, wherein the network resource information further comprises:

specifying a change in available resources within a network,

creating a second set of VMs to execute the model training routine with a reduced processing time, and

deleting a first VM of the first set of VMs to execute the model training routine with the reduced processing time;

making a first determination, in response to receiving the notification, that the existing model training routine should be recalibrated; and

in response to the first determination, instantiating a recalibration of the existing model training routine based on the recommendation, wherein the recalibration comprises:

migrating a dataset associated with the model training routine to a second VM of the second set of VMs, wherein the recalibration enables execution of the model training routine on a single VM, wherein the execution is associated with the reduced processing time.

9 . The CRM of claim 8 ,

wherein the recommendation is based on the network resource information, and

wherein the first determination that the existing model training routine should be recalibrated is based on policies and priorities associated with the existing model training routine.

10 . The CRM of claim 8 ,

wherein the change in available resources comprises additional resources becoming available within the network,

wherein the recalibration of the existing model training routine comprises using the additional resources to execute the existing model training routine, and

wherein the additional resources comprise a graphical processing unit (GPU) of a server that is, prior to the recalibration of the existing model training routine, not being used to execute the existing model training routine.

11 . The CRM of claim 9 , further comprising:

waiting a predetermined waiting period before instantiating the recalibration, wherein the predetermined waiting period is based on the policies and priorities associated with the existing model training routine; and

transmitting, to an entity that transmitted the notification, a response specifying the predetermined waiting period, wherein

the recalibration of the existing model training routine is instantiated after the predetermined waiting period expires.

12 . The CRM of claim 9 , further comprising:

after recalibrating the existing model training routine, receiving a second notification comprising a second recommendation for further recalibrating the existing model training routine;

making a second determination, in response to receiving the second notification and based on the policies and priorities associated with the existing model training routine, that the second recommendation should be ignored; and

in response to the second determination, not further recalibrating the existing model training by ignoring the second recommendation.

13 . The CRM of claim 8 ,

wherein the model training is a synchronous data parallel training, and

wherein the recalibration is instantiated after an all reduce operation of the synchronous data parallel training is completed.

14 . The CRM of claim 8 ,

wherein the existing model training routine is executing on a first set of network resources of the network,

wherein the recommendation is associated with a second set of network resources of the network, wherein the second set of network resources are distinct and independent from the first set of network resources, and

wherein instantiating the recalibration of the existing model training routine based on the recommendation comprises:

stopping the execution of the existing model training routine on the first set of network resources;

in response to stopping the execution of the existing model training routine on the first set of network resources, instantiating of a copy of the existing model training routine, wherein the copy comprises data reflecting a training progress of the existing model training routine up to when the execution of the existing model training routine is stopped;

transmitting the copy of the existing model training routine to the second set of network resources to resume the execution of the existing model training routine using the second set of network resources.

15 . A system comprising:

a memory; and

a processor coupled to the memory, wherein the processor is configured to execute a model training recalibration method comprising:

initiating a model training routine on a first set of virtual machines (VMs) hosted across a plurality of servers to produce an existing model training routine, wherein each server of the plurality of servers has insufficient computing resources to execute the model training routine;

receiving a notification comprising a recommendation for recalibrating the existing model training routine, wherein the notification comprises network resource information, wherein the network resource information further comprises:

specifying a change in available resources within a network,

creating a second set of VMs to execute the model training routine with a reduced processing time, and

deleting a first VM of the first set of VMs to execute the model training routine with the reduced processing time;

making a first determination, in response to receiving the notification, that the existing model training routine should be recalibrated; and

in response to the first determination, instantiating a recalibration of the existing model training routine based on the recommendation, wherein the recalibration comprises:

migrating a dataset associated with the model training routine to a second VM of the second set of VMs, wherein the recalibration enables execution of the model training routine on a single VM, wherein the execution is associated with the reduced processing time.

16 . The system of claim 15 ,

wherein the recommendation is based on the network resource information, and

wherein the first determination that the existing model training routine should be recalibrated is based on policies and priorities associated with the existing model training routine.

17 . The system of claim 16 ,

wherein the change in available resources comprises additional resources becoming available within the network,

wherein the recalibration of the existing model training routine comprises using the additional resources to execute the existing model training routine, and

wherein the additional resources comprise a graphical processing unit (GPU) of a server that is, prior to the recalibration of the existing model training routine, not being used to execute the existing model training routine.

18 . The system of claim 16 , further comprising:

waiting a predetermined waiting period before instantiating the recalibration, wherein the predetermined waiting period is based on the policies and priorities associated with the existing model training routine; and

transmitting, to an entity that transmitted the notification, a response specifying the predetermined waiting period, wherein

the recalibration of the existing model training routine is instantiated after the predetermined waiting period expires.

19 . The system of claim 16 , further comprising:

after recalibrating the existing model training routine, receiving a second notification comprising a second recommendation for further recalibrating the existing model training routine;

making a second determination, in response to receiving the second notification and based on the policies and priorities associated with the existing model training routine, that the second recommendation should be ignored; and

in response to the second determination, not further recalibrating the existing model training by ignoring the second recommendation.

20 . The system of claim 15 ,

wherein the model training is a synchronous data parallel training, and

wherein the recalibration is instantiated after an all reduce operation of the synchronous data parallel training is completed.

Assignments (1)

Continuity (1)

References Cited (22)