Training a neural network using periodic sampling over model weights
A computer-implemented method includes: initializing model parameters for training a neural network; performing a forward pass and backpropagation for a first minibatch of training data; determining a new weight value for each of a plurality of nodes of the neural network using a gradient descent of the first minibatch; for each determined new weight value, determining whether to update a running mean corresponding to a weight of a particular node; based on a determination to update the running mean, calculating a new mean weight value for the particular node using the determined new weight value; updating the weight parameters for all nodes based on the calculated new mean weight values corresponding to each node; assigning the running mean as the weight for the particular node when training on the first minibatch is completed; and reinitializing running means for all nodes at a start of training a second minibatch.
1. A computer-implemented method for training a neural network, the computer-implemented method comprising:
initializing one or more model parameters for training the neural network;
performing a forward pass and back propagation for a minibatch of training data comprising a plurality of batches of training data;
determining a new weight value for each of a plurality of nodes of the neural network based on an optimization algorithm;
for each determined new weight value, determining whether to update a running mean corresponding to a weight of each node from the plurality of nodes, wherein determining whether to update the running mean is based on:
whether a current batch of training data falls within a predefined last subset of batches of the minibatch; and
performing a random determination having a probability based on a value included in the one or more model parameters;
based on a determination to update the running mean, calculating a new mean weight value for each node using the determined new weight value,
wherein when the current batch of training data does not fall within the predefined last subset of batches of the minibatch, the running mean is not updated using the determined new weight value;
updating weight parameters for all nodes based on the calculated new mean weight values corresponding to each node;
assigning the running mean as the weight for each node when training on K number of minibatches is completed, wherein K is a predefined number; and
reinitializing running means for all nodes in the neural network at a start of training a K+1 minibatch of the training data.
2. The computer-implemented method of claim 1 , further comprising storing the running mean corresponding to the weight of each node before updating the running mean as a current running mean, wherein the running mean is reinitialized to a value of the stored running mean.
3. The computer-implemented method of claim 1 , wherein the value included in the one or more model parameters is a percentage representing a desired sampling percentage of the minibatch for updating the running mean.
4. The computer-implemented method of claim 1 , wherein the one or more model parameters includes a percentage value representing the predefined last subset of batches with respect to a total number of batches of the minibatch.
5. The computer-implemented method of claim 4 , further comprising reinitializing the running mean for each node for performing training on a next minibatch of training data.
6. The computer-implemented method of claim 4 , wherein the percentage value representing the predefined last subset of batches is in a range of 2 to 5%.
7. The computer-implemented method of claim 1 , wherein the determined new weight value is based on a running momentum term included in the one or more model parameters.
8. The computer-implemented method of claim 7 , wherein the running momentum term is updated after each batch of the minibatch such that new weight values for calculating the new mean weight value are increasingly weighted toward an end of the minibatch.
9. An edge device comprising one or more processors; a non-transitory memory; and one or more programs stored in the non-transitory memory, which, when executed by the one or more processors, cause the edge device to be configured to perform:
initializing one or more model parameters for training a neural network;
performing a forward pass and back propagation using a minibatch of training data comprising a plurality of batches of training data;
determining a new weight value for each of a plurality of nodes of the neural network based on an optimization algorithm;
for each determined new weight value, determining whether to update a running mean corresponding to a weight of each node from the plurality of nodes, wherein determining whether to update the running mean is based on:
whether a current batch of training data falls within a predefined last subset of batches of the minibatch; and
performing a random determination having a probability based on a value included in the one or more model parameters;
based on a determination to update the running mean, calculating a new mean weight value for each node using the determined new weight value,
wherein when the current batch of training data does not fall within the predefined last subset of batches of the minibatch, the running mean is not updated using the determined new weight value;
updating weight parameters for all nodes based on the calculated new mean weight value corresponding to each node;
assigning the running mean as the weight for each node when training on K number of minibatches is completed, wherein K is a predefined number; and
reinitializing running means for all nodes in the neural network at a start of training a K+1 minibatch of the training data.
10. The edge device of claim 9 , wherein the determined new weight value is based on a running momentum term included in the one or more model parameters.
11. A non-transitory memory storing one or more programs, which, when executed by the one or more processors of an edge device, cause the edge device to be configured to perform:
initializing one or more model parameters for training a neural network;
performing a forward pass and back propagation using a minibatch of training data comprising a plurality of batches of training data;
determining a new weight value for each of a plurality of nodes of the neural network based on an optimization algorithm;
for each determined new weight value, determining whether to update a running mean corresponding to a weight of each node from the plurality of nodes, wherein determining whether to update the running mean is based on:
whether a current batch of training data falls within a predefined last subset of batches of the minibatch; and
performing a random determination having a probability based on a value included in the one or more model parameters;
based on a determination to update the running mean, calculating a new mean weight value for each node using the determined new weight value,
wherein when the current batch of training data does not fall within the predefined last subset of batches of the minibatch, the running mean is not updated using the determined new weight value;
updating weight parameters for all nodes based on the calculated new mean weight values corresponding to each node;
assigning the running mean as the weight for each node when training on K number of minibatches is completed, wherein K is a predefined number; and
reinitializing running means for all nodes in the neural network at a start of training a K+1 minibatch of the training data.
12. The non-transitory memory of claim 11 , wherein the determined new weight value is based on a running momentum term included in the one or more model parameters.
13. The non-transitory memory of claim 12 , wherein the running momentum term is updated after each batch of the minibatch such that new weight values for calculating the new mean weight value are increasingly weighted toward an end of the minibatch.