Accelerating deep neural network training with inconsistent stochastic gradient descent
Aspects of the present disclosure describe techniques for training a convolutional neural network using an inconsistent stochastic gradient descent (ISGD) algorithm. Training effort for training batches used by the ISGD algorithm are dynamically adjusted according to a determined loss for a given training batch which are classified into two sub states—well-trained or under-trained. The ISGD algorithm provides more iterations for under-trained batches while reducing iterations for well-trained ones.
1. A method comprising:
training, by at least one computing device, a convolutional neural network (CNN) with an inconsistent stochastic gradient descent (ISGD) algorithm using a training data set;
wherein said training includes executing the ISGD algorithm for a number of iterations;
wherein said computing device includes a plurality of processors and said method further comprises parallelizing, by the at least one computing device, at least a portion of any computations of the inconsistent stochastic gradient descent algorithm on the plurality of processors;
wherein the CNN is trained with batches of training data and the batches are classified as under-trained status or well-trained status based upon a loss determination;
wherein any batches classified as under-trained are continued to be trained until its determined loss falls below a pre-determined threshold or a pre-determined number of iterations is reached; and
wherein said continued training is defined by the following problem:
min
w
φ
w
(
d
t
)
=
1
2
ψ
w
(
d
t
)
-
limit
2
2
+
ɛ
2
n
w
w
-
w
t
-
1
2
2
where n w is the number of weight parameters in the network, ψ w is a loss function with weight vector w, d t is a batch of images and ε is a parameter for the second term that adjusts a conservative constraint that is preferably 10 −1 , limit, w t−1 and d t are constants, and the first term minimizes the difference between the loss of current under-trained batch d t and a control limit.
2. A method comprising:
training, by at least one computing device, a convolutional neural network (CNN) with an inconsistent stochastic gradient descent (ISGD) algorithm using a training data set;
wherein said training includes executing the ISGD algorithm for a number of iterations;
wherein said computing device includes a plurality of processors and said method further comprises parallelizing, by the at least one computing device, at least a portion of any computations of the inconsistent stochastic gradient descent algorithm on the plurality of processors;
wherein the CNN is trained with batches of training data, and batches are classified as under-trained status or well-trained status based upon a loss determination;
wherein a batch is classified as under-trained if the loss is larger than a control limit defined by:
ψ +3*σ ψ ;
wherein ψ is a descriptive statistic representing a running average loss and σ ψ is a descriptive statistic representing a running standard deviation during training.