Systems and methods for combining stochastic average gradient and hessian-free optimization for sequence training of deep neural networks

1. A method for training a deep neural network, comprising:

receiving, using at least one processor operatively coupled to a memory of a computer system, speech data for the training;

formatting, using the at least one processor, the speech data for the training;

dividing, using the at least one processor, the speech data into a plurality of subsets;

performing, using the at least one processor, Hessian-free sequence training on a first subset of the plurality of subsets of the speech data;

iteratively performing, using the at least one processor, the Hessian-free sequence training on successive subsets of the plurality of subsets of the speech data;

wherein iteratively performing the Hessian-free sequence training comprises:

processing the first subset of the speech data to generate a first gradient of loss in a first iteration;

processing a successive subset of the speech data to generate a second gradient of loss in a second iteration;

dynamically computing weights for the first gradient of loss and for the second gradient of loss; and

reusing gradient information from at least one previous iteration, wherein reusing the gradient information from the at least one previous iteration comprises integrating a weighted first gradient of loss and a weighted second gradient of loss to generate a solution to the second iteration; and

transmitting, using the at least one processor, a result of the iterative performance of the Hessian-free sequence training to the deep neural network.

2. The method of claim 1 , wherein the gradient information comprises average gradient information.

3. The method of claim 1 , wherein the weights for first gradient of loss and the second gradient of loss are chosen based on a difference between a held-out loss of the second iteration and a held-out loss of the first iteration.

4. The method of claim 1 , wherein the dynamic computing comprises changing weights assigned to the first gradient of loss and the second gradient of loss for different iterations.

5. The method of claim 1 , wherein the dynamic computing comprises dynamically estimating weights assigned to the first gradient of loss and the second gradient of loss to provide a loss function gradient before an iteration takes place.

6. The method of claim 1 , wherein weights assigned to the first gradient of loss and the second gradient of loss are based on a tunable parameter that controls exponentiation of the weights across the plurality of subsets.

7. The method of claim 1 , wherein the dynamic computing comprises using a weighting function to assign higher weights to gradients with held-out loss values closer to a held-out loss value of a current model, and lower weights to gradients with held-out loss values farther from the held-out loss value of the current model.