Asynchronous optimization for sequence training of neural networks
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for obtaining, by a first sequence-training speech model, a first batch of training frames that represent speech features of first training utterances; obtaining, by the first sequence-training speech model, one or more first neural network parameters; determining, by the first sequence-training speech model, one or more optimized first neural network parameters based on (i) the first batch of training frames and (ii) the one or more first neural network parameters; obtaining, by a second sequence-training speech model, a second batch of training frames that represent speech features of second training utterances; obtaining one or more second neural network parameters; and determining, by the second sequence-training speech model, one or more optimized second neural network parameters based on (i) the second batch of training frames and (ii) the one or more second neural network parameters.
1. A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:
receiving, from a parameter server, a current set of parameter values for a neural network model; and
for each of a plurality of replicas of the neural network model:
obtaining a corresponding training utterance of one or more predetermined words spoken by a respective training speaker different than each other respective training speaker that spoke the corresponding training utterance obtained for the other ones of the plurality of replicas of the neural network model;
training, using the current set of parameter values for the neural network model and the corresponding training utterance obtained for the replica of the neural network model, the replica of the neural network model to generate corresponding updated parameter values for the neural network model; and
sending the corresponding updated parameter values for the neural network model to the parameter server.
2. The computer-implemented method of claim 1 , wherein training the replica of the neural network model comprises training the replica of the neural network model in parallel with training the other ones of the plurality of replicas of the neural network model.
3. The computer-implemented method of claim 1 , wherein training the replica of the neural network model comprises training the replica of the neural network model independently from training the other ones of the plurality of replicas of the neural network model.
4. The computer-implemented method of claim 1 , wherein training the replica of the neural network model comprises training the replica of the neural network model asynchronously with respect to training the other ones of the plurality of replicas of the neural network model.
5. The computer-implemented method of claim 1 , wherein training the replica of the neural network model comprises training the replica of the neural network model using stochastic gradient descent optimization.
6. The computer-implemented method of claim 1 , wherein sending the corresponding updated parameter values for the neural network model comprises sending the corresponding updated parameter values for the neural network model to the parameter server without sending the corresponding training utterance obtained for the replica of the neural network model to the parameter server.
7. The computer-implemented method of claim 1 , wherein:
the current set of parameter values for the neural network model comprise a current set of weights for the neural network model; and
the corresponding updated parameter values for the neural network model comprise corresponding updated weights for the neural network model.
8. The computer-implemented method of claim 1 , wherein the corresponding training utterance obtained for the replica of the neural network model is recorded by a respective computing device associated with the respective training speaker.
9. The computer-implemented method of claim 1 , wherein the neural network model is trained to indicate likelihoods that acoustic feature vectors represent different phonetic units.
10. The computer-implemented method of claim 1 , wherein the current set of parameter values for the neural network model comprise weights and biases of hidden layers of the neural network model.
11. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving, from a parameter server, a current set of parameter values for a neural network model; and
for each of a plurality of replicas of the neural network model:
obtaining a corresponding training utterance of one or more predetermined words spoken by a respective training speaker different than each other respective training speaker that spoke the corresponding training utterance obtained for the other ones of the plurality of replicas of the neural network model;
training, using the current set of parameter values for the neural network model and the corresponding training utterance obtained for the replica of the neural network model, the replica of the neural network model to generate corresponding updated parameter values for the neural network model; and
sending the corresponding updated parameter values for the neural network model to the parameter server.
12. The system of claim 11 , wherein training the replica of the neural network model comprises training the replica of the neural network model in parallel with training the other ones of the plurality of replicas of the neural network model.
13. The system of claim 11 , wherein training the replica of the neural network model comprises training the replica of the neural network model independently from training the other ones of the plurality of replicas of the neural network model.
14. The system of claim 11 , wherein training the replica of the neural network model comprises training the replica of the neural network model asynchronously with respect to training the other ones of the plurality of replicas of the neural network model.
15. The system of claim 11 , wherein training the replica of the neural network model comprises training the replica of the neural network model using stochastic gradient descent optimization.
16. The system of claim 11 , wherein sending the corresponding updated parameter values for the neural network model comprises sending the corresponding updated parameter values for the neural network model to the parameter server without sending the corresponding training utterance obtained for the replica of the neural network model to the parameter server.
17. The system of claim 11 , wherein:
the current set of parameter values for the neural network model comprise a current set of weights for the neural network model; and
the corresponding updated parameter values for the neural network model comprise corresponding updated weights for the neural network model.
18. The system of claim 11 , wherein the corresponding training utterance obtained for the replica of the neural network model is recorded by a respective computing device associated with the respective training speaker.
19. The system of claim 11 , wherein the neural network model is trained to indicate likelihoods that acoustic feature vectors represent different phonetic units.
20. The system of claim 11 , wherein the current set of parameter values for the neural network model comprise weights and biases of hidden layers of the neural network model.