Minimum word error rate training for attention-based sequence-to-sequence models
Methods, systems, and apparatus, including computer programs encoded on computer-readable storage media, for speech recognition using attention-based sequence-to-sequence models. In some implementations, audio data indicating acoustic characteristics of an utterance is received. A sequence of feature vectors indicative of the acoustic characteristics of the utterance is generated. The sequence of feature vectors is processed using a speech recognition model that has been trained using a loss function that uses N-best lists of decoded hypotheses, the speech recognition model including an encoder, an attention module, and a decoder. The encoder and decoder each include one or more recurrent neural network layers. A sequence of output vectors representing distributions over a predetermined set of linguistic units is obtained. A transcription for the utterance is obtained based on the sequence of output vectors. Data indicating the transcription of the utterance is provided.
1. A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:
receiving a sequence of feature vectors indicative of acoustic characteristics of a training utterance;
receiving a ground-truth label sequence corresponding to the training utterance; and
training a speech recognition model to minimize word error rate by performing operations comprising:
processing, using the speech recognition model, the sequence of feature vectors to obtain an N-best list of speech recognition hypotheses for the training utterance;
for each speech recognition hypothesis in the N-best list of speech recognition hypotheses, identifying a respective number of word errors relative to the ground-truth label sequence corresponding to the training utterance;
approximating a loss function based on the respective number of word errors identified for each speech recognition hypothesis in the N-best list of speech recognition hypotheses; and
reducing variance by adjusting for an average number of word errors over the N-best list of speech recognition hypotheses for the training utterance.
2. The computer-implemented method of claim 1 , wherein processing the sequence of feature vectors to obtain the N-best list of speech recognition hypotheses comprises:
determining, based on the sequence of feature vectors received as input to the speech recognition model, a plurality of speech recognition hypotheses for the training utterance;
ranking the plurality of speech recognition hypotheses; and
identifying the N-best list of speech recognition hypotheses as the N highest-ranking speech recognition hypotheses in the plurality of speech recognition hypotheses.
3. The computer-implemented method of claim 2 , wherein determining the plurality of speech recognition hypotheses comprises using beam search to determine the plurality of speech recognition hypotheses.
4. The computer-implemented method of claim 3 , wherein the speech recognition model is trained using a same predetermined beam size and a same predetermined value of N.
5. The computer-implemented method of claim 1 , wherein N is an integer of a predetermined value.
6. The computer-implemented method of claim 1 , wherein the speech recognition model comprises a sequence-to-sequence speech recognition model comprising an encoder and a decoder, wherein the encoder and the decoder each comprise one or more recurrent neural network layers.
7. The computer-implemented method of claim 6 , wherein the one or more recurrent neural network layers comprise long short-term memory (LSTM) cells.
8. The computer-implemented method of claim 1 , wherein after training speech recognition model, the trained speech recognition model is configured to provide streaming speech recognition results that include substantially real-time transcriptions of a portion of an utterance while a speaker of the utterance continues to speak the utterance.
9. The computer-implemented method of claim 1 , wherein performing the operations for training the speech recognition model to minimize word error rate further comprises using the loss function to distribute probability weight over items in the N-best list of speech recognition hypotheses for the training utterance.
10. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations comprising:
receiving a sequence of feature vectors indicative of acoustic characteristics of a training utterance;
receiving a ground-truth label sequence corresponding to the training utterance; and
training a speech recognition model to minimize word error rate by performing operations comprising:
processing, using the speech recognition model, the sequence of feature vectors to obtain an N-best list of speech recognition hypotheses for the training utterance;
for each speech recognition hypothesis in the N-best list of speech recognition hypotheses, identifying a respective number of word errors relative to the ground-truth label sequence corresponding to the training utterance;
approximating a loss function based on the respective number of word errors identified for each speech recognition hypothesis in the N-best list of speech recognition hypotheses; and
reducing variance by adjusting for an average number of word errors over the N-best list of speech recognition hypotheses for the training utterance.
11. The system of claim 10 , wherein processing the sequence of feature vectors to obtain the N-best list of speech recognition hypotheses comprises:
determining, based on the sequence of feature vectors received as input to the speech recognition model, a plurality of speech recognition hypotheses for the training utterance;
ranking the plurality of speech recognition hypotheses; and
identifying the N-best list of speech recognition hypotheses as the N highest-ranking speech recognition hypotheses in the plurality of speech recognition hypotheses.
12. The system of claim 11 , wherein determining the plurality of speech recognition hypotheses comprises using beam search to determine the plurality of speech recognition hypotheses.
13. The system of claim 12 , wherein the speech recognition model is trained using a same predetermined beam size and a same predetermined value of N.
14. The system of claim 10 , wherein N is an integer of a predetermined value.
15. The system of claim 10 , wherein the speech recognition model comprises a sequence-to-sequence speech recognition model comprising an encoder and a decoder, wherein the encoder and the decoder each comprise one or more recurrent neural network layers.
16. The system of claim 15 , wherein the one or more recurrent neural network layers comprise long short-term memory (LSTM) cells.
17. The system of claim 10 , wherein after training speech recognition model, the trained speech recognition model is configured to provide streaming speech recognition results that include substantially real-time transcriptions of a portion of an utterance while a speaker of the utterance continues to speak the utterance.
18. The system of claim 10 , wherein performing the operations for training the speech recognition model to minimize word error rate further comprises using the loss function to distribute probability weight over items in the N-best list of speech recognition hypotheses for the training utterance.