Minimum word error rate training for attention-based sequence-to-sequence models

1. A method performed by one or more computers of a speech recognition system, the method comprising:

receiving, by the one or more computers, audio data indicating acoustic characteristics of an utterance;

generating, by the one or more computers, a sequence of feature vectors indicative of the acoustic characteristics of the utterance;

processing, by the one or more computers, the sequence of feature vectors using a speech recognition model that has been trained using a loss function that uses N-best lists of decoded hypotheses, the speech recognition model comprising an encoder, an attention module providing multi-headed attention in which multiple different sets of weighting parameters are used to process different segments of output from the encoder, and a decoder, wherein the encoder and decoder each comprise one or more recurrent neural network layers and the attention module comprises a plurality of neural networks that are separately trained to generate output to the decoder from the different segments of output from the encoder;

obtaining, by the one or more computers as a result of the processing with the speech recognition model, a sequence of output vectors representing distributions over a predetermined set of linguistic units;

determining, by the one or more computers, a transcription for the utterance based on the sequence of output vectors; and

providing, by the one or more computers, data indicating the transcription of the utterance.

2. The method of claim 1 , wherein the speech recognition model has been trained such that the loss function distributes probability weight over items in the N-best lists.

3. The method of claim 1 , wherein the speech recognition model has been trained to directly minimize expected word error rate.

4. The method of claim 1 , wherein the speech recognition model is configured to output a probability distribution over a predetermined set of grapheme symbols.

5. The method of claim 1 , wherein the attention module comprises at least four attention heads.

6. The method of claim 1 , wherein the one or more recurrent neural network layers comprise long short-term memory (LSTM) cells.

7. The method of claim 1 , wherein the encoder and decoder each include a plurality of unidirectional LSTM layers.

8. The method of claim 1 , wherein the encoder comprises a plurality of bidirectional LSTM layers.

9. The method of claim 1 , wherein the speech recognition model has been trained by performing, for each training example of multiple training examples, operations including:

determining a plurality of speech recognition hypotheses using the speech recognition model being trained;

ranking the plurality of speech recognition hypotheses;

identifying N highest-ranking speech recognition hypotheses in the plurality of speech recognition hypotheses, where N is an integer of a predetermined value;

distributing probability mass concentrated entirely on the N highest-ranking speech recognition hypotheses; and

approximating a loss function for training according to the distributed probability mass.

10. The method of claim 9 , wherein distributing the probability mass is performed based on a respective number of word errors identified for the N highest-ranking speech recognition hypotheses.

11. The method of claim 9 , wherein determining the plurality of speech recognition hypotheses comprises using beam search to determine the plurality of speech recognition hypotheses;

wherein, for training for each of the multiple training examples, the speech recognition model is trained using a same predetermined beam size and a same predetermined value of N; and

wherein distributing the probability mass comprises normalizing, for each training example, the entire probability mass over the N highest-ranking speech recognition hypotheses for the training example.

12. The method of claim 9 , wherein the operations include reducing variance by adjusting for an average number of word errors over the N highest-ranking speech recognition hypotheses for a training example.

13. The method of claim 1 , wherein the speech recognition model is configured to provide streaming speech recognition results that include substantially real-time transcriptions of a portion of an utterance while a speaker of the utterance continues to speak the utterance.

14. A system comprising:

one or more computers; and

one or more computer-readable media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

receiving, by the one or more computers, audio data indicating acoustic characteristics of an utterance;

generating, by the one or more computers, a sequence of feature vectors indicative of the acoustic characteristics of the utterance;

processing, by the one or more computers, the sequence of feature vectors using a speech recognition model that has been trained using a loss function that uses N-best lists of decoded hypotheses, the speech recognition model comprising an encoder, an attention module providing multi-headed attention in which multiple different sets of weighting parameters are used to process different segments of output from the encoder, and a decoder, wherein the encoder and decoder each comprise one or more recurrent neural network layers and the attention module comprises a plurality of neural networks that are separately trained to generate output to the decoder from the different segments of output from the encoder;

obtaining, by the one or more computers as a result of the processing with the speech recognition model, a sequence of output vectors representing distributions over a predetermined set of linguistic units;

determining, by the one or more computers, a transcription for the utterance based on the sequence of output vectors; and

providing, by the one or more computers, data indicating the transcription of the utterance.

15. The system of claim 14 , wherein the speech recognition model is an end-to-end neural network model.

16. The system of claim 14 , wherein the speech recognition model has been trained using N-best lists determined using beam search.

17. One or more non-transitory computer-readable media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

receiving, by the one or more computers, audio data indicating acoustic characteristics of an utterance;

generating, by the one or more computers, a sequence of feature vectors indicative of the acoustic characteristics of the utterance;

processing, by the one or more computers, the sequence of feature vectors using a speech recognition model that has been trained using a loss function that uses N-best lists of decoded hypotheses, the speech recognition model comprising an encoder, an attention module providing multi-headed attention in which multiple different sets of weighting parameters are used to process different segments of output from the encoder, and a decoder, wherein the encoder and decoder each comprise one or more recurrent neural network layers and the attention module comprises a plurality of neural networks that are separately trained to generate output to the decoder from the different segments of output from the encoder;

obtaining, by the one or more computers as a result of the processing with the speech recognition model, a sequence of output vectors representing distributions over a predetermined set of linguistic units;

determining, by the one or more computers, a transcription for the utterance based on the sequence of output vectors; and

providing, by the one or more computers, data indicating the transcription of the utterance.