Enhanced attention mechanisms

1. A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:

receiving a sequence of audio features characterizing an utterance;

processing, using an encoder neural network, the sequence of audio features to generate a sequence of encodings; and

at each of a plurality of output steps:

determining a corresponding hard monotonic attention output to select an encoding from the sequence of encodings;

identifying a proper subset of the sequence of encodings based on a position of the selected encoding in the sequence of encodings;

performing soft attention over the proper subset of the sequence of encodings to generate a context vector at the corresponding output step; and

processing, using a decoder neural network, the context vector generated at the corresponding output step to predict a probability distribution over possible output labels at the corresponding output step.

2. The computer-implemented method of claim 1 , wherein output labels comprise graphemes or phonemes.

3. The computer-implemented method of claim 1 , wherein the output labels comprise wordpieces.

4. The computer-implemented method of claim 1 , wherein the proper subset of the sequence of encodings identified at each output step comprises a same number of encodings.

5. The computer-implemented method of claim 1 , wherein the proper subset of the sequence of encodings identified at each output step comprises a corresponding window of encodings bounded by the selected encoding at the corresponding output step.

6. The computer-implemented method of claim 1 , wherein the corresponding window of encodings comprises a fixed size at each of the plurality of output steps.

7. The computer-implemented method of claim 1 , wherein the context vector generated at each corresponding output step represents a weighted summary of the encodings in the proper subset in the sequence of encodings identified at the corresponding output step.

8. The computer-implemented method of claim 1 , wherein processing the context vector generated at the corresponding output step to predict the probability distribution over possible output labels at the corresponding output step comprises processing the context vector generated at the corresponding output step and information determined from a prediction made using an output of the decoder neural network for an immediately previous output step.

9. The computer-implemented method of claim 1 , wherein the operations further comprise generating a transcription of the utterance based on the probability distribution of possible output labels predicted at each of the plurality of output steps.

10. The computer-implemented method of claim 1 , wherein:

the encoder neural network comprises at least one convolutional layer, at least one convolutional long-short-term memory (LSTM) layer, and at least one unidirectional LSTM layer; and

the decoder neural network comprises a unidirectional LSTM layer.

11. A system comprising:

data processing hardware; and’

memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations comprising:

receiving a sequence of audio features characterizing an utterance;

processing, using an encoder neural network, the sequence of audio features to generate a sequence of encodings; and

at each of a plurality of output steps:

determining a corresponding hard monotonic attention output to select an encoding from the sequence of encodings;

identifying a proper subset of the sequence of encodings based on a position of the selected encoding in the sequence of encodings;

performing soft attention over the proper subset of the sequence of encodings to generate a context vector at the corresponding output step; and

processing, using a decoder neural network, the context vector generated at the corresponding output step to predict a probability distribution over possible output labels at the corresponding output step.

12. The system of claim 11 , wherein output labels comprise graphemes or phonemes.

13. The system of claim 11 , wherein the output labels comprise wordpieces.

14. The system of claim 11 , wherein the proper subset of the sequence of encodings identified at each output step comprises a same number of encodings.

15. The system of claim 11 , wherein the proper subset of the sequence of encodings identified at each output step comprises a corresponding window of encodings bounded by the selected encoding at the corresponding output step.

16. The system of claim 11 , wherein the corresponding window of encodings comprises a fixed size at each of the plurality of output steps.

17. The system of claim 11 , wherein the context vector generated at each corresponding output step represents a weighted summary of the encodings in the proper subset in the sequence of encodings identified at the corresponding output step.

18. The system of claim 11 , wherein processing the context vector generated at the corresponding output step to predict the probability distribution over possible output labels at the corresponding output step comprises processing the context vector generated at the corresponding output step and information determined from a prediction made using an output of the decoder neural network for an immediately previous output step.

19. The system of claim 11 , wherein the operations further comprise generating a transcription of the utterance based on the probability distribution of possible output labels predicted at each of the plurality of output steps.

20. The system of claim 11 , wherein:

the encoder neural network comprises at least one convolutional layer, at least one convolutional long-short-term memory (LSTM) layer, and at least one unidirectional LSTM layer; and

the decoder neural network comprises a unidirectional LSTM layer.