Sequence-to-sequence speech recognition with latency threshold

1. A computing system comprising:

one or more processors configured to:

receive an audio input;

generate a text transcription of the audio input at a sequence-to-sequence speech recognition model that includes a trained external alignment model, a trained encoder neural network, and a trained decoder neural network, the sequence-to-sequence speech recognition model being configured to at least:

assign, via the trained external alignment model, a respective plurality of external-model text tokens to a plurality of frames included in the audio input, wherein each external-model text token has an external-model alignment within the audio input;

based at least in part on the audio input, generate a plurality of hidden states via the trained encoder neural network; and

based at least in part on the plurality of hidden states and the respective external-model alignments of the external-model text tokens, generate a plurality of output text tokens corresponding to the plurality of frames via the trained decoder neural network, wherein:

each output text token has a corresponding output alignment within the audio input; and

for each output text token, a latency between the output alignment and the external-model alignment is below a predetermined latency threshold; and

output the text transcription including the plurality of output text tokens to an application program, a user interface, or a file storage location.

2. The computing system of claim 1 , wherein the trained encoder neural network and the trained decoder neural network are recurrent neural networks.

3. The computing system of claim 2 , wherein the trained decoder neural network is a monotonic chunkwise attention model.

4. The computing system of claim 3 , wherein, for each hidden state, the one or more processors are further configured to stochastically determine a binary attention state.

5. The computing system of claim 1 , wherein:

the audio input is a streaming audio input received by the one or more processors over an input time interval; and

the one or more processors are configured to output the text transcription during the input time interval concurrently with receiving the audio input.

6. The computing system of claim 1 , wherein the trained encoder neural network is trained at least in part with an encoder loss function including a sequence-to-sequence loss term and a framewise cross-entropy loss term.

7. The computing system of claim 6 , wherein the trained encoder neural network is pre-trained with the framewise cross-entropy loss term prior to training with the sequence-to-sequence loss term.

8. The computing system of claim 6 , wherein the trained encoder neural network is trained with the sequence-to-sequence loss term and the framewise cross-entropy loss term concurrently via multi-task learning.

9. The computing system of claim 8 , wherein the trained encoder neural network is trained at least in part at a first linear bottleneck layer and a second linear bottleneck layer.

10. The computing system of claim 1 , wherein the trained decoder neural network is trained at least in part with a delay constrained training loss function including a sequence-to-sequence loss term and an attention weight regularization term, wherein the attention weight regularization term is a function of a plurality of training output alignments.

11. The computing system of claim 1 , wherein the trained decoder neural network is trained at least in part with a minimum latency training loss function including a sequence-to-sequence loss term and a minimum latency loss term, wherein the minimum latency loss term is a function of a plurality of training output alignments and a corresponding plurality of training external-model alignments the latency between the output alignment and the external model alignment.

12. A method for use with a computing system, the method comprising:

receiving an audio input;

generating a text transcription of the audio input at a sequence-to-sequence speech recognition model that includes a trained external alignment model, a trained encoder neural network, and a trained decoder neural network, wherein the text transcription is generated at least by:

assigning, via the trained external alignment model, a respective plurality of external-model text tokens to a plurality of frames included in the audio input, wherein each external-model text token has an external-model alignment within the audio input;

based at least in part on the audio input, generating a plurality of hidden states via the trained encoder neural network; and

based at least in part on the plurality of hidden states and the respective external-model alignments of the external-model text tokens, generating a plurality of output text tokens corresponding to the plurality of frames via the trained decoder neural network, wherein:

each output text token has a corresponding output alignment within the audio input; and

for each output text token, a latency between the output alignment and the external-model alignment is below a predetermined latency threshold; and

outputting the text transcription including the plurality of output text tokens to an application program, a user interface, or a file storage location.

13. The method of claim 12 , wherein the trained encoder neural network and the trained decoder neural network are recurrent neural networks.

14. The method of claim 12 , wherein:

the audio input is a streaming audio input received over an input time interval; and

the text transcription is output during the input time interval concurrently with receiving the audio input.

15. The method of claim 12 , further comprising training the trained encoder neural network at least in part with an encoder loss function including a sequence-to-sequence loss term and a framewise cross-entropy loss term.

16. The method of claim 15 , further comprising pre-training the trained encoder neural network with the framewise cross-entropy loss term prior to training with the sequence-to-sequence loss term.

17. The method of claim 15 , further comprising training the trained encoder neural network with the sequence-to-sequence loss term and the framewise cross-entropy loss term concurrently via multi-task learning.

18. The method of claim 12 , further comprising training the trained decoder neural network at least in part with a delay constrained training loss function including a sequence-to-sequence loss term and an attention weight regularization term, wherein the decoder neural network is a monotonic chunkwise attention model and the attention weight regularization term is a function of a plurality of training output alignments.

19. The method of claim 12 , further comprising training the trained decoder neural network at least in part with a minimum latency training loss function including a sequence-to-sequence loss term and a minimum latency loss term, wherein:

the minimum latency loss term is a function of a plurality of training output alignments and a corresponding plurality of training external-model alignments; and

the decoder neural network is a monotonic chunkwise attention model.

20. A computing system comprising:

one or more processors configured to:

receive an audio input;

generate a text transcription of the audio input at a sequence-to-sequence speech recognition model that includes a trained external alignment model, a trained encoder neural network, and one or more trained decoder neural networks, the sequence-to-sequence speech recognition model being configured to at least:

at the trained external alignment model, assign a respective plurality of external-model text tokens to a plurality of frames included in the audio input, wherein each external-model text token has an external-model alignment within the audio input;

at the one or more trained decoder neural networks, generate a plurality of output text tokens corresponding to the plurality of frames based at least in part on the respective external-model alignments of the external-model text tokens, wherein:

the one or more trained decoder neural networks are one or more recurrent neural networks including at least a monotonic chunkwise attention model;

each output text token has a corresponding output alignment within the audio input; and

for each output text token, a latency between the output alignment and the external-model alignment is below a predetermined latency threshold; and

output the text transcription including the plurality of output text tokens to an application program, a user interface, or a file storage location, wherein the text transcription is generated in real time as a user is speaking.