Efficient streaming non-recurrent on-device end-to-end model

1. Memory hardware storing instructions that, when executed by data processing hardware, cause the data processing hardware to implement an automated speech recognition (ASR) model, the ASR model comprising:

a first encoder configured to:

receive, as input, a sequence of acoustic frames corresponding to an utterance; and

generate, at each of a plurality of output steps, a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames;

a second encoder configured to:

receive, as input, the first higher order feature representation generated by the first encoder at each of the plurality of output steps; and

generate, at each of the plurality of output steps, a second higher order feature representation for a corresponding first higher order feature frame;

a decoder configured to:

receive, as input, the second higher order feature representation generated by the second encoder at each of the plurality of output steps; and

generate, at each of the plurality of output steps, a first probability distribution over possible speech recognition hypotheses; and

a language model configured to:

receive, as input, the first probability distribution over possible speech hypotheses; and

generate, at each of the plurality of output steps, a rescored probability distribution over possible speech recognition hypotheses to generate a transcription for the utterance.

2. The memory hardware of claim 1 , wherein the second encoder generates the second higher order feature representation without receiving any of the acoustic frames as input.

3. The memory hardware of claim 1 , wherein the decoder is further configured to:

receive, as input, the first higher order feature representation generated by the first encoder at each of the plurality of output steps; and

generate, at each of the plurality of output steps, a second probability distribution over possible speech recognition hypotheses.

4. The memory hardware of claim 3 , wherein the decoder comprises:

a prediction network configured to, at each of the plurality of output steps:

receive, as input, a sequence of N previous non-blank symbols output by a final Softmax layer;

for each non-blank symbol of the sequence of N previous non-blank symbols, generate a respective embedding; and

generate an average embedding by averaging the respective embeddings; and

a joint network configured to:

receive, as input, the average embedding generated by the prediction network at each of the plurality of output steps and one of:

when the ASR model is operating in a streaming mode, the first higher order feature representation generated by the first encoder at each of the plurality of output steps; or

when the ASR model is operating in a non-streaming mode, the second higher order feature representation generated by the second encoder at each of the plurality of output steps; and

generate, at each of the plurality of output steps, one of:

when the ASR model is operating in the streaming mode, the second probability distribution over possible speech recognition hypotheses; or

when the ASR model is operating in the non-streaming mode, the first probability distribution over possible speech recognition hypotheses.

5. The memory hardware of claim 4 , wherein the prediction network comprises a V2 embedding look-up table.

6. The memory hardware of claim 1 , wherein the first encoder comprises a causal encoder comprising an initial stack of conformer layers.

7. The memory hardware of claim 6 , wherein the second encoder comprises a non-causal encoder comprising a final stack of conformer layers overlain on the initial stack of conformer layers.

8. The memory hardware of claim 1 , wherein the language model comprises a neural language model.

9. The memory hardware of claim 8 , wherein the neural language model comprises a stack of conformer layers or transformer layers.

10. The memory hardware of claim 1 , wherein:

the language model is trained on text-only data; and

the first encoder and the second encoder are trained using Hybrid Autoregressive Transducer Factorization to facilitate integration of the language model trained on the text-only data.

11. A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:

receiving, as input to an automatic speech recognition (ASR) model, a sequence of acoustic frames corresponding to an utterance;

performing, using the ASR model, streaming speech recognition and non-streaming speech recognition on the sequence of acoustic frames by:

generating, by a first encoder, at each of a plurality of output steps, a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames;

receiving, as input to a second encoder, the first higher order feature representation generated by the first encoder at each of the plurality of output steps;

generating, by the second encoder, at each of the plurality of output steps, a second higher order feature representation for a corresponding first higher order feature frame;

receiving, as input to a decoder, the second higher order feature representation generated by the second encoder at each of the plurality of output steps; and

generating, at each of the plurality of output steps, a first probability distribution over possible speech recognition hypotheses; and

rescoring, using an external language model, the first probability distribution over possible speech recognition hypotheses to generate a transcription of the utterance.

12. The computer-implemented method of claim 11 , wherein the second encoder generates the second higher order feature representation without receiving any of the acoustic frames as input.

13. The computer-implemented method of claim 11 , wherein the operations further comprise, when performing streaming speech recognition and non-streaming speech recognition on the sequence of acoustic frames:

receiving, as input to the decoder, the first high order feature representation generated by the first encoder at each of the plurality of output steps; and

generating, at each of the plurality of output steps, a second probability distribution over possible speech recognition hypotheses.

14. The computer-implemented method of claim 13 , wherein the operations further comprise, at each of the plurality of output steps:

receiving, as input to a prediction network, a sequence of N previous non-blank symbols output by a final Softmax later;

for each non-blank symbol of the sequence of N previous non-blank symbols, generating, by the prediction network, a respective embedding;

generating, by the prediction network, an average embedding by averaging the respective embeddings;

receiving, as input to a joint network, the average embedding generated by the prediction network at each of the plurality of output steps and one of:

when the ASR model is operating in a streaming mode, the first higher order feature representation generated by the first encoder at each of the plurality of output steps; or

when the ASR model is operating in a non-streaming mode, the second higher order feature representation generated by the second encoder at each of the plurality of output steps; and

generating, at each of the plurality of output steps, one of:

when the ASR model is operating in the streaming mode, the second probability distribution over possible speech recognition hypotheses; or

when the ASR model is operating in the non-streaming mode, the first probability distribution over possible speech recognition hypotheses.

15. The computer-implemented method of claim 14 , wherein the prediction network comprises a V2 embedding look-up table.

16. The computer-implemented method of claim 11 , wherein the first encoder comprises a causal encoder comprising an initial stack of conformer layers.

17. The computer-implemented method of claim 16 , wherein the second encoder comprises a non-causal encoder comprising a final stack of conformer layers overlain on the initial stack of conformer layers.

18. The computer-implemented method of claim 11 , wherein the language model comprises a neural language model.

19. The computer-implemented method of claim 18 , wherein the neural language model comprises a stack of conformer layers or transformer layers.

20. The computer-implemented method of claim 11 , wherein:

the language model is trained on text-only data; and

the first encoder and the second encoder are trained using Hybrid Autoregressive Transducer Factorization to facilitate integration of the language model trained on the text-only data.