Efficient streaming non-recurrent on-device end-to-end model
An ASR model includes a first encoder configured to receive a sequence of acoustic frames and generate a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The ASR model also includes a second encoder configured to receive the first higher order feature representation generated by the first encoder at each of the plurality of output steps and generate a second higher order feature representation for a corresponding first higher order feature frame. The ASR model also includes a decoder configured to receive the second higher order feature representation generated by the second encoder at each of the plurality of output steps and generate a first probability distribution over possible speech recognition hypothesis. The ASR model also includes a language model configured to receive the first probability distribution over possible speech hypothesis and generate a rescored probability distribution.
1. A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
receiving, as input to a cascaded encoder, a sequence of acoustic frames corresponding to an utterance as a user speaks the utterance;
at each corresponding output step of a plurality of output steps commencing a predefined duration after an initial acoustic frame in the sequence of acoustic frames is received:
generating, by the cascaded encoder, a corresponding higher order feature representation for a corresponding acoustic frame; and
generating, by a decoder configured to receive the corresponding higher order feature representation generated by the cascaded encoder at the corresponding output step, a probability distribution over possible output labels; and
rescoring, by an external language model, the probability distribution over possible output labels generated by the decoder at each of the plurality of output steps to generate a transcription of the utterance.
2. The computer-implemented method of claim 1 , wherein the cascaded encoder comprises a causal encoder and a non-causal encoder overlain on the causal encoder.
3. The computer-implemented method of claim 2 , wherein:
the causal encoder comprises an initial stack of multi-headed attention layers; and
the non-causal encoder comprises a final stack of multi-headed attention layers overlain on the initial stack of multi-headed attention layers.
4. The computer-implemented method of claim 3 , wherein the initial stack of multi-headed attention layers comprises a greater number of multi-headed attention layers than the final stack of multi-headed attention layers.
5. The computer-implemented method of claim 3 , wherein the multi-headed attention layers comprise Conformer layers.
6. The computer-implemented method of claim 3 , wherein the multi-headed attention layers comprise Transformer layers.
7. The computer-implemented method of claim 1 , wherein the possible output labels comprise wordpieces.
8. The computer-implemented method of claim 1 , wherein the external language model comprises a neural language model.
9. The computer-implemented method of claim 8 , wherein the neural language model comprises a stack of Conformer layers or Transformer layers.
10. The computer-implemented method of claim 1 , wherein:
the language model is trained on text-only data; and
the first encoder and the second encoder are trained using Hybrid Autoregressive Transducer Factorization to facilitate integration of the external language model trained on text-only data.
11. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations comprising:
receiving, as input to a cascaded encoder, a sequence of acoustic frames corresponding to an utterance as a user speaks the utterance;
at each corresponding output step of a plurality of output steps commencing a predefined duration after an initial acoustic frame in the sequence of acoustic frames is received:
generating, by the cascaded encoder, a corresponding higher order feature representation for a corresponding acoustic frame; and
generating, by a decoder configured to receive the corresponding higher order feature representation generated by the cascaded encoder at the corresponding output step, a probability distribution over possible output labels; and
rescoring, by an external language model, the probability distribution over possible output labels generated by the decoder at each of the plurality of output steps to generate a transcription of the utterance.
12. The system of claim 11 , wherein the cascaded encoder comprises a causal encoder and a non-causal encoder overlain on the causal encoder.
13. The system of claim 12 , wherein:
the causal encoder comprises an initial stack of multi-headed attention layers; and
the non-causal encoder comprises a final stack of multi-headed attention layers overlain on the initial stack of multi-headed attention layers.
14. The system of claim 13 , wherein the initial stack of multi-headed attention layers comprises a greater number of multi-headed attention layers than the final stack of multi-headed attention layers.
15. The system of claim 13 , wherein the multi-headed attention layers comprise Conformer layers.
16. The system of claim 13 , wherein the multi-headed attention layers comprise Transformer layers.
17. The system of claim 11 , wherein the possible output labels comprise wordpieces.
18. The system of claim 11 , wherein the external language model comprises a neural language model.
19. The system of claim 18 , wherein the neural language model comprises a stack of Conformer layers or Transformer layers.
20. The system of claim 11 , wherein:
the language model is trained on text-only data; and
the first encoder and the second encoder are trained using Hybrid Autoregressive Transducer Factorization to facilitate integration of the external language model trained on text-only data.