Efficient streaming non-recurrent on-device end-to-end model
An ASR model includes a first encoder configured to receive a sequence of acoustic frames and generate a first higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The ASR model also includes a second encoder configured to receive the first higher order feature representation generated by the first encoder at each of the plurality of output steps and generate a second higher order feature representation for a corresponding first higher order feature frame. The ASR model also includes a decoder configured to receive the second higher order feature representation generated by the second encoder at each of the plurality of output steps and generate a first probability distribution over possible speech recognition hypothesis. The ASR model also includes a language model configured to receive the first probability distribution over possible speech hypothesis and generate a rescored probability distribution.
1 . A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
receiving a sequence of acoustic frames corresponding to an utterance as a user speaks the utterance;
at a corresponding output step of a plurality of output steps each associated with a corresponding acoustic frame in the sequence of acoustic frames:
processing, using a stack of multi-headed attention layers, the corresponding acoustic frame to generate a corresponding higher order feature representation; and
generating, by a decoder configured to receive the corresponding higher order feature representation generated at the corresponding output step, a probability distribution over possible output labels; and
rescoring, by an external language model, the probability distribution over possible output labels generated by the decoder at each of the plurality of output steps to generate a transcription of the utterance.
2 . The computer-implemented method of claim 1 , wherein processing the corresponding acoustic frame commences a predefined duration after an initial acoustic frame in the sequence of acoustic frames is received.
3 . The computer-implemented method of claim 1 , wherein the stack of multi-headed attention layers comprises a stack of Transformer layers.
4 . The computer-implemented method of claim 1 , wherein the stack of multi-headed attention layers comprises a stack of Conformer layers.
5 . The computer-implemented method of claim 1 , wherein the possible output labels comprise wordpieces.
6 . The computer-implemented method of claim 1 , wherein the possible output labels comprise graphemes.
7 . The computer-implemented method of claim 1 , wherein the external language model comprises a neural language model.
8 . The computer-implemented method of claim 7 , wherein the neural language model comprises a plurality of multi-headed attention layers.
9 . The computer-implemented method of claim 1 , wherein the external language model is trained on text-only data.
10 . The computer-implemented method of claim 1 , wherein the utterance comprises a long-form utterance that comprises a plurality of sentences.
11 . A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations comprising:
receiving a sequence of acoustic frames corresponding to an utterance as a user speaks the utterance;
at a corresponding output step of a plurality of output steps each associated with a corresponding acoustic frame in the sequence of acoustic frames:
processing, using a stack of multi-headed attention layers, the corresponding acoustic frame to generate a corresponding higher order feature representation; and
generating, by a decoder configured to receive the corresponding higher order feature representation generated at the corresponding output step, a probability distribution over possible output labels; and
rescoring, by an external language model, the probability distribution over possible output labels generated by the decoder at each of the plurality of output steps to generate a transcription of the utterance.
12 . The system of claim 11 , wherein processing the corresponding acoustic frame commences a predefined duration after an initial acoustic frame in the sequence of acoustic frames is received.
13 . The system of claim 11 , wherein the stack of multi-headed attention layers comprises a stack of Transformer layers.
14 . The system of claim 11 , wherein the stack of multi-headed attention layers comprises a stack of Conformer layers.
15 . The system of claim 11 , wherein the possible output labels comprise wordpieces.
16 . The system of claim 11 , wherein the possible output labels comprise graphemes.
17 . The system of claim 11 , wherein the external language model comprises a neural language model.
18 . The system of claim 17 , wherein the neural language model comprises a plurality of multi-headed attention layers.
19 . The system of claim 11 , wherein the external language model is trained on text-only data.
20 . The system of claim 11 , wherein the utterance comprises a long-form utterance that comprises a plurality of sentences.