Granted Patent B2

US 12,597,417 · App. 18/494,763 · Granted Apr 7, 2026

Inventors: Rami Magdi Fahmi Botros (Mountain View, CA); Rohit Prakash Prabhavalkar (Palo Alto, CA); Johan Schalkwyk (Scarsdale, NY); Tara N. Sainath (Jersey City, NJ); Ciprian Ioan Chelba (Mountain View, CA); Francoise Beaufays (Mountain View, CA)

Assignee: Google LLC

G10L15/16

Abstract

Claims (58)

1 . A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:

obtaining a base encoder from a pre-trained model, the pre-trained model comprising the base encoder and a decoder;

receiving training data comprising a sequence of acoustic frames characterizing an utterance paired with a ground-truth transcription of the utterance;

at each corresponding output step of a plurality of output steps:

generating, by the base encoder, a first encoded representation for a corresponding acoustic frame in the sequence of acoustic frames;

generating, by an exporter network configured to receive a continuous sequence of first encoded representations generated by the base encoder, a second encoded representation for a corresponding acoustic frame in the sequence of acoustic frames;

generating, by an exporter decoder, a probability distribution over possible logits; and

determining a corresponding exporter decoder loss based on the ground-truth transcription and the probability distribution over possible logits generated by the exporter decoder at the corresponding output step; and

training the exporter network based on the corresponding exporter decoder loss determined at each corresponding output step of the plurality of output steps while parameters of the base encoder are frozen.

2 . The method of claim 1 , wherein the operations further comprise obtaining the base encoder from a pre-trained streaming recognition model that comprises the base encoder, a prediction network, and a joint network.

3 . The method of claim 1 , wherein:

the base encoder comprises a first plurality of multi-head self-attention blocks; and

the exporter network comprises a second plurality of multi-head self-attention blocks.

4 . The method of claim 3 , wherein the second plurality of multi-head self-attention blocks of the exporter network are non-causal.

5 . The method of claim 3 , wherein the first plurality of multi-head self-attention blocks of the base encoder and the second plurality of multi-head self-attention blocks of the exporter network comprise conformer blocks.

6 . The method of claim 1 , wherein:

the exporter decoder comprises a connectionist temporal classification (CTC) decoder;

the corresponding exporter decoder loss comprises a CTC loss; and

the logits comprise sub-word units.

7 . The method of claim 6 , wherein the sub-word units comprise wordpieces, graphemes, phonemes, or triphones.

8 . The method of claim 1 , wherein the operations further comprise, at each corresponding output step of the plurality of output steps:

determining a modular encoded representation for a corresponding acoustic frame in the sequence of acoustic frames by extracting a top-k indices of logits from the probability distribution over possible logits generated by the exporter decoder at the corresponding output step;

re-embedding the modular encoded representation determined at the corresponding output step;

generating, by an importer network, an importer representation for a corresponding re-embedded modular encoded representation; and

generating, by a speech decoder, a speech recognition hypothesis for a corresponding importer representation.

9 . The method of claim 8 , wherein the speech decoder comprises a recurrent neural network-transducer (RNN-T) decoder.

10 . The method of claim 8 , wherein the speech decoder comprises a listen-attend-spell (LAS) decoder.

11 . The method of claim 10 , wherein the exporter decoder comprises a recurrent neural network-transducer (RNN-T) decoder.

12 . A system comprising:

data processing hardware; and

memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations comprising:

obtaining a base encoder from a pre-trained model, the pre-trained model comprising the base encoder and a decoder;

receiving training data comprising a sequence of acoustic frames characterizing an utterance paired with a ground-truth transcription of the utterance;

at each corresponding output step of a plurality of output steps:

generating, by the base encoder, a first encoded representation for a corresponding acoustic frame in the sequence of acoustic frames;

generating, by an exporter network configured to receive a continuous sequence of first encoded representations generated by the base encoder, a second encoded representation for a corresponding acoustic frame in the sequence of acoustic frames;

generating, by an exporter decoder, a probability distribution over possible logits; and

determining a corresponding exporter decoder loss based on the ground-truth transcription and the probability distribution over possible logits generated by the exporter decoder at the corresponding output step; and

training the exporter network based on the corresponding exporter decoder loss determined at each corresponding output step of the plurality of output steps while parameters of the base encoder are frozen.

13 . The system of claim 12 , wherein the operations further comprise obtaining the base encoder from a pre-trained streaming recognition model that comprises the base encoder, a prediction network, and a joint network.

14 . The system of claim 12 , wherein:

the base encoder comprises a first plurality of multi-head self-attention blocks; and

the exporter network comprises a second plurality of multi-head self-attention blocks.

15 . The system of claim 14 , wherein the second plurality of multi-head self-attention blocks of the exporter network are non-causal.

16 . The system of claim 14 , wherein the first plurality of multi-head self-attention blocks of the base encoder and the second plurality of multi-head self-attention blocks of the exporter network comprise conformer blocks.

17 . The system of claim 12 , wherein:

the exporter decoder comprises a connectionist temporal classification (CTC) decoder;

the corresponding exporter decoder loss comprises a CTC loss; and

the logits comprise sub-word units.

18 . The system of claim 17 , wherein the sub-word units comprise wordpieces, graphemes, phonemes, or triphones.

19 . The system of claim 12 , wherein the operations further comprise, at each corresponding output step of the plurality of output steps:

determining a modular encoded representation for a corresponding acoustic frame in the sequence of acoustic frames by extracting a top-k indices of logits from the probability distribution over possible logits generated by the exporter decoder at the corresponding output step;

re-embedding the modular encoded representation determined at the corresponding output step;

generating, by an importer network, an importer representation for a corresponding re-embedded modular encoded representation; and

generating, by a speech decoder, a speech recognition hypothesis for a corresponding importer representation.

20 . The system of claim 19 , wherein the speech decoder comprises a recurrent neural network-transducer (RNN-T) decoder.

21 . The system of claim 19 , wherein the speech decoder comprises a listen-attend-spell (LAS) decoder.

22 . The system of claim 21 , wherein the exporter decoder comprises a recurrent neural network-transducer (RNN-T) decoder.

Assignments (2)

Continuity (2)

References Cited (16)