Transducer-Based Streaming Deliberation for Cascaded Encoders

1 . A transducer-based deliberation model comprising:

an encoder configured to:

receive, as input, a sequence of acoustic frames; and

generate, at each of a plurality of output steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames;

a first pass transducer decoder configured to:

receive, as input, the higher order feature representation generated by the encoder at each of the plurality of output steps; and

generate, at each of the plurality of output steps, a first pass speech recognition hypothesis for a corresponding higher order feature representation;

a text encoder configured to:

receive, as input, the first pass speech recognition hypothesis generated at each of the plurality of output steps; and

generate, at each of the plurality of output steps, a text encoding for a corresponding first pass speech recognition hypothesis; and

a second pass transducer decoder configured to:

receive, as input, the text encoding generated by the text encoder at each of the plurality of output steps; and

generate, at each of the plurality of output steps, a second pass speech recognition hypothesis.

2 . The transducer-based deliberation model of claim 1 , further comprising a prediction network shared by the first pass transducer decoder and the second pass transducer decoder, the prediction network configured to:

receive, as input, a sequence of non-blank symbols output by a final softmax layer; and

generate, at each of the plurality of output steps, a dense representation.

3 . The transducer-based deliberation model of claim 2 , wherein the second pass transducer decoder further comprises a joint network configured to:

receive, as input, the dense representation generated by the prediction network at each of the plurality of output steps, the text encoding generated by the text encoder at each of the plurality of output steps; and

generate, at each of the plurality of output steps, the second pass speech recognition hypothesis.

4 . The transducer-based deliberation model of claim 1 , wherein the encoder comprises a stack of multi-headed attention layers.

5 . The transducer-based deliberation model of claim 5 , wherein the stack of multi-headed attention layers comprises a stack of Conformer layers.

6 . The transducer-based deliberation model of claim 5 , wherein the stack of multi-head attention layers comprises a stack of Transformer layers.

7 . The transducer-based deliberation model of claim 1 , wherein the encoder comprises a causal encoder.

8 . The transducer-based deliberation model of claim 1 , wherein the second pass transducer decoder trains without using any text-only data.

9 . The transducer-based deliberation model of claim 1 , wherein receiving the text encoding generated by the text encoder at each of the plurality of output steps comprises receiving a partial sequence of the text encoding in a streaming fashion.

10 . The transducer-based deliberation model of claim 1 , wherein the first and second pass speech recognition hypotheses each correspond to a partial speech recognition result.

11 . A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:

receiving a sequence of acoustic frames;

generating, by an encoder, at each of a plurality of output steps, a higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames;

generating, by a first pass transducer decoder, at each of the plurality of output steps, a first pass speech recognition hypothesis for a corresponding higher order feature representation;

generating, by a text encoder, at each of the plurality of output steps, a text encoding for a corresponding first pass speech recognition hypothesis; and

generating, by a second pass transducer decoder, at each of the plurality of output steps, a second pass speech recognition hypothesis using a corresponding text encoding.

12 . The computer-implemented method of claim 11 , wherein the operations further comprise:

generating, by a prediction network based on a sequence of non-blank symbols output by a final softmax layer, at each of the plurality of output steps, a dense representation,

wherein the first pass transducer decoder and the second pass transducer decoder share the prediction network.

13 . The computer-implemented method of claim 12 , wherein the operations further comprise, at each of the plurality of output steps, generating, by a joint network, the second pass speech recognition hypothesis based on the dense representation generated by the prediction network at each of the plurality of output steps and the text encoding generated by the text encoder at each of the plurality of output steps.

14 . The computer-implemented method of claim 11 , wherein the encoder comprises a stack of multi-headed attention layers.

15 . The computer-implemented method of claim 14 , wherein the stack of multi-headed attention layers comprises a stack of Conformer layers.

16 . The computer-implemented method of claim 14 , wherein the stack of multi-headed attention layers comprises a stack of Transformer layers.

17 . The computer-implemented method of claim 11 , wherein the encoder comprises a causal encoder.

18 . The computer-implemented method of claim 11 , wherein the second pass transducer decoder trains without using any text-only data.

19 . The computer-implemented method of claim 11 , wherein receiving the text encoding comprises receiving a partial sequence of the text encoding in a streaming fashion.

20 . The computer-implemented method of claim 11 , wherein the first and second pass speech recognition hypotheses each correspond to a partial speech recognition result.