Mask-conformer augmenting conformer with mask-predict decoder unifying speech recognition and rescoring
A method includes receiving a sequence of acoustic frames characterizing an utterance. During a first pass, the method includes generating first-pass audio encodings based on the sequence of acoustic frames using a stack of mask-conformer blocks of an acoustic encoder, generating a first-pass transcription of the utterance based on the first-pass audio encodings using a speech recognition decoder, and generating a first-pass masked output sequence using a mask-predict decoder of the acoustic encoder. During a second pass, the method includes generating second-pass audio encodings by performing cross-attention on the sequence of acoustic frames and the masked first-pass transcription using the stack of mask-conformer blocks of the acoustic encoder and generating a second-pass transcription of the utterance based on the second-pass audio encodings using the speech recognition decoder.
1 . A computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations comprising:
receiving a sequence of acoustic frames characterizing an utterance;
during a first pass:
generating, as output from each mask-conformer block in using a stack of mask-conformer blocks of an acoustic encoder, first-pass audio encodings based on the sequence of acoustic frames;
generating, using a speech recognition decoder, a first-pass transcription of the utterance based on the first-pass audio encodings generated as output from each mask-conformer block, the first-pass transcription comprising a sequence of speech recognition tokens; and
generating, using a mask-predict decoder of the acoustic encoder, a first-pass masked output sequence by scoring each speech recognition token from the sequence of speech recognition tokens of the first-pass transcription generated based on the first-pass audio encodings output from an immediately preceding mask-conformer block in the stack of mask-conformer blocks, and masking one or more of the speech recognition tokens from the sequence of speech recognition tokens with a masked token; and
during a second pass:
generating, using the stack of mask-conformer blocks of the acoustic encoder, second-pass audio encodings by performing cross-attention on the sequence of acoustic frames and the first-pass masked output sequence, wherein each mask-conformer block in the stack of mask-conformer blocks comprises a cross-attention layer that is disabled during the first pass and enabled during the second pass; and
generating, using the speech recognition decoder, a second-pass transcription of the utterance based on the second-pass audio encodings.
2 . The computer-implemented method of claim 1 , wherein the operations further comprise:
generating, using one or more convolution subsampling layers, a convolution output based on the sequence of acoustic frames,
wherein generating the first-pass audio encodings is further based on the convolution output.
3 . The computer-implemented method of claim 1 , wherein the speech recognition decoder comprises one of a Connectionist Temporal Classification (CTC) decoder, a listen-attend-spell decoder, a Recurrent Neural Network-Transducer (RNN-T) decoder, a hybrid autoregressive transducer (HAT) decoder, a Globally Normalized Autoregressive Transducer (GNAT) decoder, or a mask-predict decoder.
4 . The computer-implemented method of claim 1 , wherein the speech recognition decoder is connected to an output of each mask-conformer block in the stack of mask-conformer blocks.
5 . The computer-implemented method of claim 1 , wherein each mask-conformer block in the stack of mask-conformer blocks comprises at least one of:
a stack of self-attention layers;
a cross-attention layer;
a convolution layer; or
a mask-predict decoder.
6 . The computer-implemented method of claim 1 , wherein each mask-conformer block in the stack of mask-conformer blocks comprises a respective mask-predict decoder not shared with any of the other mask-conformer blocks in the stack of mask-conformer blocks.
7 . The computer-implemented method of claim 1 , wherein each mask-conformer block in the stack of mask-conformer blocks comprises a same mask-predict decoder shared with each other mask-conformer block in the stack of mask-conformer blocks.
8 . The computer-implemented method of claim 1 , wherein generating the first-pass masked output sequence comprises, for each respective speech recognition token from the sequence of speech recognition tokens, determining a confidence value indicating a likelihood that the respective speech recognition token is correct.
9 . The computer-implemented method of claim 8 , wherein the operations further comprise:
determining that the confidence value for a respective one of the speech recognition tokens satisfies a confidence value threshold; and
based on determining that the confidence value for the respective one of the speech recognition tokens satisfies the confidence value threshold, replacing the respective one of the speech recognition tokens with the masked token.
10 . A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving a sequence of acoustic frames characterizing an utterance;
during a first pass:
generating, as output from each mask-conformer block in using a stack of mask-conformer blocks of an acoustic encoder, first-pass audio encodings based on the sequence of acoustic frames;
generating, using a speech recognition decoder, a first-pass transcription of the utterance based on the first-pass audio encodings generated as output from each mask-conformer block, the first-pass transcription comprising a sequence of speech recognition tokens; and
generating, using a mask-predict decoder of the acoustic encoder, a first-pass masked output sequence by scoring each speech recognition token from the sequence of speech recognition tokens of the first-pass transcription generated based on the first-pass audio encodings output from an immediately preceding mask-conformer block in the stack of mask-conformer blocks, and masking one or more of the speech recognition tokens from the sequence of speech recognition tokens with a masked token; and
during a second pass:
generating, using the stack of mask-conformer blocks of the acoustic encoder, second-pass audio encodings by performing cross-attention on the sequence of acoustic frames and the first-pass masked output sequence, wherein each mask-conformer block in the stack of mask-conformer blocks comprises a cross-attention layer that is disabled during the first pass and enabled during the second pass; and
generating, using the speech recognition decoder, a second-pass transcription of the utterance based on the second-pass audio encodings.
11 . The system of claim 10 , wherein the operations further comprise:
generating, using one or more convolution subsampling layers, a convolution output based on the sequence of acoustic frames,
wherein generating the first-pass audio encodings is further based on the convolution output.
12 . The system of claim 10 , wherein the speech recognition decoder comprises one of a Connectionist Temporal Classification (CTC) decoder, a listen-attend-spell decoder, a Recurrent Neural Network-Transducer (RNN-T) decoder, a hybrid autoregressive transducer (HAT) decoder, a Globally Normalized Autoregressive Transducer (GNAT) decoder, or a mask-predict decoder.
13 . The system of claim 10 , wherein the speech recognition decoder is connected to an output of each mask-conformer block in the stack of mask-conformer blocks.
14 . The system of claim 10 , wherein each mask-conformer block in the stack of mask-conformer blocks comprises at least one of:
a stack of self-attention layers;
a cross-attention layer;
a convolution layer; or
a mask-predict decoder.
15 . The system of claim 10 , wherein each mask-conformer block in the stack of mask-conformer blocks comprises a respective mask-predict decoder not shared with any of the other mask-conformer blocks in the stack of mask-conformer blocks.
16 . The system of claim 10 , wherein each mask-conformer block in the stack of mask-conformer blocks comprises a same mask-predict decoder shared with each other mask-conformer block in the stack of mask-conformer blocks.
17 . The system of claim 10 , wherein generating the first-pass masked output sequence comprises, for each respective speech recognition token from the sequence of speech recognition tokens, determining a confidence value indicating a likelihood that the respective speech recognition token is correct.
18 . The system of claim 17 , wherein the operations further comprise:
determining that the confidence value for a respective one of the speech recognition tokens satisfies a confidence value threshold; and
based on determining that the confidence value for the respective one of the speech recognition tokens satisfies the confidence value threshold, replacing the respective one of the speech recognition tokens with the masked token.