Attention-based joint acoustic and text on-device end-to-end model
A method includes receiving a training example for a listen-attend-spell (LAS) decoder of a two-pass streaming neural network model and determining whether the training example corresponds to a supervised audio-text pair or an unpaired text sequence. When the training example corresponds to an unpaired text sequence, the method also includes determining a cross entropy loss based on a log probability associated with a context vector of the training example. The method also includes updating the LAS decoder and the context vector based on the determined cross entropy loss.
1 . A computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations comprising:
receiving a sequence of acoustic frames characterizing an utterance;
encoding, using a shared encoder, each acoustic frame in the sequence of acoustic frames to generate a corresponding encoded acoustic frame;
generating, using a first-pass decoder, based on the encoded acoustic frames, streamed hypotheses;
at each output step of a plurality of output steps for a second-pass decoder:
determining, using a same attention mechanism, both:
an acoustic context vector that summarizes the encoded acoustic frames; and
a linguistic context vector based on a sequence of decoded labels previously output by the second-pass decoder;
determining, using the acoustic context vector, as output from the second-pass decoder, an acoustic-based probability distribution over possible output labels;
determining, using the linguistic context vector, as output from the second-pass decoder, a text-based probability distribution over possible output labels; and
interpolating the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels, wherein the possible output labels in the acoustic-based probability distribution and the possible output labels in the text-based probability distribution comprise a same set subword units; and
determining a transcription of the utterance based on the interpolating of the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels at each of the plurality of output steps.
2 . The method of claim 1 , wherein determining the linguistic context vector based on the sequence of decoded labels previously output by the second-pass decoder ignores the encoded acoustic frames.
3 . The method of claim 1 , wherein determining the acoustic-based probability distribution over possible output labels using the acoustic context vector is further based on the sequence of decoded labels previously output by the second-pass decoder.
4 . The method of claim 1 , wherein the second-pass decoder operates in a beam search mode based on the streamed hypotheses generated by the first-pass decoder during a first pass.
5 . The method of claim 1 , wherein the operations further comprise:
processing, using the first-pass decoder, the encoded acoustic frames to generate a top-K list of speech recognition hypotheses for the utterance, each speech recognition hypotheses in the top-K list of speech recognition hypotheses corresponding to a candidate transcription of the utterance; and
the second-pass decoder operations in a rescoring mode to rescore each speech recognition hypotheses in the top-K list of speech recognition hypotheses.
6 . The method of claim 1 , wherein the output labels in the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels comprise wordpieces.
7 . The method of claim 1 , wherein the second-pass decoder comprises a listen-attend-spell (LAS) decoder.
8 . The method of claim 1 , wherein interpolating the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels comprises using a mixing weight to interpolate the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels so that the acoustic-based probability distribution over possible output labels are weighted differently than the text-based probability distribution over possible output labels.
9 . The method of claim 1 , wherein:
wherein the utterance characterized by the sequence of acoustic frames is captured in streaming audio by a user device; and
the data processing hardware resides on the user device.
10 . The method of claim 9 , wherein the operations further comprise performing natural language processing on the transcription to identify an action for a digital assistant application to perform.
11 . A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving a sequence of acoustic frames characterizing an utterance;
encoding, using a shared encoder, each acoustic frame in the sequence of acoustic frames to generate a corresponding encoded acoustic frame;
generating, using a first-pass decoder, based on the encoded acoustic frames, streamed hypotheses;
at each output step of a plurality of output steps for a second-pass decoder:
determining, using a same attention mechanism, both:
an acoustic context vector that summarizes the encoded acoustic frames; and
a linguistic context vector based on a sequence of decoded labels previously output by the second-pass decoder;
determining, using the acoustic context vector, as output from the second-pass decoder, an acoustic-based probability distribution over possible output labels;
determining, using the linguistic context vector, as output from the second-pass decoder, a text-based probability distribution over possible output labels; and
interpolating the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels, wherein the possible output labels in the acoustic-based probability distribution and the possible output labels in the text-based probability distribution comprise a same set subword units; and
determining a transcription of the utterance based on the interpolating of the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels at each of the plurality of output steps.
12 . The system of claim 11 , wherein determining the linguistic context vector based on the sequence of decoded labels previously output by the second-pass decoder ignores the encoded acoustic frames.
13 . The system of claim 11 , wherein determining the acoustic-based probability distribution over possible output labels using the acoustic context vector is further based on the sequence of decoded labels previously output by the second-pass decoder.
14 . The system of claim 11 , wherein the second-pass decoder operates in a beam search mode based on the streamed hypotheses generated by the first-pass decoder during a first pass.
15 . The system of claim 11 , wherein the operations further comprise:
processing, using the first-pass decoder, the encoded acoustic frames to generate a top-K list of speech recognition hypotheses for the utterance, each speech recognition hypotheses in the top-K list of speech recognition hypotheses corresponding to a candidate transcription of the utterance; and
the second-pass decoder operations in a rescoring mode to rescore each speech recognition hypotheses in the top-K list of speech recognition hypotheses.
16 . The system of claim 11 , wherein the output labels in the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels comprise wordpieces.
17 . The system of claim 11 , wherein the second-pass decoder comprises a listen-attend-spell (LAS) decoder.
18 . The system of claim 11 , wherein interpolating the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels comprises using a mixing weight to interpolate the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels so that the acoustic-based probability distribution over possible output labels are weighted differently than the text-based probability distribution over possible output labels.
19 . The system of claim 11 , wherein:
wherein the utterance characterized by the sequence of acoustic frames is captured in streaming audio by a user device; and
the data processing hardware resides on the user device.
20 . The system of claim 19 , wherein the operations further comprise performing natural language processing on the transcription to identify an action for a digital assistant application to perform.