Sub-models for neural contextual biasing
A method for contextual biasing for speech recognition includes obtaining a base automatic speech recognition (ASR) model trained on non-biased data and a sub-model trained on biased data representative of a particular domain. The method includes receiving a speech recognition request including audio data characterizing an utterance captured in streaming audio. The method further includes determining whether the speech recognition request includes a contextual indicator indicating the particular domain. When the speech recognition request does not include the contextual indicator, the method includes generating, using the base ASR model, a first speech recognition result of the utterance by processing the audio data. When the speech recognition request includes the contextual indicator the method includes biasing, using the sub-model, the base ASR model toward the particular domain and generating, using the biased base ASR model, a second speech recognition result of the utterance by processing the audio data.
1. A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising:
obtaining a base automatic speech recognition (ASR) model trained on non-biased data;
obtaining a sub-model trained on biased data, the biased data representative of a particular domain;
receiving a speech recognition request comprising audio data characterizing an utterance captured in streaming audio;
determining whether the speech recognition request includes a contextual indicator indicating the particular domain;
when the speech recognition request does not include the contextual indicator, generating, using the base ASR model, a first speech recognition result of the utterance by processing the audio data; and
when the speech recognition request includes the contextual indicator:
generating, using the base ASR model, an encoded output by processing the audio data;
biasing, using the sub-model, the base ASR model toward the particular domain;
generating, using the biased base ASR model, a sub-model output by processing the audio data, the sub-model output generated in parallel with the encoded output; and
generating, using a decoder of the base ASR model, a second speech recognition result of the utterance by processing the encoded output and the sub-model output, the second speech recognition result biased toward one or more terms in the particular domain.
2. The computer-implemented method of claim 1 , wherein the contextual indicator comprises a one-hot vector.
3. The computer-implemented method of claim 2 , wherein the one-hot vector indicates a particular sub-model from a plurality of sub-models to be activated, each sub-model of the plurality of sub-models associated with a different domain.
4. The computer-implemented method of claim 2 , wherein the operations further comprise projecting the one-hot vector into a phrase set embedding of an embedding space.
5. The computer-implemented method of claim 4 , wherein projecting the one-hot vector into the phrase set embedding causes the phrase set embedding to activate a portion of the sub-model.
6. The computer-implemented method of claim 1 , wherein the sub-model is disposed in a layer of the base ASR model.
7. The computer-implemented method of claim 6 , wherein:
the base ASR model comprises an encoder and the decoder; and
the sub-model is disposed in between two layers of the encoder.
8. The computer-implemented method of claim 1 , wherein one or more parameters of the base ASR model are frozen when generating the first and second speech recognition results.
9. The computer-implemented method of claim 1 , wherein:
the data processing hardware resides on a user device that captured the utterance in the streaming audio; or
the data processing hardware resides on a remote system in communication with the user device via a network.
10. The computer-implemented method of claim 1 , wherein the first speech recognition result is different from the second speech recognition result.
11. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
obtaining a base automatic speech recognition (ASR) model trained on non-biased data;
obtaining a sub-model trained on biased data, the biased data representative of a particular domain;
receiving a speech recognition request comprising audio data characterizing an utterance captured in streaming audio;
determining whether the speech recognition request includes a contextual indicator indicating the particular domain;
when the speech recognition request does not include the contextual indicator, generating, using the base ASR model, a first speech recognition result of the utterance by processing the audio data; and
when the speech recognition request includes the contextual indicator:
generating, using the base ASR model, an encoded output by processing the audio data;
biasing, using the sub-model, the base ASR model toward the particular domain;
generating, using the biased base ASR model, a sub-model output by processing the audio data, the sub-model output generated in parallel with the encoded output; and
generating, using a decoder of the base ASR model, a second speech recognition result of the utterance by processing the audio data encoded output and the sub-model output, the second speech recognition result biased toward one or more terms in the particular domain.
12. The system of claim 11 , wherein the contextual indicator comprises a one-hot vector.
13. The system of claim 12 , wherein the one-hot vector indicates a particular sub-model from a plurality of sub-models to be activated, each sub-model of the plurality of sub-models associated with a different domain.
14. The system of claim 12 , wherein the operations further comprise projecting the one-hot vector into a phrase set embedding of an embedding space.
15. The system of claim 14 , wherein projecting the one-hot vector into the phrase set embedding causes the phrase set embedding to activate a portion of the sub-model.
16. The system of claim 11 , wherein the sub-model is disposed in a layer of the base ASR model.
17. The system of claim 16 , wherein:
the base ASR model comprises an encoder and the decoder; and
the sub-model is disposed in between two layers of the encoder.
18. The system of claim 11 , wherein one or more parameters of the base ASR model are frozen when generating the first and second speech recognition results.
19. The system of claim 11 , wherein:
the data processing hardware resides on a user device that captured the utterance in the streaming audio; or
the data processing hardware resides on a remote system in communication with the user device via a network.
20. The system of claim 11 , wherein the first speech recognition result is different from the second speech recognition result.