Sub-models for neural contextual biasing with attention and embedding space
A method for training a sub-model for contextual biasing for speech recognition includes obtaining a base speech recognition model trained on non-biased data. The method includes obtaining a set of training utterances representative of a particular domain, each training utterance in the set of training utterances including audio data characterizing the training utterances and a ground truth transcription of the training utterance. The method further includes, for each corresponding training utterance in the set of training utterances, determining, using an embedding encoder, a corresponding document embedding from the ground truth transcription of the corresponding training utterance. The method includes training, using the corresponding document embeddings determined from the ground truth transcriptions of the set of training utterances, a sub-model to bias the base speech recognition model to recognize speech in the particular domain.
1. A computer-implemented method executed by data processing hardware that causes the data processing hardware to perform operations comprising:
obtaining a base speech recognition model of an automatic speech recognition (ASR) system, the base speech recognition model trained on non-biased data; and
training, using a training process, a sub-model of the ASR system for biasing the base speech recognition model to recognize speech in a particular domain by:
obtaining a set of training utterances representative of the particular domain, each training utterance in the set of training utterances comprising:
corresponding audio data characterizing the training utterance; and
a corresponding ground truth transcription of the training utterance;
for each training utterance in the set of training utterances;
determining, using an embedding encoder of the ASR model, a corresponding document embedding from the corresponding ground truth transcription of the training utterance;
processing, using the base speech recognition model configured to receive a sub-model output of the sub-model, the corresponding audio data characterizing the training utterance to generate a corresponding predicted speech recognition result, wherein the sub-model output of the sub-model is based on:
the corresponding document embedding determined from the corresponding ground truth transcription of the training utterance; and
a history of predicted speech recognition results generated by the base speech recognition model during one or more previous output steps; and
determining a corresponding supervised loss term based on the corresponding predicted speech recognition result and the corresponding ground truth transcription of the training utterance; and
updating parameters of the sub-model based on the corresponding supervised loss terms to teach the sub-model to learn how to bias the base speech recognition model to recognize speech in the particular domain.
2. The computer-implemented method of claim 1 , wherein parameters of the base speech recognition model are frozen while training the sub-model.
3. The computer-implemented method of claim 1 , wherein the operations further comprise, for at least one training utterance in the set of training utterances, converting, using a text-to-speech (TTS) system, the corresponding ground truth transcription of the at least one training utterance to generate the corresponding audio data comprising a corresponding synthetic speech representation of the at least one training utterance.
4. The computer-implemented method of claim 3 , wherein the corresponding ground truth transcription of the at least one training utterance is generated using a background language model and an in-domain language model trained on transcribed speech utterances associated with the particular domain.
5. The computer-implemented method of claim 1 , wherein the operations further comprise, for at least one training utterance in the set of training utterances, applying data augmentation to the corresponding audio data characterizing the at least one training utterance.
6. The computer-implemented method of claim 5 , wherein applying data augmentation comprises at least one of adding noise, adding reverberation, or manipulating timing.
7. The computer-implemented method of claim 1 , wherein the sub-model comprises one or more neural network layers.
8. The computer-implemented method of claim 1 , wherein the sub-model is disposed in a layer of the base speech recognition model.
9. The computer-implemented method of claim 1 , wherein:
the base speech recognition model comprises an encoder and a decoder; and
the sub-model is disposed in between two layers of the encoder of the base speech recognition model.
10. The computer-implemented method of claim 1 , wherein the operations further comprise, after training the sub-model, deploying the base speech recognition model and the trained sub-model for execution on a user device, the user device configured to:
receive a speech recognition request comprising audio data characterizing an utterance captured in streaming audio;
determine the speech recognition request comprises a contextual indicator indicating the particular domain;
bias, using the trained sub-model, the base speech recognition model toward the particular domain; and
generate, using the biased base speech recognition model, a transcription of the utterance by processing the audio data, the transcription biased toward one or more terms in the particular domain.
11. The computer-implemented method of claim 1 , wherein the operations further comprise, after training the sub-model:
receiving, from a user device in communication with the data processing hardware, a speech recognition request comprising audio data characterizing an utterance captured by the user device in streaming audio;
determining the speech recognition request comprises a contextual indicator indicating the particular domain;
biasing, using the trained sub-model, the base speech recognition model toward the particular domain; and
generating, using the biased base speech recognition model, a transcription of the utterance by processing the audio data, the transcription biased toward one or more terms in the particular domain.
12. The computer-implemented method of claim 1 , wherein the sub-model comprises the embedding encoder.
13. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations comprising:
obtaining a base speech recognition model of an automatic speech recognition (ASR) system, the base speech recognition model trained on non-biased data; and
training, using a training process, a sub-model of the ASR system for biasing the base speech recognition model to recognize speech in a particular domain by:
obtaining a set of training utterances representative of the particular domain, each training utterance in the set of training utterances comprising:
corresponding audio data characterizing the training utterance; and
a corresponding ground truth transcription of the training utterance;
for each training utterance in the set of training utterances:
determining, using an embedding encoder of the ASR model, a corresponding document embedding from the corresponding ground truth transcription of the training utterance;
processing, using the base speech recognition model configured to receive a sub-model output of the sub-model, the corresponding audio data characterizing the training utterance to generate a corresponding predicted speech recognition result, wherein the sub-model output of the sub-model is based on:
the corresponding document embedding determined from the corresponding ground truth transcription of the training utterance; and
a history of predicted speech recognition results generated by the base speech recognition model during one or more previous output steps; and
determining a corresponding supervised loss term based on the corresponding predicted speech recognition result and the corresponding ground truth transcription of the training utterance; and
updating parameters of the sub-model based on the corresponding supervised loss terms to teach the sub-model to learn how to bias the base speech recognition model to recognize speech in the particular domain.
14. The system of claim 13 , wherein parameters of the base speech recognition model are frozen while training the sub-model.
15. The system of claim 13 , wherein the operations further comprise, for at least one training utterance in the set of training utterances, converting, using a text-to-speech (TTS) system, the corresponding ground truth transcription of the at least one training utterance to generate the corresponding audio data comprising a corresponding synthetic speech representation of the at least one training utterance.
16. The system of claim 15 , wherein the corresponding ground truth transcription of the at least one training utterance is generated using a background language model and an in-domain language model trained on transcribed speech utterances associated with the particular domain.
17. The system of claim 13 , wherein the operations further comprise, for at least one training utterance in the set of training utterances, applying data augmentation to the corresponding audio data characterizing the at least one training utterance.
18. The system of claim 17 , wherein applying data augmentation comprises at least one of adding noise, adding reverberation, or manipulating timing.
19. The system of claim 13 , wherein the sub-model comprises one or more neural network layers.
20. The system of claim 13 , wherein the sub-model is disposed in a layer of the base speech recognition model.
21. The system of claim 13 , wherein:
the base speech recognition model comprises an encoder and a decoder; and
the sub-model is disposed in between two layers of the encoder of the base speech recognition model.
22. The system of claim 13 , wherein the operations further comprise, after training the sub-model, deploying the base speech recognition model and the trained sub-model for execution on a user device, the user device configured to:
receive a speech recognition request comprising audio data characterizing an utterance captured in streaming audio;
determine the speech recognition request comprises a contextual indicator indicating the particular domain;
bias, using the trained sub-model, the base speech recognition model toward the particular domain; and
generate, using the biased base speech recognition model, a transcription of the utterance by processing the audio data, the transcription biased toward one or more terms in the particular domain.
23. The system of claim 13 , wherein the operations further comprise, after training the sub-model:
receiving, from a user device in communication with the data processing hardware, a speech recognition request comprising audio data characterizing an utterance captured by the user device in streaming audio;
determining the speech recognition request comprises a contextual indicator indicating the particular domain;
biasing, using the trained sub-model, the base speech recognition model toward the particular domain; and
generating, using the biased base speech recognition model, a transcription of the utterance by processing the audio data, the transcription biased toward one or more terms in the particular domain.
24. The system of claim 13 , wherein the sub-model comprises the embedding encoder.