Transcription error resilient training of neural semantic parsers
A method includes obtaining a set of training queries that each specify a corresponding operation to perform and include a corresponding plurality of speech recognition hypotheses that each represent a corresponding candidate transcription of the training query, and a corresponding ground-truth transcription of the training query. For each training query, the method includes processing, using an encoder of a neural semantic parsing (NSP) model, the corresponding plurality of speech recognition hypotheses to generate a corresponding NSP embedding, processing, using a transcription decoder, the corresponding NSP embedding to generate a corresponding predicted transcription, and determining a corresponding first loss based on the corresponding predicted transcription and the corresponding ground-truth transcription. The method further includes training, based on the first losses determined for the set of training queries, the NSP model to learn how to predict user intents associated with the operations specified by the training queries.
1. A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
obtaining a set of training queries, each training query in the set of training queries specifying a corresponding operation to perform and comprising:
a corresponding plurality of speech recognition hypotheses, each speech recognition hypothesis representing a corresponding candidate transcription of the training query and generated by a speech recognizer from audio data characterizing the training query; and
a corresponding ground-truth transcription of the training query;
for each training query in the set of training queries:
processing, using an encoder of a neural semantic parsing (NSP) model, the corresponding plurality of speech recognition hypotheses to generate a corresponding NSP embedding;
processing, using a transcription decoder, the corresponding NSP embedding to generate a corresponding predicted transcription of the training query; and
determining a corresponding first loss based on the corresponding predicted transcription of the training query and the corresponding ground-truth transcription of the training query; and
training, based on the first losses determined for the set of training queries, the NSP model to learn how to predict user intents associated with the operations specified by the training queries.
2. The computer-implemented method of claim 1 , wherein the corresponding plurality of speech recognition hypotheses comprises:
a top speech recognition hypothesis associated with the speech recognition hypothesis having a highest respective speech recognition confidence score among the corresponding plurality of speech recognition hypotheses generated by the speech recognizer; and
one or more alternative speech recognition hypotheses each having a respective speech recognition confidence score that is lower than the respective speech recognition confidence score of the top speech recognition hypothesis.
3. The computer-implemented method of claim 1 , wherein:
each training query in the set of training queries further comprises a corresponding ground-truth user intent associated with the corresponding operation specified by the training query; and
the operations further comprise, for each training query in the set of training queries:
processing, using an NSP decoder of the NSP model, the corresponding NSP embedding to generate a corresponding predicted user intent for the training query; and
determining a corresponding second loss based on the corresponding predicted user intent and the corresponding ground-truth user intent associated with the corresponding operation specified by the training query; and
training the NSP model further comprises training the NSP model based on the second losses determined for the set of training queries.
4. The computer-implemented method of claim 3 , wherein training the NSP model comprises:
weighting the first losses with a first weight; and
weighting the second losses with a second weight different from the first weight.
5. The computer-implemented method of claim 3 , wherein training the NSP model comprises:
updating first coefficients of the encoder; and
updating second coefficients of the NSP decoder.
6. The computer-implemented method of claim 1 , wherein the operations further comprise training the transcription decoder based on the first losses.
7. The computer-implemented method of claim 1 , wherein the operations further comprise, for each training query in the set of training queries:
obtaining corresponding contextual information associated with the training query; and
generating the corresponding NSP embedding using the encoder by also processing, using the encoder, the corresponding contextual information associated with the training query.
8. The computer-implemented method of claim 7 , wherein the corresponding contextual information comprises at least one of:
user device information for the training query;
location information associated with the training query;
time information associated with the training query;
user information associated with the training query; or
a previous query submitted during a dialog prior to the training query.
9. The computer-implemented method of claim 1 , wherein the transcription decoder is not constrained to generate the corresponding predicted transcription for the training query as one of the corresponding plurality of speech recognition hypotheses.
10. The computer-implemented method of claim 1 , wherein the transcription decoder is not executed during an inference operation of the NSP model.
11. The computer-implemented method of claim 1 , wherein the operations further comprise, for at least one training query in the set of training queries, generating, using a teacher model, the corresponding ground-truth transcription from the audio data characterizing the training query.
12. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that, when executed on the data processing hardware, causes the data processing hardware to perform operations comprising:
obtaining a set of training queries, each training query in the set of training queries specifying a corresponding operation to perform and comprising:
a corresponding plurality of speech recognition hypotheses, each speech recognition hypothesis representing a corresponding candidate transcription of the training query and generated by a speech recognizer from audio data characterizing the training query; and
a corresponding ground-truth transcription of the training query;
for each training query in the set of training queries:
processing, using an encoder of a neural semantic parsing (NSP) model, the corresponding plurality of speech recognition hypotheses to generate a corresponding NSP embedding;
processing, using a transcription decoder, the corresponding NSP embedding to generate a corresponding predicted transcription of the training query; and
determining a corresponding first loss based on the corresponding predicted transcription of the training query and the corresponding ground-truth transcription of the training query; and
training, based on the first losses determined for the set of training queries, the NSP model to learn how to predict user intents associated with the operations specified by the training queries.
13. The system of claim 12 , wherein the corresponding plurality of speech recognition hypotheses comprises:
a top speech recognition hypothesis associated with the speech recognition hypothesis having a highest respective speech recognition confidence score among the corresponding plurality of speech recognition hypotheses generated by the speech recognizer; and
one or more alternative speech recognition hypotheses each having a respective speech recognition confidence score that is lower than the respective speech recognition confidence score of the top speech recognition hypothesis.
14. The system of claim 12 , wherein:
each training query in the set of training queries further comprises a corresponding ground-truth user intent associated with the corresponding operation specified by the training query; and
the operations further comprise, for each training query in the set of training queries:
processing, using an NSP decoder of the NSP model, the corresponding NSP embedding to generate a corresponding predicted user intent for the training query; and
determining a corresponding second loss based on the corresponding predicted user intent and the corresponding ground-truth user intent associated with the corresponding operation specified by the training query; and
training the NSP model further comprises training the NSP model based on the second losses determined for the set of training queries.
15. The system of claim 14 , wherein training the NSP model comprises:
weighting the first losses with a first weight; and
weighting the second losses with a second weight different from the first weight.
16. The system of claim 14 , wherein training the NSP model comprises:
updating first coefficients of the encoder; and
updating second coefficients of the NSP decoder.
17. The system of claim 12 , wherein the operations further comprise training the transcription decoder based on the first losses.
18. The system of claim 12 , wherein the operations further comprise, for each training query in the set of training queries:
obtaining corresponding contextual information associated with the training query; and
generating the corresponding NSP embedding using the encoder by also processing, using the encoder, the corresponding contextual information associated with the training query.
19. The system of claim 18 , wherein the corresponding contextual information comprises at least one of:
user device information for the training query;
location information associated with the training query;
time information associated with the training query;
user information associated with the training query; or
a previous query submitted during a dialog prior to the training query.
20. The system of claim 12 , wherein the transcription decoder is not constrained to generate the corresponding predicted transcription for the training query as one of the corresponding plurality of speech recognition hypotheses.
21. The system of claim 12 , wherein the transcription decoder is not executed during an inference operation of the NSP model.
22. The system of claim 12 , wherein the operations further comprise, for at least one training query in the set of training queries, generating, using a teacher model, the corresponding ground-truth transcription from the audio data characterizing the training query.