Lattice Speech Corrections
A method includes receiving audio data corresponding to a query spoken and processing the audio data to generate multiple candidate hypotheses each represented by a respective sequence of hypothesized terms. For each candidate hypothesis, the method includes determining whether the sequence of hypothesized terms includes a source phrase from a list of phrase correction pairs. Each phrase correction pair includes a corresponding source phrase that was misrecognized and a corresponding target phrase replacing the source phrase. When the respective sequence of hypothesized terms includes the source phrase, the method includes generating a corresponding additional candidate hypothesis that replaces the source phrase. The method also includes ranking the multiple candidate hypotheses and each corresponding additional candidate hypothesis generated and generating a transcription of the query spoken by the user by selecting the highest ranking one of the multiple candidate hypotheses and each additional candidate hypothesis.
1 . A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:
receiving audio data corresponding to a query spoken by a user;
processing, using a speech recognizer, the audio data to generate multiple candidate hypotheses, each candidate hypothesis corresponding to a candidate transcription for the query and represented by a respective sequence of hypothesized terms;
for a corresponding candidate hypothesis among the multiple candidate hypotheses represented by the respective sequence of hypothesized terms that includes a source phrase from a list of phrase correction pairs each comprising a corresponding source phrase and a corresponding target phrase replacing the source phrase:
generating a corresponding additional candidate hypothesis that replaces the source phrase included in the respective sequence of hypothesized terms with the corresponding target phrase;
obtaining a first sequence of n-grams that includes the source phrase included in the respective sequence of hypothesized terms and one or more other terms that precede and/or are subsequent to the source phrase;
obtaining a second sequence of n-grams that replaces the source phrase in the first sequence of n-grams with the corresponding target phrase, the second corrected sequence of n-grams including the corresponding target phrase and the same one or more other terms that precede and/or are subsequent to the source phrase;
conditioning a language model on the first sequence of n-grams and the second sequence of n-grams to determine a higher prior likelihood score for a number of n-grams from the second sequence of n-grams that includes the corresponding target phrase than for a same number of n-grams from the first sequence of n-grams that includes the source phrase; and
for the corresponding additional candidate hypothesis, determining, using the language model conditioned on the first sequence of n-grams and the second sequence of n-grams, a corresponding prior likelihood score;
ranking the multiple candidate hypotheses and the corresponding additional candidate hypothesis based on the corresponding prior likelihood score for the corresponding additional candidate hypothesis; and
generating a transcription of the query spoken by the user by selecting the highest ranking one of the multiple candidate hypotheses and the corresponding additional candidate hypothesis.
2 . The computer-implemented method of claim 1 , wherein the speech recognizer includes an acoustic model and the language model.
3 . The computer-implemented method of claim 1 , wherein the speech recognizer comprises an end-to-end speech recognition model.
4 . The computer-implemented method of claim 1 , wherein a margin between the prior likelihood score determined using the language model for the number of n-grams from the second sequence of n-grams and the same number of n-grams from the first sequence of n-grams increases as the number of n-grams from the first and second sequences of n-grams increases.
5 . The computer-implemented method of claim 1 , wherein conditioning the language model further comprises conditioning the language model on the first sequence of n-grams and the second sequence of n-grams to determine a lower prior likelihood score for a first number of n-grams from the second sequence of n-grams that includes the corresponding target phrase than for a greater second number of n-grams from the first sequence of n-grams that includes the source phrase.
6 . The computer-implemented method of claim 1 , wherein the first and second sequences of n-grams each further include n-grams representing sentence boundaries of a corresponding previous transcription transcribed by the speech recognizer for a previous utterance spoken by the user.
7 . The computer-implemented method of claim 1 , wherein the operations further comprise:
for each of the multiple candidate hypotheses generated by the speech recognizer, obtaining a corresponding likelihood score that the speech recognizer assigned to the corresponding candidate hypothesis; and
after generating the corresponding additional candidate hypothesis, determining, using an additional hypothesis scorer, a corresponding likelihood score for the corresponding additional candidate hypothesis generated,
wherein ranking the multiple candidate hypotheses and the corresponding additional candidate hypothesis is based on the corresponding likelihood scores assigned to the multiple candidate hypothesis by the speech recognizer and the corresponding likelihood score determined for the corresponding additional candidate hypothesis using the additional hypothesis scorer.
8 . The computer-implemented method of claim 7 , wherein:
the additional hypothesis scorer comprises at least one of:
an acoustic model configured to process audio data to determine an acoustic modeling score for a portion of the audio data that includes either the source phrase or the corresponding target phrase; or
a language model configured to receive the corresponding additional candidate hypothesis as input and determine a corresponding prior likelihood score for each additional candidate hypothesis; and
the corresponding likelihood score determined for each additional candidate hypothesis is based on at least one of the acoustic modeling score or the corresponding prior likelihood score determined for the additional candidate hypothesis.
9 . The computer-implemented method of claim 8 , wherein the language model comprises an auxiliary language model external to the speech recognizer or an internal language model integrated with the speech recognizer.
10 . The computer-implemented method of claim 7 , wherein:
the speech recognizer comprises an acoustic model and a language model; and
the corresponding likelihood score that the speech recognizer assigned to each of the multiple candidate hypotheses is based on at least one of an acoustic modeling score output by the acoustic model or a corresponding prior likelihood score output by the language model.
11 . A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations comprising:
receiving audio data corresponding to a query spoken by a user;
processing, using a speech recognizer, the audio data to generate multiple candidate hypotheses, each candidate hypothesis corresponding to a candidate transcription for the query and represented by a respective sequence of hypothesized terms;
for a corresponding candidate hypothesis among the multiple candidate hypotheses represented by the respective sequence of hypothesized terms that includes a source phrase from a list of phrase correction pairs each comprising a corresponding source phrase and a corresponding target phrase replacing the source phrase:
generating a corresponding additional candidate hypothesis that replaces the source phrase included in the respective sequence of hypothesized terms with the corresponding target phrase;
obtaining a first sequence of n-grams that includes the source phrase included in the respective sequence of hypothesized terms and one or more other terms that precede and/or are subsequent to the source phrase;
obtaining a second sequence of n-grams that replaces the source phrase in the first sequence of n-grams with the corresponding target phrase, the second corrected sequence of n-grams including the corresponding target phrase and the same one or more other terms that precede and/or are subsequent to the source phrase;
conditioning a language model on the first sequence of n-grams and the second sequence of n-grams to determine a higher prior likelihood score for a number of n-grams from the second sequence of n-grams that includes the corresponding target phrase than for a same number of n-grams from the first sequence of n-grams that includes the source phrase; and
for the corresponding additional candidate hypothesis, determining, using the language model conditioned on the first sequence of n-grams and the second sequence of n-grams, a corresponding prior likelihood score;
ranking the multiple candidate hypotheses and the corresponding additional candidate hypothesis based on the corresponding prior likelihood score for the corresponding additional candidate hypothesis; and
generating a transcription of the query spoken by the user by selecting the highest ranking one of the multiple candidate hypotheses and the corresponding additional candidate hypothesis.
12 . The system of claim 11 , wherein the speech recognizer includes an acoustic model and the language model.
13 . The system of claim 11 , wherein the speech recognizer comprises an end-to-end speech recognition model.
14 . The system of claim 11 , wherein a margin between the prior likelihood score determined using the language model for the number of n-grams from the second sequence of n-grams and the same number of n-grams from the first sequence of n-grams increases as the number of n-grams from the first and second sequences of n-grams increases.
15 . The system of claim 11 , wherein conditioning the language model further comprises conditioning the language model on the first sequence of n-grams and the second sequence of n-grams to determine a lower prior likelihood score for a first number of n-grams from the second sequence of n-grams that includes the corresponding target phrase than for a greater second number of n-grams from the first sequence of n-grams that includes the source phrase.
16 . The system of claim 11 , wherein the first and second sequences of n-grams each further include n-grams representing sentence boundaries of a corresponding previous transcription transcribed by the speech recognizer for a previous utterance spoken by the user.
17 . The system of claim 11 , wherein the operations further comprise:
for each of the multiple candidate hypotheses generated by the speech recognizer, obtaining a corresponding likelihood score that the speech recognizer assigned to the corresponding candidate hypothesis; and
after generating the corresponding additional candidate hypothesis, determining, using an additional hypothesis scorer, a corresponding likelihood score for the corresponding additional candidate hypothesis generated,
wherein ranking the multiple candidate hypotheses and the corresponding additional candidate hypothesis is based on the corresponding likelihood scores assigned to the multiple candidate hypothesis by the speech recognizer and the corresponding likelihood score determined for the corresponding additional candidate hypothesis using the additional hypothesis scorer.
18 . The system of claim 17 , wherein:
the additional hypothesis scorer comprises at least one of:
an acoustic model configured to process audio data to determine an acoustic modeling score for a portion of the audio data that includes either the source phrase or the corresponding target phrase; or
a language model configured to receive the corresponding additional candidate hypothesis as input and determine a corresponding prior likelihood score for each additional candidate hypothesis; and
the corresponding likelihood score determined for each additional candidate hypothesis is based on at least one of the acoustic modeling score or the corresponding prior likelihood score determined for the additional candidate hypothesis.
19 . The system of claim 18 , wherein the language model comprises an auxiliary language model external to the speech recognizer or an internal language model integrated with the speech recognizer.
20 . The system of claim 17 , wherein:
the speech recognizer comprises an acoustic model and a language model; and
the corresponding likelihood score that the speech recognizer assigned to each of the multiple candidate hypotheses is based on at least one of an acoustic modeling score output by the acoustic model or a corresponding prior likelihood score output by the language model.