Training speech recognizers based on biased transcriptions
A method includes receiving a biased transcription for a voice command spoken by a user and captured by a user device, the biased transcription biased to include a biasing phrase from a set of biasing phrases specific to the user. The method also includes instructing an application executing on the user device to perform an action specified by the biased transcription for the voice command, and receiving one or more user behavior signals responsive to the application performing the action specified by the biased transcription. The method further includes generating, as output from a confidence model, a confidence score of the biased transcription based on the one or more user behavior signals input to the confidence model and, based on the confidence score output from the confidence model, training a speech recognizer on the biased transcription.
1 . A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
receiving a biased transcription for a voice command spoken by a user and captured by a user device, the biased transcription biased to include a biasing phrase from a set of biasing phrases specific to the user;
instructing an application executing on the user device to perform an action specified by the biased transcription for the voice command;
receiving, from a monitoring module executing on the user device, one or more user behavior signals captured by the monitoring module responsive to the application performing the action specified by the biased transcription, the one or more behavior signals representing interactions between the user and the application associated with the application performing the action specified by the biased transcription;
processing, by a confidence model, the received one or more user behavior signals captured by the monitoring module to generate a confidence score of the biased transcription, the confidence score indicating a likelihood that the biased transcription accurately transcribes the voice command;
determining the confidence score satisfies a confidence threshold; and
based on determining the confidence score satisfies the confidence threshold, training a speech recognizer on the biased transcription.
2 . The computer-implemented method of claim 1 , wherein the operations further comprise:
receiving audio data characterizing the voice command captured by the user device;
processing, using the speech recognizer, the audio data to generate one or more candidate hypotheses for the voice command, each candidate hypothesis corresponding to a candidate transcription for the voice command and represented by a respective sequence of hypothesized terms;
identifying, from the one or more candidate hypotheses for the voice command, a particular carrier phrase; and
based on the particular carrier phrase, biasing, using a biasing model, the one or more candidate hypotheses toward the set of biasing phrases specific to the user to determine the biased transcription that includes the biasing phrase.
3 . The computer-implemented method of claim 2 , wherein training the speech recognizer on the biased transcription comprises:
generating a personalized training data pair that comprises the biased transcription paired with the audio data characterizing the voice command; and
training the speech recognizer on the personalized training data pair.
4 . The computer-implemented method of claim 2 , wherein:
the particular carrier phrase comprises a phrase intended to invoke a voice calling application or a messaging application;
the application instructed to perform the action specified by the biased transcription of the voice command comprises the voice calling application or the messaging application; and
the set of biasing phrases specific to the user comprises personal contacts of the user.
5 . The computer-implemented method of claim 2 , wherein:
the particular carrier phrase comprises a phrase intended to invoke a media player application;
the application instructed to perform the action specified by the biased transcription of the voice command comprises the media player application; and
the set of biasing phrases specific to the user comprises a list of names present in a content playlist for the media player application.
6 . The computer-implemented method of claim 2 , wherein:
the particular carrier phrase comprises a phrase intended to invoke a navigation application;
the application instructed to perform the action specified by the biased transcription of the voice command comprises the navigation application; and
the set of biasing phrases specific to the user comprises at least one of street names, city names, or relevant points of interest.
7 . The computer-implemented method of claim 2 , wherein the operations further comprise selecting, based on the particular carrier phrase, the set of biasing phrases from a plurality of sets of biasing phrases.
8 . The computer-implemented method of claim 2 , wherein the biasing model comprises an external language model.
9 . The computer-implemented method of claim 8 , wherein the language model comprises a neural finite state transducer.
10 . The computer-implemented method of claim 1 , wherein the speech recognizer comprises an end-to-end speech recognition model.
11 . The computer-implemented method of claim 1 , wherein the speech recognizer comprises a language model.
12 . The computer-implemented method of claim 1 , wherein the confidence model comprises a heuristically-based model or a trained machine learning model.
13 . A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving a biased transcription for a voice command spoken by a user and captured by a user device, the biased transcription biased to include a biasing phrase from a set of biasing phrases specific to the user;
instructing an application executing on the user device to perform an action specified by the biased transcription for the voice command;
receiving, from a monitoring module executing on the user device, one or more user behavior signals captured by the monitoring module responsive to the application performing the action specified by the biased transcription, the one or more behavior signals representing interactions between the user and the application associated with the application performing the action specified by the biased transcription;
processing, by a confidence model, the received one or more user behavior signals captured by the monitoring module to generate a confidence score of the biased transcription, the confidence score indicating a likelihood that the biased transcription accurately transcribes the voice command;
determining the confidence score satisfies a confidence threshold; and
based on determining the confidence score satisfies the confidence threshold, training a speech recognizer on the biased transcription.
14 . The system of claim 13 , wherein the operations further comprise:
receiving audio data characterizing the voice command captured by the user device;
processing, using the speech recognizer, the audio data to generate one or more candidate hypotheses for the voice command, each candidate hypothesis corresponding to a candidate transcription for the voice command and represented by a respective sequence of hypothesized terms;
identifying, from the one or more candidate hypotheses for the voice command, a particular carrier phrase; and
based on the particular carrier phrase, biasing, using a biasing model, the one or more candidate hypotheses toward the set of biasing phrases specific to the user to determine the biased transcription that includes the biasing phrase.
15 . The system of claim 14 , wherein training the speech recognizer on the biased transcription comprises:
generating a personalized training data pair that comprises the biased transcription paired with the audio data characterizing the voice command; and
training the speech recognizer on the personalized training data pair.
16 . The system of claim 14 , wherein:
the particular carrier phrase comprises a phrase intended to invoke a voice calling application or a messaging application;
the application instructed to perform the action specified by the biased transcription of the voice command comprises the voice calling application or the messaging application; and
the set of biasing phrases specific to the user comprises personal contacts of the user.
17 . The system of claim 14 , wherein:
the particular carrier phrase comprises a phrase intended to invoke a media player application;
the application instructed to perform the action specified by the biased transcription of the voice command comprises the media player application; and
the set of biasing phrases specific to the user comprises a list of names present in a content playlist for the media player application.
18 . The system of claim 14 , wherein:
the particular carrier phrase comprises a phrase intended to invoke a navigation application;
the application instructed to perform the action specified by the biased transcription of the voice command comprises the navigation application; and
the set of biasing phrases specific to the user comprises at least one of street names, city names, or relevant points of interest.
19 . The system of claim 14 , wherein the operations further comprise selecting, based on the particular carrier phrase, the set of biasing phrases from a plurality of sets of biasing phrases.
20 . The system of claim 14 , wherein the biasing model comprises an external language model.
21 . The system of claim 20 , wherein the language model comprises a neural finite state transducer.
22 . The system of claim 13 , wherein the speech recognizer comprises an end-to-end speech recognition model.
23 . The system of claim 13 , wherein the speech recognizer comprises a language model.
24 . The system of claim 13 , wherein the confidence model comprises a heuristically-based model or a trained machine learning model.