Freeze words
A method for detecting freeze words includes receiving audio data that corresponds to an utterance spoken by a user and captured by a user device associated with the user. The method also includes processing, using a speech recognizer, the audio data to determine that the utterance includes a query for a digital assistant to perform an operation. The speech recognizer is configured to trigger endpointing of the utterance after a predetermined duration of non-speech in the audio data. Before the predetermined duration of non-speech, the method includes detecting a freeze word in the audio data. In response to detecting the freeze word in the audio data, the method also includes triggering a hard microphone closing event at the user device. The hard microphone closing event prevents the user device from capturing any audio subsequent to the freeze word.
1. A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
receiving a first instance of audio data corresponding to a dictation-based query for a digital assistant to dictate audible contents spoken by a user, the dictation-based query spoken by the user;
receiving a second instance of the audio data corresponding to an utterance of the audible contents spoken by the user;
processing the second instance of the audio data to generate a transcription of the audible contents; and
during the processing of the second instance of the audio data:
detecting a freeze word in the second instance of the audio data; and
in response to detecting the freeze word in the second instance of the audio data, providing, for output from an assistant-enabled device associated with the user, the transcription of the audible contents spoken by the user.
2. The computer-implemented method of claim 1 , wherein:
the dictation-based query is spoken by the user and captured by the assistant-enabled device; and
the utterance of the audible contents spoken by the user is captured by the assistant-enabled device.
3. The computer-implemented method of claim 1 , wherein the freeze word follows the audible contents in the utterance spoken by the user.
4. The computer-implemented method of claim 1 , wherein the operations further comprise:
processing, using a speech recognizer, the first instance of the audio data to generate a speech recognition result; and
performing semantic interpretation on the speech recognition result for the first instance of the audio data to determine that the first instance of the audio data comprises the dictation-based query to dictate the audible contents spoken by the user.
5. The computer-implemented method of claim 4 , wherein the operations further comprise, prior to initiating processing on the second instance of the audio data to generate the transcription:
determining that the dictation-based query specifies the freeze word based on the semantic interpretation performed on the speech recognition result for the first instance of the audio data; and
instructing an endpointer to increase an endpointing timeout duration for endpointing the utterance of the audible contents.
6. The computer-implemented method of claim 1 , wherein the operations further comprise, in response to detecting the freeze word in the second instance of the audio data:
initiating a hard microphone closing event at the assistant-enabled device to prevent the assistant-enabled device form capturing any further audio for the utterance subsequent to the freeze word; and
ceasing any active processing on the second instance of the audio data.
7. The computer-implemented method of claim 1 , wherein detecting the freeze word in the second instance of the audio data comprises recognizing, using a speech recognizer executing on the data processing hardware, the freeze word in the second instance of the audio data.
8. The computer-implemented method of claim 1 , wherein the freeze word comprises one of:
a predefined freeze word comprising one or more fixed terms across all users in a given language;
a user-selected freeze word comprising one or more terms specified by the user; or
an action-specific freeze word associated with the operation to be performed by the digital assistant.
9. The computer-implemented method of claim 1 , wherein detecting the freeze word in the second instance of the audio data comprises:
extracting audio features from the second instance of the audio data;
generating, using a freeze word detection model, a freeze word confidence score by processing the extracted audio features, the freeze word detection model executing on the data processing hardware; and
determining that the second instance of the audio data corresponding to the utterance of the audible contents includes the freeze word when the freeze word confidence score satisfies a freeze word confidence threshold.
10. The computer-implemented method of claim 1 , wherein the data processing hardware resides on the assistant-enabled device.
11. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving a first instance of audio data corresponding to a dictation-based query for a digital assistant to dictate audible contents spoken by a user, the dictation-based query spoken by the user;
receiving a second instance of the audio data corresponding to an utterance of the audible contents spoken by the user;
processing the second instance of the audio data to generate a transcription of the audible contents; and
during the processing of the second instance of the audio data:
detecting a freeze word in the second instance of the audio data; and
in response to detecting the freeze word in the second instance of the audio data, providing, for output from an assistant-enabled device associated with the user, the transcription of the audible contents spoken by the user.
12. The system of claim 11 , wherein:
the dictation-based query is spoken by the user and captured by the assistant-enabled device; and
the utterance of the audible contents spoken by the user is captured by the assistant-enabled device.
13. The system of claim 11 , wherein the freeze word follows the audible contents in the utterance spoken by the user.
14. The system of claim 11 , wherein the operations further comprise:
processing, using a speech recognizer, the first instance of the audio data to generate a speech recognition result; and
performing semantic interpretation on the speech recognition result for the first instance of the audio data to determine that the first instance of the audio data comprises the dictation-based query to dictate the audible contents spoken by the user.
15. The system of claim 14 , wherein the operations further comprise, prior to initiating processing on the second instance of the audio data to generate the transcription:
determining that the dictation-based query specifies the freeze word based on the semantic interpretation performed on the speech recognition result for the first instance of the audio data; and
instructing an endpointer to increase an endpointing timeout duration for endpointing the utterance of the audible contents.
16. The system of claim 11 , wherein the operations further comprise, in response to detecting the freeze word in the second instance of the audio data:
initiating a hard microphone closing event at the assistant-enabled device to prevent the assistant-enabled device form capturing any further audio for the utterance subsequent to the freeze word; and
ceasing any active processing on the second instance of the audio data.
17. The system of claim 11 , wherein detecting the freeze word in the second instance of the audio data comprises recognizing, using a speech recognizer executing on the data processing hardware, the freeze word in the second instance of the audio data.
18. The system of claim 11 , wherein the freeze word comprises one of:
a predefined freeze word comprising one or more fixed terms across all users in a given language;
a user-selected freeze word comprising one or more terms specified by the user; or
an action-specific freeze word associated with the operation to be performed by the digital assistant.
19. The system of claim 11 , wherein detecting the freeze word in the second instance of the audio data comprises:
extracting audio features from the second instance of the audio data;
generating, using a freeze word detection model, a freeze word confidence score by processing the extracted audio features, the freeze word detection model executing on the data processing hardware; and
determining that the second instance of the audio data corresponding to the utterance of the audible contents includes the freeze word when the freeze word confidence score satisfies a freeze word confidence threshold.
20. The system of claim 11 , wherein the data processing hardware resides on the assistant-enabled device.