Speech recognition using word or phoneme time markers based on user input
A method for separating target speech from background noise contained in an input audio signal includes receiving the input audio signal captured by a user device, wherein the input audio signal corresponds to target speech of multiple words spoken by a target user and containing background noise in the presence of the user device while the target user spoke the multiple words in the target speech. The method also includes receiving a sequence of time markers input by the target user in cadence with the target user speaking the multiple words in the target speech, and correlating the sequence of time markers with the input audio signal to generate enhanced audio features that separate the target speech from the background noise in the input audio signal. The method also includes processing, using a speech recognition model, the enhanced audio features to generate a transcription of the target speech.
1 . A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations comprising:
receiving an input audio signal captured by a user device, the input audio signal corresponding to target speech of multiple words spoken by a target user and containing background noise in the presence of the user device while the target user spoke the multiple words in the target speech;
receiving a sequence of time markers input by the target user as the target speaker speaks the target speech and in cadence with the target user speaking the multiple words in the target speech, wherein each time marker in the sequence of time markers is received responsive to the user device detecting, via a sensor, a respective user input indication from the target speaker directed toward the user device;
correlating the sequence of time markers with the input audio signal to generate enhanced audio features that separate the target speech from the background noise in the input audio signal; and
processing, using a speech recognition model, the enhanced audio features to generate a transcription of the target speech.
2 . The computer-implemented method of claim 1 , wherein correlating the sequence of time markers with the input audio signal comprises:
computing, using the sequence of time markers, a sequence of word time stamps each designating a respective time corresponding to one of the multiple words in the target speech that was spoken by the target user; and
separating, using the sequence of computed word time stamps, the target speech from the background noise in the input audio signal to generate the enhanced audio features.
3 . The computer-implemented method of claim 2 , wherein separating the target speech from the background noise in the input audio signal comprises removing, from inclusion in the enhanced audio features, the background noise.
4 . The computer-implemented method of claim 2 , wherein separating the target speech from the background noise in the input audio signal comprises designating the sequence of word time stamps to corresponding audio segments of the enhanced audio features to differentiate the target speech from the background noise.
5 . The computer-implemented method of claim 1 , wherein receiving the sequence of time markers input by the target user comprises receiving each time marker in the sequence of time markers in response to the target user touching or pressing a predefined region of the user device or another device in communication with the data processing hardware.
6 . The computer-implemented method of claim 5 , wherein the predefined region of the user device or the other device comprises a physical button disposed on the user device or the other device.
7 . The computer-implemented method of claim 5 , wherein the predefined region of the user device or the other device comprises a graphical button displayed on a graphical user interface of the user device.
8 . The computer-implemented method of claim 1 , wherein receiving the sequence of time markers input by the target user comprises receiving each time marker in the sequence of time markers in response to a sensor in communication with the data processing hardware detecting the target user performing a predefined gesture.
9 . The computer-implemented method of claim 1 , wherein a number of time markers in the sequence of time markers input by the user is equal to a number of the multiple words spoken by the target user in the target speech.
10 . The computer-implemented method of claim 1 , wherein the data processing hardware resides on the user device associated with the target user.
11 . The computer-implemented method of claim 1 , wherein the data processing hardware resides on a remote server in communication with the user device associated with the target user.
12 . The computer-implemented method of claim 1 , wherein the background noise in contained in the input audio signal comprises competing speech spoken by one or more other users.
13 . The computer-implemented method of claim 1 , wherein the target speech spoken by the target user comprises a query directed toward a digital assistant executing on the data processing hardware, the query specifying an operation for the digital assistant to perform.
14 . A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform comprising:
receiving an input audio signal captured by a user device, the input audio signal corresponding to target speech of multiple words spoken by a target user and containing background noise in the presence of the user device while the target user spoke the multiple words in the target speech;
receiving a sequence of time markers input by the target user as the target speaker speaks the target speech and in cadence with the target user speaking the multiple words in the target speech, wherein each time marker in the sequence of time markers is received responsive to the user device detecting, via a sensor, a respective user input indication from the target speaker directed toward the user device;
correlating the sequence of time markers with the input audio signal to generate enhanced audio features that separate the target speech from the background noise in the input audio signal; and
processing, using a speech recognition model, the enhanced audio features to generate a transcription of the target speech.
15 . The system of claim 14 , wherein correlating the sequence of time markers with the input audio signal comprises:
computing, using the sequence of time markers, a sequence of word time stamps each designating a respective time corresponding to one of the multiple words in the target speech that was spoken by the target user; and
separating, using the sequence of computed word time stamps, the target speech from the background noise in the input audio signal to generate the enhanced audio features.
16 . The system of claim 15 , wherein separating the target speech from the background noise in the input audio signal comprises removing, from inclusion in the enhanced audio features, the background noise.
17 . The system of claim 15 , wherein separating the target speech from the background noise in the input audio signal comprises designating the sequence of word time stamps to corresponding audio segments of the enhanced audio features to differentiate the target speech from the background noise.
18 . The system of claim 14 , wherein receiving the sequence of time markers input by the target user comprises receiving each time marker in the sequence of time markers in response to the target user touching or pressing a predefined region of the user device or another device in communication with the data processing hardware.
19 . The system of claim 18 , wherein the predefined region of the user device or the other device comprises a physical button disposed on the user device or the other device.
20 . The system of claim 18 , wherein the predefined region of the user device or the other device comprises a graphical button displayed on a graphical user interface of the user device.
21 . The system of claim 14 , wherein receiving the sequence of time markers input by the target user comprises receiving each time marker in the sequence of time markers in response to a sensor in communication with the data processing hardware detecting the target user performing a predefined gesture.
22 . The system of claim 14 , wherein a number of time markers in the sequence of time markers input by the user is equal to a number of the multiple words spoken by the target user in the target speech.
23 . The system of claim 14 , wherein the data processing hardware resides on the user device associated with the target user.
24 . The system of claim 14 , wherein the data processing hardware resides on a remote server in communication with the user device associated with the target user.
25 . The system of claim 14 , wherein the background noise in contained in the input audio signal comprises competing speech spoken by one or more other users.
26 . The system of claim 14 , wherein the target speech spoken by the target user comprises a query directed toward a digital assistant executing on the data processing hardware, the query specifying an operation for the digital assistant to perform.