Key phrase spotting
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for detecting utterances of a key phrase in an audio signal. One of the methods includes receiving, by a key phrase spotting system, an audio signal encoding one or more utterances; while continuing to receive the audio signal, generating, by the key phrase spotting system, an attention output using an attention mechanism that is configured to compute the attention output based on a series of encodings generated by an encoder comprising one or more neural network layers; generating, by the key phrase spotting system and using attention output, output that indicates whether the audio signal likely encodes the key phrase; and providing, by the key phrase spotting system, the output that indicates whether the audio signal likely encodes the key phrase.
1 . A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
receiving a textual input characterizing a key phrase;
determining, from the textual input for the key phrase, using a key phrase encoder network of a key phrase spotting system, a key phrase encoding for a sequence of target sub-word units that represent the key phrase;
for each corresponding frame of multiple frames representing an incoming audio signal, processing, by an acoustic encoder network of the key phrase spotting system, the corresponding frame to generate an encoder output that represents an acoustic encoding of the corresponding frame;
determining, without performing speech recognition and without generating a transcription of the incoming audio signal, the incoming audio signal encodes an utterance of the key phrase based on the key phrase encoding and the encoder outputs generated for the multiple frames; and
in response to determining that the incoming audio signal encodes the utterance of the key phrase, triggering an automatic speech recognition system to generate a transcription of other words following the key phrase that are encoded in the incoming audio signal, the automatic speech recognition system different than the key phrase spotting system.
2 . The method of claim 1 , wherein determining the incoming audio signal encodes an utterance of the key phrase based on the sub-word units predicted for the multiple frames comprises:
for each corresponding frame of the multiple frames representing the incoming audio signal, predicting, based on the key phrase encoding and the encoder output generated for the corresponding frame, a sub-word unit;
generating, by a softmax layer, a probability that the incoming audio signal includes the sub-word units predicted for the multiple frames; and
generating, using the probability, an output that indicates the incoming audio signal likely encodes the utterance of the key phrase.
3 . The method of claim 1 , wherein the target sub-word units comprise phonemes.
4 . The method of claim 1 , wherein each of the target sub-word units comprises the same one of a grapheme, a phoneme, or a hidden Markov model state.
5 . The method of claim 1 , wherein, when the corresponding frame is located after an initial frame of the multiple frames representing the incoming audio signal, the operations further comprise predicting a sub-word unit for the corresponding frame based on data representing a previously predicted sub-word for a previous frame of the multiple frames.
6 . The method of claim 1 , wherein the data processing hardware resides on a user device.
7 . The method of claim 6 , wherein the user device comprises a mobile phone.
8 . The method of claim 1 , wherein the operations further comprise, for each corresponding frame of the multiple frames:
generating, using an attention mechanism, a context vector for the corresponding phrase based on the key phrase encoding output from the key phrase encoder network,
predicting, using the context vector generated for the corresponding frame and the encoder output generated for the corresponding frame, sub-word unit.
9 . The method of claim 8 , wherein generating the context vector for the corresponding frame is further based on a representation of a prior state of the attention mechanism after processing data for one or more of any frames of the multiple frames that precede the corresponding frame.
10 . A system comprising:
data processing hardware; and
memory hardware storing instructions that when executed cause the data processing hardware to perform operations comprising:
receiving a textual input characterizing a key phrase;
determining, from the textual input for the key phrase, using a key phrase encoder network of a key phrase spotting system, a key phrase encoding for a sequence of target sub-word units that represent the key phrase;
for each corresponding frame of multiple frames representing an incoming audio signal, processing, by an acoustic encoder network of the key phrase spotting system, the corresponding frame to generate an encoder output that represents an acoustic encoding of the corresponding frame;
determining, without performing speech recognition and without generating a transcription of the incoming audio signal, the incoming audio signal encodes an utterance of the key phrase based on the key phrase encoding and the encoder outputs generated for the multiple frames; and
in response to determining that the incoming audio signal encodes the utterance of the key phrase, triggering an automatic speech recognition system to generate a transcription of other words following the key phrase that are encoded in the incoming audio signal, the automatic speech recognition system different than the key phrase spotting system.
11 . The system of claim 10 , wherein determining the incoming audio signal encodes an utterance of the key phrase based on the sub-word units predicted for the multiple frames comprises:
for each corresponding frame of the multiple frames representing the incoming audio signal, predicting, based on the key phrase encoding and the encoder output generated for the corresponding frame, a sub-word unit;
generating, by a softmax layer, a probability that the incoming audio signal includes the sub-word units predicted for the multiple frames; and
generating, using the probability, an output that indicates the incoming audio signal likely encodes the utterance of the key phrase.
12 . The system of claim 10 , wherein the target sub-word units comprise phonemes.
13 . The system of claim 10 , wherein each of the target sub-word units comprises the same one of a grapheme, a phoneme, or a hidden Markov model state.
14 . The system of claim 10 , wherein, when the corresponding frame is located after an initial frame of the multiple frames representing the incoming audio signal, the operations further comprise predicting a sub-word unit for the corresponding frame based on data representing a previously predicted sub-word for a previous frame of the multiple frames.
15 . The system of claim 10 , wherein the data processing hardware resides on a user device.
16 . The system of claim 15 , wherein the user device comprises a mobile phone.
17 . The system of claim 10 , wherein the operations further comprise, for each corresponding frame of the multiple frames:
generating, using an attention mechanism, a context vector for the corresponding phrase based on the key phrase encoding output from the key phrase encoder network,
predicting, using the context vector generated for the corresponding frame and the encoder output generated for the corresponding frame, sub-word unit.
18 . The system of claim 17 , wherein generating the context vector for the corresponding frame is further based on a representation of a prior state of the attention mechanism after processing data for one or more of any frames of the multiple frames that precede the corresponding frame.