Anchored speech detection and speech recognition
A system configured to process speech commands may classify incoming audio as desired speech, undesired speech, or non-speech. Desired speech is speech that is from a same speaker as reference speech. The reference speech may be obtained from a configuration session or from a first portion of input speech that includes a wakeword. The reference speech may be encoded using a recurrent neural network (RNN) encoder to create a reference feature vector. The reference feature vector and incoming audio data may be processed by a trained neural network classifier to label the incoming audio data (for example, frame-by-frame) as to whether each frame is spoken by the same speaker as the reference speech. The labels may be passed to an automatic speech recognition (ASR) component which may allow the ASR component to focus its processing on the desired speech.
1. A computer-implemented method comprising:
receiving, as part of a first interaction, first audio data representing first speech spoken by a first speaker;
determining, using the first audio data, a reference feature vector corresponding to the first speaker;
after the first interaction, receiving, as part of a second interaction, second audio data different from the first audio data;
determining, using the reference feature vector and a trained model, that a first portion of the second audio data does not correspond to the first speaker, the first portion corresponding to a first time segment of the second audio data;
determining, using the reference feature vector and the trained model, that a second portion of the second audio data corresponds to the first speaker, the second portion corresponding to a second time segment of the second audio data;
based at least in part on determining that the second portion of the second audio data corresponds to the first speaker, executing a command corresponding to the second portion of the second audio data; and
based at least in part on determining that the first portion of the second audio data does not correspond to the first speaker, excluding the first portion of the second audio data from processing relating to execution of the command.
2. The computer-implemented method of claim 1 , further comprising:
determining a plurality of audio feature vectors corresponding to the first audio data; and
processing the plurality of audio feature vectors using a recurrent neural network (RNN) to determine the reference feature vector.
3. The computer-implemented method of claim 2 , wherein:
the RNN comprises an encoder with log-filterbank energy (LFBE) features that are normalized by applying recursive log-amplitude mean subtraction (LAMS).
4. The computer-implemented method of claim 1 , wherein:
the first audio data comprises a plurality of audio feature vectors,
the reference feature vector corresponds to a mean of the plurality of audio feature vectors in a log-filterbank energy (LFBE) domain,
the trained model corresponds to a feed-forward deep neural network (DNN) classifier, and
and the method further comprises:
receiving a first audio feature vector as part of the first portion of the second audio data;
subtracting the reference feature vector from the first audio feature vector to determine energy level difference data; and
processing the energy level difference data using the feed-forward DNN classifier to determine that the first portion of the second audio data does not correspond to the first speaker.
5. The computer-implemented method of claim 1 , wherein the first portion of the second audio data includes a first audio frame, a second audio frame, and a third audio frame and the method further comprises:
processing the reference feature vector, the first audio frame, the second audio frame, and the third audio frame using the trained model to determine a score corresponding to the second audio frame; and
using the score to determine that the first portion of the second audio data does not correspond to the first speaker.
6. The computer-implemented method of claim 1 , further comprising:
storing an association between the reference feature vector and a user profile;
determining that the second audio data is associated with the user profile; and
using the association to process the second audio data with respect to the reference feature vector.
7. The computer-implemented method of claim 1 , wherein:
the first speech is captured by a first device; and
the second audio data corresponds to audio captured by a second device.
8. A system, comprising:
at least one processor; and
at least one memory including instructions operable to be executed by the at least one processor to configure the system to:
receive, as part of a first interaction, first audio data representing first speech spoken by a first speaker;
determine, using the first audio data, a reference feature vector corresponding to the first speaker;
after the first interaction, receive, as part of a second interaction, second audio data different from the first audio data;
determine, using the reference feature vector and a trained model, that a first portion of the second audio data does not correspond to the first speaker, the first portion corresponding to a first time segment of the second audio data;
determine, using the reference feature vector and the trained model, that a second portion of the second audio data corresponds to the first speaker, the second portion corresponding to a second time segment of the second audio data;
based at least in part on a determination that the second portion of the second audio data corresponds to the first speaker, execute a command corresponding to the second portion of the second audio data; and
based at least in part on the determination that the first portion of the second audio data does not correspond to the first speaker, exclude the first portion of the second audio data from processing related to execution of the command.
9. The system of claim 8 , wherein the memory further comprises instructions that, when executed by the at least one processor, further configure the system to:
determine a plurality of audio feature vectors corresponding to the first audio data; and
process the plurality of audio feature vectors using a recurrent neural network (RNN) to determine the reference feature vector.
10. The system of claim 9 , wherein:
the RNN comprises an encoder with log-filterbank energy (LFBE) features that are normalized by applying recursive log-amplitude mean subtraction (LAMS).
11. The system of claim 8 , wherein:
the first audio data comprises a plurality of audio feature vectors,
the reference feature vector corresponds to a mean of the plurality of audio feature vectors in a log-filterbank energy (LFBE) domain,
the trained model corresponds to a feed-forward deep neural network (DNN) classifier, and
and the memory further comprises instructions that, when executed by the at least one processor, further configure the system to:
receive a first audio feature vector as part of the first portion of the second audio data;
subtract the reference feature vector from the first audio feature vector to determine energy level difference data; and
process the energy level difference data using the feed-forward DNN classifier to determine that the first portion of the second audio data does not correspond to the first speaker.
12. The system of claim 8 , wherein the first portion of the second audio data includes a first audio frame, a second audio frame, and a third audio frame and the memory further comprises instructions that, when executed by the at least one processor, further configure the system to:
process the reference feature vector, the first audio frame, the second audio frame, and the third audio frame using the trained model to determine a score corresponding to the second audio frame; and
use the score to determine that the first portion of the second audio data does not correspond to the first speaker.
13. The system of claim 8 , wherein the memory further comprises instructions that, when executed by the at least one processor, further configure the system to:
store an association between the reference feature vector and a user profile;
determine that the second audio data is associated with the user profile; and
use the association to process the second audio data with respect to the reference feature vector.
14. The system of claim 8 , wherein:
the first speech is captured by a first device; and
the second audio data corresponds to audio captured by a second device.
15. A computer-implemented method comprising:
receiving audio data as part of an interaction with a device;
determining that a first portion of the audio data represents a wakeword;
based at least in part on determining that the first portion of the audio data represents the wakeword, processing the first portion of the audio data to determine a reference feature vector corresponding to a first speaker of the wakeword;
determining, using the reference feature vector and a trained model, that a second portion of the audio data corresponds to a second speaker different from the first speaker;
determining, using the reference feature vector and the trained model, that a third portion of the audio data corresponds to the first speaker;
based at least in part on determining that the third portion corresponds to the first speaker, executing a command corresponding to the third portion; and
based at least in part on determining that the second portion of the audio data corresponds to the second speaker, excluding the second portion of the audio data from processing related to execution of the command.
16. The computer-implemented method of claim 15 , further comprising:
determining a plurality of audio feature vectors corresponding to the first portion of the audio data; and
processing the plurality of audio feature vectors using a recurrent neural network (RNN) to determine the reference feature vector.
17. The computer-implemented method of claim 16 , wherein:
the RNN comprises an encoder with log-filterbank energy (LFBE) features that are normalized by applying recursive log-amplitude mean subtraction (LAMS).
18. The computer-implemented method of claim 15 , wherein:
the first portion of the audio data comprises a plurality of audio feature vectors,
the reference feature vector corresponds to a mean of the plurality of audio feature vectors in a log-filterbank energy (LFBE) domain,
the trained model corresponds to a feed-forward deep neural network (DNN) classifier, and
and the method further comprises:
receiving a first audio feature vector as part of the second portion of the audio data;
subtracting the reference feature vector from the first audio feature vector to determine energy level difference data; and
processing the energy level difference data using the feed-forward DNN classifier to determine that the second portion corresponds to the second speaker.
19. The computer-implemented method of claim 15 , wherein the second portion of the audio data includes a first audio frame, a second audio frame, and a third audio frame and the method further comprises:
processing the reference feature vector, the first audio frame, the second audio frame, and the third audio frame using the trained model to determine a score corresponding to the second audio frame; and
using the score to determine that the second portion corresponds to the second speaker.