User presence detection
A speech-capture device can capture audio data during wakeword monitoring and use the audio data to determine if a user is present nearby the device, even if no wakeword is spoken. Audio such as speech, human originating sounds (e.g., coughing, sneezing), or other human related noises (e.g., footsteps, doors closing) can be used to detect audio. Audio frames are individually scored as to whether a human presence is detected in the particular audio frames. The scores are then smoothed relative to nearby frames to create a decision for a particular frame. Presence information can then be sent according to a periodic schedule to a remote device to create a presence “heartbeat” that regularly identifies whether a user is detected proximate to a speech-capture device.
1. A computer-implemented method comprising:
receiving first audio data corresponding to first audio in an environment of a device;
processing the first audio data using a deep neural network (DNN) model to determine a first score corresponding to a first audio frame of the first audio data, the first score corresponding to a likelihood that human activity was present in the environment;
determining that the first score is above a score threshold;
sending, at a first time, a first indicator of human presence to a remote device;
receiving second audio data corresponding to second audio in the environment, the second audio occurring after the first audio;
processing the second audio data using the DNN model to determine a second score corresponding to a second audio frame of the second audio data, the second score corresponding to a likelihood that human activity was present in the environment;
determining that the second score is above the score threshold; and
sending, at a second time, a second indicator of human presence to the remote device, the second time being a preconfigured time interval after the first time.
2. The computer-implemented method of claim 1 , wherein the first audio data comprises a plurality of audio feature vectors, and the method further comprises:
sending, to a wakeword detection component, the first audio data;
processing the first audio data using the wakeword detection component to detect a wakeword represented in the first audio data; and
sending at least a portion of the first audio data to the remote device for speech processing.
3. The computer-implemented method of claim 1 , further comprising:
processing the first audio data using the DNN model to determine a third score corresponding to a third audio frame of the first audio data, the third score corresponding to a likelihood that human activity was present in the environment;
determining that the third score is above the score threshold;
identifying a plurality of audio frames between the first audio frame and the third audio frame;
determining that a quantity of the plurality of audio frames have a corresponding score above the score threshold; and
determining that the quantity of the plurality is above a quantity threshold.
4. The computer-implemented method of claim 1 , further comprising:
receiving third audio data corresponding to third audio in the environment, the third audio occurring after the second audio;
processing the third audio data using the DNN model to determine a third score corresponding to a third audio frame of the third audio data, the third score corresponding to a likelihood that human activity was present in the environment;
determining that the third score is below the score threshold; and
sending, at a third time, a third indicator to the remote device, the third indicator indicating that there is no human presence, the third time being the preconfigured time interval after the second time.
5. A device comprising:
at least one processor; and
memory including instructions operable to be executed by the at least one processor to perform a set of actions to configure the device to:
receive first audio data corresponding to first audio captured by at least one microphone;
process the first audio data to determine a first score corresponding to a likelihood that human activity was present in an environment of the at least one microphone;
determine that the first score is above a score threshold;
determine a first indicator of user presence based at least in part on the first score;
send, at a first time, the first indicator of user presence to a remote device;
receive second audio data corresponding to second audio captured by the at least one microphone;
process the second audio data to determine that user presence is represented in the second audio data; and
send, at a second time, a second indicator of user presence to the remote device, wherein the second time is a preconfigured time interval after the first time.
6. The device of claim 5 , wherein the first audio data corresponds to a plurality of audio frames and the memory includes additional instructions operable to be executed by the at least one processor to further configure the device to:
determine the first score corresponding to a first audio frame of the plurality of audio frames.
7. The device of claim 6 , wherein the memory includes additional instructions operable to be executed by the at least one processor to further configure the device to:
determine a second score corresponding to a second audio frame of the plurality of audio frames, the second score corresponding to a likelihood that human activity was present in the environment;
determine that the second score is above a score threshold;
identify a subset of audio frames between the first audio frame and the second audio frame; and
determine that a majority of audio frames of the subset of audio frames correspond to a respective score above the score threshold.
8. The device of claim 7 , wherein the memory includes additional instructions operable to be executed by the at least one processor to further configure the device to:
determine that a quantity of audio frames of the subset is above a quantity threshold.
9. The device of claim 5 , wherein the memory includes additional instructions operable to be executed by the at least one processor to further configure the device to:
process the first audio data to determine a plurality of audio feature vectors; and
send at least a portion of the plurality of audio feature vectors to the remote device for speech processing.
10. The device of claim 5 , wherein the preconfigured time interval is a time interval from 30 seconds to two minutes.
11. The device of claim 5 , wherein the memory includes additional instructions operable to be executed by the at least one processor to further configure the device to:
use a trained model to determine the first score.
12. The device of claim 5 , wherein the first and second indicators comprise respective binary values.
13. A device, comprising:
at least one processor; and
memory including instructions operable to be executed by the at least one processor to perform a set of actions to configure the device to:
receive first audio data corresponding to first audio captured by at least one microphone,
process the first audio data to determine that user presence is represented in the first audio data,
send, at a first time, a first indicator of user presence to a remote device,
receive second audio data corresponding to second audio captured by the at least one microphone,
process the second audio data to determine that user presence is represented in the second audio data,
send, at a second time, a second indicator of user presence to the remote device, wherein the second time is a preconfigured time interval after the first time, and
send at least a portion of the first audio data to the remote device for speech processing.
14. The device of claim 13 , wherein the memory includes additional instructions operable to be executed by the at least one processor to further configure the device to:
use a trained model to determine that user presence is represented in the first audio data; and
use the trained model to determine that user presence is represented in the second audio data.
15. The device of claim 13 , further comprising a wakeword detection component, and wherein the first audio data comprises a plurality of audio feature vectors and the memory includes additional instructions operable to be executed by the at least one processor to further configure the device to:
send, to the wakeword detection component, the first audio data; and
process the first audio data using the wakeword detection component to detect a wakeword represented in the first audio data.
16. The device of claim 13 , wherein the memory includes additional instructions operable to be executed by the at least one processor to further configure the device to:
process the first audio data to determine a plurality of audio feature vectors; and
send at least a portion of the plurality of audio feature vectors to the remote device for speech processing.
17. A device, comprising:
at least one processor; and
memory including instructions operable to be executed by the at least one processor to perform a set of actions to configure the device to:
receive first audio data corresponding to first audio captured by at least one microphone,
process the first audio data to determine that user presence is represented in the first audio data,
send, at a first time, a first indicator of user presence to a remote device,
receive second audio data corresponding to second audio captured by the at least one microphone,
process the second audio data to determine that user presence is represented in the second audio data,
send, at a second time, a second indicator of user presence to the remote device, wherein the second time is a preconfigured time interval after the first time, and wherein the preconfigured time interval is associated with a user profile corresponding to the at least one microphone.
18. The device of claim 17 , wherein the memory includes additional instructions operable to be executed by the at least one processor to further configure the device to:
use a trained model to determine that user presence is represented in the first audio data; and
use the trained model to determine that user presence is represented in the second audio data.
19. A computer-implemented method comprising:
receiving first audio data corresponding to first audio captured by at least one microphone;
processing the first audio data to determine a first score corresponding to a likelihood that human activity was present in an environment of the at least one microphone;
determining that the first score is above a score threshold;
determining a first indicator of user presence based at least in part on the first score;
sending, at a first time, the first indicator of user presence to a remote device;
receiving second audio data corresponding to second audio captured by the at least one microphone;
processing the second audio data to determine that user presence is represented in the second audio data; and
sending, at a second time, a second indicator of user presence to the remote device, wherein the second time is a preconfigured time interval after the first time.
20. The computer-implemented method of claim 19 , wherein the first audio data corresponds to a plurality of audio frames and determining the first score further comprises:
determining the first score corresponding to a first audio frame of the plurality of audio frames.
21. The computer-implemented method of claim 20 , further comprising:
determining a second score corresponding to a second audio frame of the plurality of audio frames, the second score corresponding to a likelihood that human activity was present in the environment;
determining that the second score is above a score threshold;
identifying a subset of audio frames between the first audio frame and the second audio frame; and
determining that a majority of audio frames of the subset of audio frames correspond to a respective score above the score threshold.
22. The computer-implemented method of claim 21 , further comprising:
determining that a quantity of audio frames of the subset is above a quantity threshold.
23. The computer-implemented method of claim 19 , further comprising:
processing the first audio data to determine a plurality of audio feature vectors; and
sending at least a portion of the plurality of audio feature vectors to the remote device for speech processing.
24. The computer-implemented method of claim 19 , wherein the preconfigured time interval is a time interval from 30 seconds to two minutes.
25. The computer-implemented method of claim 19 , wherein:
processing the first audio data further comprises using a trained model to determine the first score.
26. A computer-implemented method, comprising:
receiving first audio data corresponding to first audio captured by at least one microphone;
sending at least a portion of the first audio data to a remote device for speech processing;
processing the first audio data to determine that user presence is represented in the first audio data;
sending, at a first time, a first indicator of user presence to a remote device;
receiving second audio data corresponding to second audio captured by the at least one microphone;
processing the second audio data to determine that user presence is represented in the second audio data;
sending, at a second time, a second indicator of user presence to the remote device, wherein the second time is a preconfigured time interval after the first time.
27. The computer-implemented method of claim 26 , wherein the first audio data comprises a plurality of audio feature vectors, and the method further comprises:
sending, to a wakeword detection component, the first audio data; and
processing the first audio data using the wakeword detection component to detect a wakeword represented in the first audio data.
28. The computer-implemented method of claim 26 , further comprising:
processing the first audio data to determine a plurality of audio feature vectors; and
sending at least a portion of the plurality of audio feature vectors to the remote device for speech processing.
29. The computer-implemented method of claim 26 , wherein:
processing the first audio data further comprises using a trained model to determine that user presence is represented in the first audio data; and
processing the second audio data further comprises using the trained model to determine that user presence is represented in the second audio data.
30. A computer-implemented method, comprising:
receiving first audio data corresponding to first audio captured by at least one microphone;
processing the first audio data to determine that user presence is represented in the first audio data;
sending, at a first time, a first indicator of user presence to a remote device;
receiving second audio data corresponding to second audio captured by the at least one microphone;
processing the second audio data to determine that user presence is represented in the second audio data; and
sending, at a second time, a second indicator of user presence to the remote device, wherein the second time is a preconfigured time interval after the first time, and wherein the preconfigured time interval is associated with a user profile corresponding to the at least one microphone.
31. The computer-implemented method of claim 30 , wherein:
processing the first audio data further comprises using a trained model to determine that user presence is represented in the first audio data; and
processing the second audio data further comprises using the trained model to determine that user presence is represented in the second audio data.