Environment aware voice-assistant devices, and related systems and methods
An appliance can include a microphone transducer, a processor, and a memory storing instructions. The appliance is configured to receive an audio signal at the microphone transducer and to detect an utterance in the audio signal. The appliance is further configured to classify a speech mode based on the utterance. The appliance is further configured to determine conditions of an environment of the appliance. The appliance is further configured to select at least one of a playback volume or a speech output mode from a plurality of speech output modes based on the classification, and the conditions of the environment of the appliance. The appliance is further configured to adapt the playback volume and/or mode of played-back speech according to the speech output mode. The appliance may be configured to synthesize speech according to the speech output mode, or to modify synthesized speech according to the speech output mode.
1. A method comprising:
obtaining an audio signal;
analyzing the audio signal to identify a sound; and
in response to identifying the sound in the audio signal:
classifying a speech mode of the sound, the speech mode indicating a first volume of the sound;
determining an acoustic cue from the audio signal;
determining respective directions of arrival for the sound and the acoustic cue; and
selecting a speech output mode, wherein the speech output mode indicates a second volume based on the first volume, the acoustic cue, and the determined respective directions of arrival.
2. The method of claim 1 , wherein the acoustic cue comprises one or more additional sounds in the audio signal captured from a listening environment from which the audio signal is generated.
3. The method of claim 1 , wherein:
the sound comprises at least one word, and
the speech mode corresponds to an energy content of the at least one word.
4. The method of claim 1 , further comprising:
determining a command from the sound; and
outputting synthesized speech, at the selected speech output mode, in response to the command.
5. The method of claim 1 , further comprising:
selecting a frequency of the audio signal; and
filtering the selected frequency of the audio signal, wherein filtering the selected frequency raises a level of the selected frequency above a noise level.
6. The method of claim 1 , further comprising:
determining a question from the sound; and
outputting synthesized speech, at the selected speech output mode, as a response to the question.
7. The method of claim 1 , wherein:
determining the respective directions of arrival comprises:
obtaining a first direction of arrival of the sound; and
obtaining a second direction of arrival of the acoustic cue, the second direction of arrival that being different from the first direction of arrival, and
selecting the speech output mode further based on a difference between the first direction of arrival and the second direction of arrival.
8. The method of claim 7 , wherein:
obtaining the first direction of arrival comprising obtaining the audio signal from a first microphone, and
obtaining the second direction of arrival comprises obtaining the acoustic cue from a second microphone.
9. The method of claim 1 , further comprising:
determining a non-acoustic cue comprising a room layout; and
selecting the speech output mode further based on the non-acoustic cue.
10. The method of claim 1 , further comprising:
determining a non-acoustic cue selected from at least one of a time of day and a determination an appliance generating the selected speech output mode is located in a vehicle; and
selecting the speech output mode further based on the non-acoustic cue.
11. An appliance, comprising:
one or more microphones;
a memory configured to store instructions; and
a controller configured to execute the instructions stored on the memory, the instructions causing the controller to:
obtain, from the one or more microphones, an audio signal;
analyze the audio signal to identify a sound; and
in response to identifying the sound in the audio signal:
classify a speech mode of the sound, the speech mode indicating a first volume of the sound;
determine an acoustic cue from the audio signal;
determine respective directions of arrival for the sound and the acoustic cue; and
select a speech output mode, wherein the speech output mode indicates a second volume based on the first volume, the acoustic cue, and the determined respective directions of arrival.
12. The appliance of claim 11 , wherein the controller is further configured to:
determine a speech rate of the audio signal; and
select the selected speech output mode further based on the speech rate.
13. The appliance of claim 11 , wherein the controller is further configured to:
obtain, using a first microphone of the one or more microphones, a first direction of arrival of the sound; and
obtain, using a second microphone of the one or more microphones, a second direction of arrival of the acoustic cue, wherein the second direction of arrival is different from the first direction of arrival, and
determine, based on the first direction of arrival and the second direction of arrival, the respective directions of arrival.
14. The appliance of claim 11 , wherein the controller is further configured to:
use the acoustic cue to determine a distance between the appliance and a user; and
select the speech output mode further based on the distance.
15. The appliance of claim 11 , wherein the controller is further configured to:
select a frequency of the audio signal; and
filter the selected frequency of the audio signal, wherein filtering the selected frequency raises a level of the selected frequency above a noise level.
16. A non-transitory computer-readable media including computer executable instructions that, when executed, cause a computing environment in an appliance to implement steps that include:
obtaining an audio signal;
analyzing the audio signal to identify a sound; and
in response to identifying the sound in the audio signal:
classifying a speech mode of the sound, the speech mode indicating a first volume of the sound;
determining an acoustic cue from the audio signal;
obtaining a first direction of arrival of the sound;
obtaining a second direction of arrival of the acoustic cue, the second direction of arrival being different from the first direction of arrival; and
selecting a speech output mode, wherein the speech output mode indicates a second volume based on the first volume, the acoustic cue, and a difference between the first direction of arrival and the second direction of arrival.
17. The non-transitory computer-readable media of claim 16 , wherein the acoustic cue comprises one or more additional sounds in the audio signal captured from a listening environment from which the audio signal is generated.
18. The non-transitory computer-readable media of claim 16 , wherein the steps further include:
determining a command from the sound; and
outputting synthesized speech, at the selected speech output mode, in response to the command.
19. The non-transitory computer-readable media of claim 16 , wherein the steps further include:
determining a non-acoustic cue selected from at least a time of day; and
selecting the speech output mode further based on the non-acoustic cue.
20. The non-transitory computer-readable media of claim 16 , wherein the steps further include:
determining a speech rate of the audio signal; and
selecting the selected speech output mode further based on the speech rate.