Granted Patent B1

US 12,531,064 · App. 18/620,703 · Granted Jan 20, 2026

Inventor: Wai Chung Chu (San Jose, CA)

Assignee: Amazon Technologies, Inc.

G10L15/22G10L25/21H04R3/005

Abstract

Claims (73)

1 . An electronic device comprising:

a plurality of microphones;

a loudspeaker;

one or more processors; and

one or more non-transitory computer readable media storing processor executable instructions which, when executed using the one or more processors, cause the electronic device to perform operations comprising:

determining, based on first audio data corresponding to sound captured by at least two microphones of the plurality of microphones, first sound source localization data comprising:

first cell data indicating a first three-dimensional vector and a first power value associated with the first three-dimensional vector, and

second cell data indicating a second three-dimensional vector and a second power value associated with the second three-dimensional vector;

determining, based on the first sound source localization data:

a first metric representing direct-to-reverberant ratio information, and

a second metric representing direction variance information;

based on the first metric and the second metric, using a first machine learning model to determine model output estimating user engagement; and

executing, based on the model output, a first operation.

2 . The electronic device of claim 1 , wherein:

the first sound source localization data comprises data indicating, for each respective cell of a plurality of cells, a respective three-dimensional vector and a respective power value associated with the respective three-dimensional vector, the plurality of cells including a first cell corresponding to the first cell data and a second cell corresponding to the second cell data; and

the one or more computer readable media store further processor executable instructions which, when executed using the one or more processors, cause the electronic device to perform operations comprising:

determining, based on the first sound source localization data, that the first power value is a peak power value among the plurality of cells,

determining, based on the first sound source localization data, a mean power value for the plurality of cells, and

determining a peak-to-mean ratio value based on the first power value and the mean power value,

wherein the peak-to-mean ratio value is the first metric.

3 . The electronic device of claim 1 , wherein:

the first sound source localization data comprises data indicating, for each respective cell of a plurality of cells, a respective three-dimensional vector and a respective power value associated with the respective three-dimensional vector, the plurality of cells including a first cell corresponding to the first cell data and a second cell corresponding to the second cell data; and

the one or more computer readable media store further processor executable instructions which, when executed using the one or more processors, cause the electronic device to perform operations comprising:

determining, based on the first sound source localization data, first data indicating a power-weighted mean direction vector,

wherein the second metric is determined based on the first data.

4 . The electronic device of claim 1 , wherein the first sound source localization data represents sound source localization data for a first frame of audio data, and wherein the one or more computer readable media store further processor executable instructions which, when executed using the one or more processors, cause the electronic device to perform operations comprising:

determining, based on first audio data corresponding to sound captured by a second set of microphones of the plurality of microphones, second sound source localization data comprising data indicating, for each respective cell of a plurality of cells, a respective three-dimensional vector and a respective power value associated with the respective three-dimensional vector;

wherein the first metric is determined based on the first sound source localization data and the second sound source localization data; and

wherein the second metric is determined based on the first sound source localization data and the second sound source localization data.

5 . The electronic device of claim 1 , wherein the first sound source localization data represents sound source localization data for a first frame of audio data, and wherein the one or more computer readable media store further processor executable instructions which, when executed using the one or more processors, cause the electronic device to perform operations comprising:

determining, based on first audio data corresponding to sound captured by a second set of microphones of the plurality of microphones, second sound source localization data comprising data indicating, for each respective cell of a plurality of cells, a respective three-dimensional vector and a respective power value associated with the respective three-dimensional vector; and

determining, based on the second sound source localization data:

a third metric representing direct-to-reverberant ratio information, and

a fourth metric representing direction variance information,

wherein the determining of the model output using the first machine learning model is based on the third metric and the fourth metric.

6 . The electronic device of claim 1 , wherein the model output indicates that a user is engaging with the electronic device.

7 . The electronic device of claim 1 , wherein the model output indicates that a user is not engaging with the electronic device.

8 . The electronic device of claim 1 , wherein the first operation comprises turning off a light of the electronic device.

9 . The electronic device of claim 1 , wherein the model output comprises an engagement value indicating an engagement level and a confidence value indicating a confidence level in the engagement value.

10 . The electronic device of claim 1 , wherein the first operation comprises sending at least a subset of the first audio data to a remote system, and wherein the one or more computer readable media store further processor executable instructions which, when executed using the one or more processors, cause the electronic device to perform operations comprising:

outputting, using a speaker of the electronic device and based on first response data received from the remote system, audio representing speech responding to user speech.

11 . The electronic device of claim 1 , wherein the model output indicates that a user is not engaging with the electronic device, and wherein the first operation comprises powering down a component of the electronic device.

12 . The electronic device of claim 1 , wherein the first operation comprises transitioning to a different state.

13 . The electronic device of claim 1 , wherein the model output indicates that a user is engaging with the electronic device, and wherein the first operation comprises transitioning to an awake or active state.

14 . The electronic device of claim 1 , wherein the model output indicates that a user is not engaging with the electronic device, and wherein the first operation comprises transitioning to an inactive or sleep state.

15 . The electronic device of claim 1 , wherein the model output is a numerical value indicating an engagement level on a scale of engagement.

16 . An electronic device comprising:

a plurality of microphones;

a loudspeaker;

one or more processors; and

one or more non-transitory computer readable media storing processor executable instructions which, when executed using the one or more processors, cause the electronic device to perform operations comprising:

determining, based on first audio data corresponding to sound captured by at least two microphones of the plurality of microphones, first sound source localization data indicating, for each respective cell of a plurality of cells:

a respective three-dimensional vector, and

a respective power value associated with the respective three-dimensional vector;

determining, based on the first sound source localization data, a first power value that is a peak power value among the plurality of cells;

determining, based on the first sound source localization data, a mean power value for the plurality of cells;

determining a peak-to-mean ratio value based on the first power value and the mean power value;

determining, using a first machine learning model and based on the peak-to-mean ratio value, model output estimating user engagement; and

executing, based on the model output, a first operation.

17 . The electronic device of claim 16 , wherein the first operation comprises performing speech transcription on at least a portion of the first audio data.

18 . The electronic device of claim 16 , wherein the first operation comprises generating, based on at least a portion of the first audio data and using a second machine learning model, transcription data.

19 . An electronic device comprising:

a plurality of microphones;

a loudspeaker;

one or more processors; and

one or more non-transitory computer readable media storing processor executable instructions which, when executed using the one or more processors, cause the electronic device to perform operations comprising:

determining, based on first audio data corresponding to sound captured by at least two microphones of the plurality of microphones, first sound source localization data indicating, for each respective cell of a plurality of cells:

a respective three-dimensional vector, and

a respective power value associated with the respective three-dimensional vector;

determining, based on the first sound source localization data, a power-weighted mean direction vector representing direction variance information;

determining, using a first machine learning model and based on the power-weighted mean direction vector, model output estimating user engagement; and

executing, based on the model output, a first operation.

20 . The electronic device of claim 19 , wherein the first operation comprises performing speech transcription on at least a portion of the first audio data.

Assignments (1)

References Cited (35)