IP Library › Granted Patent US 11,437,057
Granted Patent B2
US 11,437,057 · App. 16/950,158 · Granted Sep 6, 2022

Detection of TV state using sub-audible signal

Inventors: Trausti Thor Kristjansson (San Jose, CA); Srivatsan Kandadai (San Jose, CA); Mark Lawrence (Bainbridge Island, WA); Balsa Laban (Mercer Island, WA); Anna Chen Santos (Mountain View, CA); Joseph Pedro Tavares (Kenmore, WA); Miroslav Ristic (Santa Clara, CA); Valere Joseph Vanderschaegen (Kalama, WA)
Assignee: Amazon Technologies, Inc.
G10L25/57G06F3/167G10L15/22H04N5/602H04N17/04H04N21/42204G10L2015/226H04N21/42206
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,437,057
App. No.
16/950,158
Granted
Sep 6, 2022
Kind
B2
Abstract

A computer-implemented method includes receiving, at a microphone of a voice-controlled device, a speech input, generating an electrical signal having a first gain level that is below a gain threshold for audible detection by a user, transmitting the electrical signal to the speaker and detecting, by the microphone, an audio signal that includes a combination of ambient noise and a probe audio signal, wherein the probe audio signal is output by the speaker based on the electrical signal. The method further includes determining a power level of the probe audio signal and determining a state of the display based on the power level of the probe audio signal.

Claims (65)

1. A device comprising:

a microphone;

a signal port configured to couple the device to a display that is coupled to a speaker;

a subaudible channel monitoring module comprising at least one processor and operatively coupled to the signal port and the microphone; and

memory in communication with the at least one processor and storing instructions that, when executed by the at least one processor, cause the device to:

generate an electrical signal having a first gain level that is below a gain threshold for audible detection;

transmit the electrical signal to the speaker;

detect, based on the microphone, an audio signal that includes a combination of ambient noise and a probe audio signal, wherein the probe audio signal is output by the speaker based on the electrical signal;

determine a power level of the probe audio signal; and

determine a state of the display based on the power level of the probe audio signal.

2. The device of claim 1 , wherein the electrical signal is transmitted to the speaker via the signal port.

3. The device of claim 1 , wherein the memory stores further instructions that, when executed by the at least one processor, additionally cause the device to:

generate a cross-correlation signal by computing a cross-correlation between the electrical signal and the audio signal, wherein determining the state of the display further comprises comparing a maximum value of the cross-correlation signal to a detection threshold power level, wherein the detection threshold power level is a predefined power level where the probe audio signal is inaudible.

4. The device of claim 1 , wherein execution of the instructions causes the device to determine the power level of the probe audio signal by:

generating a transformed electrical signal by computing a first Fourier transform of the electrical signal;

generating a transformed audio signal by computing a second Fourier transform of the audio signal;

generating a normalized audio signal by computing a phase transform of the transformed audio signal; and

computing a cross-correlation signal by computing a cross-correlation between the transformed electrical signal and the normalized audio signal.

5. The device of claim 4 , wherein the execution of the instructions causes the device to determine the state of the display by further comparing a maximum value of the cross-correlation signal to a detection threshold power level, wherein the detection threshold power level is a predefined power level where the probe audio signal is inaudible.

6. The device of claim 1 , wherein execution of the instructions causes the device to determine the power level of the probe audio signal by:

generating a transformed audio signal by computing a Fourier transform of the audio signal;

generating a frequency domain corrected audio signal by applying an inverse room impulse response function to the transformed audio signal;

generating a time domain corrected audio signal by applying an inverse Fourier transform to the frequency domain corrected audio signal;

generating a cross-correlation signal by computing a cross-correlation between the electrical signal and a time domain corrected audio signal; and

comparing a maximum value of the cross-correlation signal to a detection threshold power level.

7. The device of claim 1 , wherein execution of the instructions causes the device to determine the power level of the probe audio signal by:

determining a transformed electrical signal by computing a first Fourier transform of the electrical signal;

determining a transformed audio signal by computing a second Fourier transform of the audio signal;

determining a normalized audio signal by computing a phase transform of the transformed audio signal;

determining a phase transformed cross-correlation signal by taking an inverse Fourier transform of a product of the transformed electrical signal and the normalized audio signal;

determining a corrected cross-correlation signal by computing a cross-correlation between the phase transformed cross-correlation signal and a room impulse response function; and

comparing a maximum value of the corrected cross-correlation signal to a detection threshold power level.

8. The device of claim 1 , wherein the power level of the probe audio signal is lower than a detection threshold power level, wherein the detection threshold power level is a predefined power level where the probe audio signal is inaudible, and wherein execution of the instructions causes the device to:

determine the state of the display to be in an OFF state; and

modify the electrical signal by applying an increased gain level to the electrical signal.

9. The device of claim 1 , wherein the power level of the probe audio signal is lower than a detection threshold power level, wherein the detection threshold power level is a predefined power level where the probe audio signal is inaudible, and wherein execution of the instructions causes the device to:

determine the state of the display to be in an OFF state; and

increase a duration of time over which to compute a cross-correlation between the electrical signal and the audio signal.

10. The device of claim 1 , wherein execution of the instructions causes the device to determine the power level of the probe audio signal by generating a cross-correlation signal by computing a cross-correlation between the electrical signal and the audio signal, wherein the execution of the instructions further causes the device to determine the state of the display by comparing a maximum value of the cross-correlation signal to a detection threshold power level.

11. The device of claim 10 , wherein the memory stores further instructions that, when executed by the at least one processor, additionally cause the device to:

store running values of the cross-correlation signal; and

determine the maximum value of the cross-correlation from the running values.

12. The device of claim 11 , wherein the running values are computed recursively and an ith iteration of the running values of the cross-correlation ŷi(t) is computed using a recursive integrator of the form

ŷ i( t )=α ŷ (i−1)( t )+(1−α) y_i ( t )

where α is a constant between 0 and 1 exclusive, y_i (t) are cross-correlation values acquired during an ith iteration, ŷi(i-1) (t) are the running values computed during a previous, (i-1)th iteration.

13. The device of claim 10 , wherein the electrical signal has a predetermined waveform represents pseudo-random bit sequence.

14. One or more non-transitory computer-readable storage media storing instructions that, when executed by a device, cause the device to perform operations comprising:

generating an electrical signal having a first gain level that is below a gain threshold for audible detection;

transmitting the electrical signal to a speaker, the device being coupled via a signal port to a display that is coupled to the speaker;

detecting, based on a microphone of the device, an audio signal that includes a combination of ambient noise and a probe audio signal, wherein the probe audio signal is output by the speaker based on the electrical signal;

determining a power level of the probe audio signal; and

determining a state of the display based on the power level of the probe audio signal.

15. The one or more non-transitory computer-readable storage media of claim 14 , wherein determining the power level of the probe audio signal comprises generating a cross-correlation signal by computing a cross-correlation between the electrical signal and the audio signal, and wherein determining the state of the display further comprises comparing a maximum value of the cross-correlation signal to a threshold power level.

16. The one or more non-transitory computer-readable storage media of claim 15 , wherein the threshold power level corresponds to a probe audio signal sound pressure level of between −18 dB and 20 dB SPL, inclusive, as measure at a distance of 1 meter from the speaker.

17. A system comprising:

at least one processor; and

memory in communication with the at least one processor and storing instructions that, when executed by the at least one processor, cause the system to:

generate an electrical signal having a first gain level that is below a gain threshold for audible detection;

transmit the electrical signal to a speaker, the system being coupled via a signal port to a display that is coupled to the speaker;

detect, based on a microphone, an audio signal that includes a combination of ambient noise and a probe audio signal, wherein the probe audio signal is output by the speaker based on the electrical signal;

determine a power level of the probe audio signal; and

determine a state of the display based on the power level of the probe audio signal.

18. The system of claim 17 , wherein determining the power level of the probe audio signal comprises generating a cross-correlation signal by computing a cross-correlation between the electrical signal and the audio signal, and wherein determining the state of the display further comprises comparing a maximum value of the cross-correlation signal to a detection threshold power level.

19. The system of claim 18 , wherein the memory stores further instructions that, when executed by the at least one processor, additionally cause the system to determine a volume setting of the speaker based on the maximum value of the cross-correlation signal and at least one reference gain.

20. The system of claim 17 , wherein the memory stores further instructions that, when executed by the at least one processor, additionally cause the system to receiving, based on the microphone, a speech input, wherein the electrical signal is generated in response to the speech input.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 17, 2020
From: KRISTJANSSON, TRAUSTI THOR; KANDADAI, SRIVATSAN; LAWRENCE, MARK; LABAN, BALSA; SANTOS, ANNA CHEN; TAVARES, JOSEPH PEDRO; RISTIC, MIROSLAV; VANDERSCHAEGEN, VALERE JOSEPH
To: AMAZON TECHNOLOGIES, INC.
Reel/Frame 054390/0914 →
Continuity (2)
Division 15919108 · Mar 12, 2018
Related Publication 20210074314A1 · Mar 11, 2021
Cited By (1)
US 12,197,815