IP Library Granted Patent US 11,264,047
Granted Patent B2
US 11,264,047 · App. 16/858,200 · Granted Mar 1, 2022

Causing a voice enabled device to defend against inaudible signal attacks

Inventors: Nirupam Roy (Champaign, IL); Romit Roy Choudhury (Champaign, IL); Haitham Al Hassanieh (Champaign, IL)
Assignee: Board of Trustees of the University of Illinois
G10L21/06B06B1/0292G10K9/122G10K11/178G10K15/02H04B1/04H04B1/20H04R1/323H04R1/40H04R27/00H04R2217/03H04R2227/003H04R2227/005
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,264,047
App. No.
16/858,200
Granted
Mar 1, 2022
Kind
B2
Abstract

A voice enabled device includes a transducer to capture multiple inaudible signals received from multiple ultrasonic speakers and audio recording electronics to process the multiple inaudible signals to generate digital output samples, which are recorded sound data comprising non-linearities from frequency-shifted versions of the multiple inaudible signals to within an audible frequency range. A processing device is to detect, within the recorded sound data, at least a portion of the non-linearities, e.g., via: comparison of the recorded sound data with expected patterns from an audible audio signal generated by human voice; and detection of non-linear variations within the recorded sound data as compared to the expected patterns. In response to the detection, the processing device is further to suppress an action programmed for response to a voice command corresponding to the recorded sound data.

Claims (66)

1. A voice enabled device comprising:

a transducer to capture multiple inaudible signals received from multiple ultrasonic speakers;

audio recording electronics coupled to the transducer, the audio recording electronics to process the multiple inaudible signals to generate digital output samples, which are recorded sound data comprising non-linearities from frequency-shifted versions of the multiple inaudible signals to within an audible frequency range; and

a processing device coupled to the audio recording electronics, wherein the processing device is to detect, within the recorded sound data, at least a portion of the non-linearities, wherein to detect the at least a portion of the non-linearities, the processing device is to:

compare the recorded sound data with expected patterns from an audible audio signal generated by human voice; and

detect non-linear variations within the recorded sound data as compared to the expected patterns, wherein the non-linear variations are detected as a result of the at least a portion of the non-linearities located within the recorded sound data corresponding to a strongest portion of the expected patterns and being in a sub-50 hertz (Hz) band; and

wherein, in response to the detection, the processing device is further to suppress an action programmed for response to a voice command corresponding to the recorded sound data.

2. The voice enabled device of claim 1 , wherein a width of a fundamental frequency combined with widths of corresponding harmonics of the fundamental frequency within the audible audio signal is a time-varying frequency, and to detect the at least a portion of the non-linearities, the processing device is further to:

determine a first energy variation over time within a frequency band that is between zero and the time-varying frequency; and

correlate the first energy variation with a second energy variation at a fundamental frequency in the recorded sound data that is greater than the frequency band.

3. The voice enabled device of claim 2 , wherein to determine the first energy variation, the processing device is to use standard acoustic libraries, the processing device further to:

determine a first average power of the fundamental frequency around the width of the time-varying frequency;

determine a second average power, over time, of the recorded sound data that is within the frequency band;

remove, from the first average power and the second average power, windows of time during which the fundamental frequency falls below the first average power; and

compute a correlation coefficient between the first average power and the second average power.

4. The voice enabled device of claim 2 , wherein the processing device is further to employ an average width of the frequency band that is approximately 20 hertz (Hz).

5. The voice enabled device of claim 1 , wherein, to detect the non-linear variations, the processing device is further to detect that the at least a portion of the non-linearities are at positively-biased harmonics comprising an amplitude skew.

6. The voice enabled device of claim 5 , wherein to detect that the at least a portion of the non-linearities are at positively-biased harmonics comprising the amplitude skew, the processing device is further to:

determine a first ratio of maximum and minimum amplitude of the audible audio signal;

determine a second ratio of maximum and minimum amplitude of the recorded sound data; and

compare the second ratio to the first ratio.

7. The voice enabled device of claim 1 , wherein the processing device is further to:

compare the recorded sound data to pre-recorded voice commands; and

determine that the recorded sound data corresponds to the voice command listed among the pre-recorded voice commands.

8. A method comprising:

capturing, using a transducer, multiple inaudible signals received from multiple ultrasonic speakers;

generating, using audio recording electronics coupled to the transducer, digital output samples of the multiple inaudible signals, wherein the digital output samples are recorded sound data comprising non-linearities from frequency-shifted versions of the multiple inaudible signals to within an audible frequency range;

detecting, within the recorded sound data using a processing device, at least a portion of the non-linearities, wherein the detecting comprises:

comparing the recorded sound data with expected patterns from an audible audio signal generated by human voice; and

detecting non-linear variations within the recorded sound data as compared to the expected patterns, wherein the non-linear variations are detected as a result of the at least a portion of the non-linearities located within the recorded sound data corresponding to a strongest portion of the expected patterns and being in a sub-50 hertz (Hz) band; and

in response to the detecting, suppressing, using the processing device, an action programmed for response to a voice command corresponding to the recorded sound data.

9. The method of claim 8 , wherein a width of a fundamental frequency combined with widths of corresponding harmonics of the fundamental frequency within the audible audio signal is a time-varying frequency, and detecting the at least a portion of the non-linearities further comprises:

determining a first energy variation over time within a frequency band that is between zero and the time-varying frequency; and

correlating the first energy variation with a second energy variation at a fundamental frequency in the recorded sound data that is greater than the frequency band.

10. The method of claim 9 , further comprising:

employing standard acoustic libraries to determine the first energy variation, the fundamental frequency, and the corresponding harmonics;

determining a first average power of the fundamental frequency around the width of the time-varying frequency;

determining a second average power, over time, of the recorded sound data that is within the frequency band;

removing, from the first average power and the second average power, windows of time during which the fundamental frequency falls below the first average power; and

compute a correlation coefficient between the first average power and the second average power.

11. The method of claim 9 , further comprising employing an average width of the frequency band that is approximately 20 hertz (Hz).

12. The method of claim 8 , wherein detecting the non-linear variations further comprises detecting that the at least a portion of the non-linearities are at positively-biased harmonics comprising an amplitude skew.

13. The method of claim 12 , wherein detecting that the at least a portion of the non-linearities are at positively-biased harmonics comprising the amplitude skew further comprises:

determining a first ratio of maximum and minimum amplitude of the audible audio signal;

determining a second ratio of maximum and minimum amplitude of the recorded sound data; and

comparing the second ratio to the first ratio.

14. The method of claim 8 , further comprising:

comparing the recorded sound data to pre-recorded voice commands; and

determining that the recorded sound data corresponds to the voice command listed among the pre-recorded voice commands.

15. A system comprising:

a microphone comprising:

a transducer to capture a combination of multiple inaudible signals received from multiple ultrasonic speakers; and

audio recording electronics coupled to the transducer, the audio recording electronics to process the combination of the multiple inaudible signals to generate digital output samples comprising recorded sound data, the recorded sound data including non-linearities from a frequency-shifted version of the combination of the multiple inaudible signals to within an audible frequency range; and

a processing device coupled to the microphone, wherein the processing device is to detect, within the recorded sound data, at least a portion of the non-linearities, wherein to detect the at least a portion of the non-linearities, the processing device is to:

compare the recorded sound data with expected patterns from an audible audio signal generated by human voice; and

detect non-linear variations within the recorded sound data as compared to the expected patterns, wherein the non-linear variations are detected as a result of the at least a portion of the non-linearities located within the recorded sound data corresponding to a strongest portion of the expected patterns and being in a sub-50 hertz (Hz) band; and

wherein, in response to the detection, the processing device is further to:

determine that the recorded sound data correspond to a voice command; and

suppress an action programmed for response to the voice command.

16. The system of claim 15 , wherein a width of a fundamental frequency combined with widths of corresponding harmonics of the fundamental frequency within the audible audio signal is a time-varying frequency, and to detect the at least a portion of the non-linearities, the processing device is further to:

determine a first energy variation over time within a frequency band that is between zero and the time-varying frequency; and

correlate the first energy variation with a second energy variation at a fundamental frequency in the recorded sound data that is greater than the frequency band.

17. The system of claim 15 , wherein, to detect the non-linear variations, the processing device is further to:

determine a first ratio of maximum and minimum amplitude of the audible audio signal;

determine a second ratio of maximum and minimum amplitude of the recorded sound data; and

detect, via comparison of the first ratio to the second ratio, that the at least a portion of the non-linearities are at positively-biased harmonics comprising an amplitude skew.

Assignments (2)
CONFIRMATORY LICENSE Recorded Jan 5, 2021
From: UNIVERSITY OF ILLINOIS, URBANA-CHAMPAIGN
To: NATIONAL SCIENCE FOUNDATION
Reel/Frame 054899/0050 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 24, 2020
From: ROY, NIRUPAM; CHOUDHURY, ROMIT ROY; HASSANIEH, HAITHAM AL
To: THE BOARD OF TRUSTEES OF THE UNIVERSITY OF ILLINOIS
Reel/Frame 052494/0628 →
Continuity (3)
Continuation 16163172 · Oct 17, 2018
Provisional Application 62575077 · Oct 20, 2017
Related Publication 20200251125A1 · Aug 6, 2020