Granted Patent B2

US 11,448,728 · App. 17/492,598 · Granted Sep 20, 2022

Inventors: Muhammed Zahid Ozturk (Beltsville, MD); Chenshu Wu (Hong Kong, CN); Beibei Wang (Clarksville, MD); K. J. Ray Liu (Potomac, MD); Oscar Chi-Lim Au (San Jose, CA)

Assignee: ORIGIN WIRELESS, INC.

G01S7/417G01S7/2883G01S13/003G01S13/34G01S13/46H04W4/021G01S2013/462

Abstract

Claims (132)

1. A system for sound sensing, comprising:

a transmitter configured to transmit a first wireless signal through a wireless channel of a venue;

a receiver configured to receive a second wireless signal through the wireless channel, wherein the second wireless signal comprises a reflection of the first wireless signal by at least one object in the venue; and

a processor configured for:

obtaining a time series of channel information (CI) of the wireless channel based on the second wireless signal,

constructing a radar spectrogram as a function of frequency,

determining a first magnitude of the radar spectrogram for positive frequencies,

determining a second magnitude of the radar spectrogram for negative frequencies,

calculating a sound metric based on a similarity between the first magnitude and the second magnitude,

calculating a time average of the sound metric,

subtracting the time average from the sound metric to generate sound metric values,

determining a presence of a vibrating object in the venue based on the time series of CI (TSCI) and the sound metric values,

extracting a sound signal from the TSCI, and

reconstructing at least one speech based on the sound signal.

2. The system of claim 1 , wherein:

each CI comprises a CIR; and

the first wireless signal is carried on a millimeter wave.

3. The system of claim 2 , wherein the vibrating object is at least one of:

a person actively giving the at least one speech;

a device actively outputting the at least one speech; or

an object passively vibrating according to the at least one speech.

4. The system of claim 3 , wherein obtaining the TSCI comprises:

applying a fast Fourier transform on frequency modulated carrier wave (FMCW) waveforms carried by the second wireless signal with respect to a short-time index.

5. The system of claim 4 , wherein the radar spectrogram is constructed based on:

applying a short-time Fourier transform on the TSCI, wherein the radar spectrogram is a function having: a frequency index representing Doppler frequency shift, a long-time index representing time frames, and a distance range index representing distance ranges from the receiver.

6. The system of claim 5 , wherein

the sound metric is calculated with respect to each time frame and each distance range from the receiver.

7. The system of claim 6 , wherein calculating the sound metric comprises:

calculating a distance function between the first magnitude and the second magnitude, with respect to each time frame and each distance range from the receiver,

wherein the distance function comprises at least one of: a cosine distance, L0 distance, L1 distance, L2 distance, or L_inf distance.

8. The system of claim 6 , wherein:

the time average of the sound metric is calculated for each distance range from the receiver; and

the sound metric values are generated at each time frame and each distance range from the receiver.

9. The system of claim 8 , wherein determining the presence of the vibrating object comprises:

determining the presence of the vibrating object by applying a predetermined threshold on the sound metric values; and

localizing the vibrating object.

10. The system of claim 8 , wherein determining the presence of the vibrating object comprises:

determining the presence of the vibrating object by detecting outliers in the sound metric values based on time frame variance or absolute deviation from mean; and

localizing the vibrating object.

11. The system of claim 8 , wherein determining the presence of the vibrating object comprises:

determining the presence of the vibrating object by extracting peaks of the radar spectrogram along the long-time index and/or the distance range index; and

localizing the vibrating object.

12. The system of claim 8 , wherein determining the presence of the vibrating object comprises:

determining the presence of the vibrating object by applying a constant false alarm rate (CFAR) detection rule on the radar spectrogram; and

localizing the vibrating object.

13. The system of claim 8 , wherein extracting the sound signal from the TSCI comprises:

filtering out background motion and low-speed motion to generate a filtered signal, wherein the filtering comprises at least one of:

applying a low-pass filter on the TSCI; or

removing frequency components in a proximity of zero frequency on the radar spectrogram.

14. The system of claim 13 , wherein extracting the sound signal from the TSCI comprises:

generating a projected signal based on the filtered signal, wherein the projected signal is generated by at least one of:

projecting the filtered signal onto a line in complex domain to represent complex values of the filtered signal with scalars; or

extracting a real part or an imaginary part of the filtered signal.

15. The system of claim 14 , wherein extracting the sound signal from the TSCI comprises:

recalculating the radio spectrogram based on the projected signal.

16. The system of claim 15 , wherein extracting the sound signal from the TSCI comprises:

extracting a positive magnitude spectrogram of the recalculated radar spectrogram for positive frequencies;

extracting a negative magnitude spectrogram of the recalculated radar spectrogram for negative frequencies; and

determining a sound spectrogram based on a maximum of the positive magnitude spectrogram and the negative magnitude spectrogram.

17. The system of claim 16 , wherein extracting the sound signal from the TSCI comprises:

applying an inverse short-time Fourier transform on the sound spectrogram to extract a real-valued sound signal.

18. The system of claim 17 , wherein extracting the sound signal from the TSCI comprises:

generating an enhanced sound signal from the real-valued sound signal based on receiver diversity, wherein

the second wireless signal is received by multiple antennas on the receiver,

the receiver diversity is obtained by selecting a best antenna on the receiver for each distance range from the receiver, and

the best antenna is predetermined based on calculating signal-to-noise ratio (SNR) for each antenna when there is no sound in the venue.

19. The system of claim 17 , wherein extracting the sound signal from the TSCI comprises:

generating an enhanced sound signal from the real-valued sound signal based on beamforming.

20. The system of claim 17 , wherein extracting the sound signal from the TSCI comprises:

generating an enhanced sound signal from the real-valued sound signal based on multipath diversity, wherein

the wireless channel includes a plurality of multipath components,

the multipath diversity is obtained by selecting a best multipath component of the wireless channel for each distance range from the receiver, and

the best multipath component is predetermined based on calculating signal-to-noise ratio (SNR) for each multipath component when there is no sound in the venue.

21. The system of claim 20 , wherein reconstructing the at least one speech based on the sound signal comprises:

applying a deep learning model on an input spectrogram to extract a magnitude spectrogram, wherein the input spectrogram is generated based on the enhanced sound signal; and

reconstructing time domain waveforms of the at least one speech based on the magnitude spectrogram and phase information of the input spectrogram.

22. The system of claim 21 , wherein the deep learning model is pre-trained to enhance input signals by de-noising and bandwidth expansion at the same time.

23. The system of claim 22 , wherein the processor is further configured for:

training the deep learning model based on a training data set, wherein the training data set includes:

frequency responses of different types of objects in the venue,

background noise in the venue, and

synthetic audio signals simulated based on open-source data.

24. The system of claim 22 , wherein the processor is further configured for:

training the deep learning model based on a training data set, wherein the training data set includes:

radio signals and audio signals collected at the same time in the venue, and

background noise in the venue.

25. The system of claim 22 , wherein:

the sound signal includes sound information from multiple sources at different locations in the venue; and

the processor is further configured for

separating, based on the radar spectrogram, an individual spectrogram of sound information from each of the multiple sources, and

reconstructing a speech based on each individual spectrogram.

26. The system of claim 25 , wherein the processor is further configured for

obtaining physical characteristics of each individual spectrogram; and

classifying, based on the physical characteristics of each individual spectrogram, each respective one of the multiple sources as a human or an inanimate source.

27. A wireless device of a sound sensing system, comprising:

a processor;

a memory communicatively coupled to the processor; and

a receiver communicatively coupled to the processor, wherein:

an additional wireless device of the sound sensing system is configured to transmit a first wireless signal through a wireless channel of a venue,

the receiver is configured to receive a second wireless signal through the wireless channel,

the second wireless signal comprises a reflection of the first wireless signal by at least one object in the venue, and

the processor is configured for:

obtaining a time series of channel information (CI) of the wireless channel based on the second wireless signal,

constructing a radar spectrogram as a function of frequency,

determining a first magnitude of the radar spectrogram for positive frequencies,

determining a second magnitude of the radar spectrogram for negative frequencies,

calculating a sound metric based on a similarity between the first magnitude and the second magnitude,

calculating a time average of the sound metric,

subtracting the time average from the sound metric to generate sound metric values,

determining a presence of a vibrating object in the venue based on the time series of CI (TSCI) and the sound metric values,

extracting a sound signal from the TSCI, and

reconstructing at least one speech based on the sound signal.

28. The wireless device of claim 27 , wherein obtaining the TSCI comprises:

applying a fast Fourier transform on frequency modulated carrier wave (FMCW) waveforms carried by the second wireless signal with respect to a short-time index; and

applying a short-time Fourier transform on the TSCI to construct the radar spectrogram, wherein the radar spectrogram is a function having: a frequency index representing Doppler frequency shift, a long-time index representing time frames, and a distance range index representing distance ranges from the receiver.

29. The wireless device of claim 28 , wherein:

the sound metric is calculated with respect to each time frame and each distance range from the receiver;

the time average of the sound metric is calculated for each distance range from the receiver; and

the sound metric values are generated at each time frame and each distance range from the receiver.

30. A method of a sound sensing system, comprising:

transmitting a first wireless signal through a wireless channel of a venue;

receiving a second wireless signal through the wireless channel, wherein the second wireless signal comprises a reflection of the first wireless signal by at least one object in the venue;

obtaining a time series of channel information (CI) of the wireless channel based on the second wireless signal;

constructing a radar spectrogram as a function of frequency;

determining a first magnitude of the radar spectrogram for positive frequencies;

determining a second magnitude of the radar spectrogram for negative frequencies;

calculating a sound metric based on a similarity between the first magnitude and the second magnitude;

calculating a time average of the sound metric;

subtracting the time average from the sound metric to generate sound metric values;

determining a presence of a vibrating object in the venue based on the time series of CI (TSCI) and the sound metric values;

extracting a sound signal from the TSCI; and

reconstructing at least one speech based on the sound signal.

Assignments (2)

Continuity (38)

Cited By (1)

US 12,351,176