IP Library Granted Patent US 12,039,991
Granted Patent B1
US 12,039,991 · App. 17/532,720 · Granted Jul 16, 2024

Distributed speech enhancement using generalized eigenvalue decomposition

Inventors: Vinay Kumar Kothapally (Dallas, TX); Jacob Ryan Donley (Kirkland, WA); Buye Xu (Sammamish, WA)
Assignee: META PLATFORMS TECHNOLOGIES, LLC
G10L21/0232G10L25/51H04R1/406H04R3/005G10L2021/02166
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,039,991
App. No.
17/532,720
Granted
Jul 16, 2024
Kind
B1
Abstract

An artificial reality headset enhances audio signals from a target sound source using information from other devices in the local area. A primary headset broadcasts a location of a target sound source to secondary headsets in a local area. The secondary headsets transmit audio signals to the primary headset to enhance the audio content presented by the primary headset to a user. The secondary headset may select an array transfer function for the location of the target sound source. The secondary headsets correlate known transfer functions in the target direction with estimated transfer functions. The secondary headset may perform beamforming on the target sound source and transmit the output audio signal to the primary headset. In some embodiments, the secondary headset may transmit the array transfer function and a raw audio signal to the primary headset. The primary headset generates audio content based on the received audio signal.

Claims (48)

1. A method comprising:

receiving, at a first device, an acoustic signal from a target sound source physically located in a same environment as the first device;

determining a location of the target sound source based on the acoustic signal;

transmitting the location of the target sound source to a second device, wherein:

the second device selects a first array transfer function for the target sound source from a list of array transfer functions generated based on sound sources detected by the second device,

the sound sources detected by the second device comprise the target sound source and one or more noise sources,

the first array transfer function is selected based on being more closely associated with the location of the target sound source than other array transfer functions in the list of array transfer functions, and

the second device generates a first audio signal for the target sound source using the first array transfer function;

receiving, from the second device, the first audio signal for the target sound source; and

presenting, by the first device and based on the first audio signal, audio content for the target sound source.

2. The method of claim 1 , further comprising receiving, by the first device, the first array transfer function from the second device.

3. The method of claim 1 , further comprising generating, by the first device, a second audio signal for the target sound source.

4. The method of claim 3 , further comprising selecting, by the first device, the first audio signal or the second audio signal based on a signal to noise ratio (SNR) of the first audio signal and a SNR of the second audio signal.

5. The method of claim 3 , further comprising determining, based on comparing the first audio signal and the second audio signal, whether the first audio signal corresponds to the target sound source.

6. The method of claim 3 , further comprising:

receiving, by the first device, a third audio signal from a third device; and

correlating, by the first device, the first audio signal, the second audio signal, and the third audio signal.

7. The method of claim 1 , wherein the second device generates the list of array transfer functions based on a generalized eigenvalue decomposition.

8. A non-transitory computer-readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to perform operations comprising:

receiving, by a processor of a first device, an acoustic signal from a target sound source physically located in a same environment as the first device;

determining, by the processor, a location of the target sound source;

transmitting, by the processor, the location of the target sound source to a second device, wherein:

the second device selects a first array transfer function for the target sound source from a list of array transfer functions generated based on sound sources detected by the second device,

the sound sources detected by the second device comprise the target sound source and one or more noise sources,

the first array transfer function is selected based on being more closely associated with the location of the target sound source than other array transfer functions in the list of array transfer functions, and

the second device generates a first audio signal for the target sound source using the first array transfer function;

receiving, by the processor and from the second device, the first audio signal for the target sound source; and

presenting, by the processor and based on the first audio signal, audio content for the target sound source.

9. The non-transitory computer-readable storage medium of claim 8 , wherein the instructions further cause the processor to receive the first array transfer function from the second device.

10. The non-transitory computer-readable storage medium of claim 8 , wherein the instructions further cause the processor to perform operations comprising generating, by the processor, a second audio signal for the target sound source.

11. The non-transitory computer-readable storage medium of claim 10 , wherein the instructions further cause the processor to perform operations comprising selecting, by the processor, the first audio signal or the second audio signal based on a signal to noise ratio (SNR) of the first audio signal and a SNR of the second audio signal.

12. The non-transitory computer-readable storage medium of claim 10 , wherein the instructions further cause the processor to perform operations comprising determining, by the processor and based on comparing the first audio signal and the second audio signal, whether the first audio signal corresponds to the target sound source.

13. The non-transitory computer-readable storage medium of claim 10 , wherein the instructions further cause the processor to perform operations comprising:

receiving, by the processor, a third audio signal from a third device; and

correlating, by the processor, the first audio signal, the second audio signal, and the third audio signal.

14. The non-transitory computer-readable storage medium of claim 8 , wherein the second device generates the list of array transfer functions based on a generalized eigenvalue decomposition.

15. A method comprising:

receiving, at a first device, a location of a target sound source from a second device, wherein the target sound source is physically located in a same environment as the first device and the second device;

retrieving, from a stored set of array transfer functions, an estimated array transfer function for the location of the target sound source, wherein the array transfer functions in the stored set of array transfer functions are associated with different locations;

performing, by the first device, a generalized eigenvalue decomposition to generate a list of array transfer functions for sound sources detected by the first device, wherein the sound sources detected by the first device comprise the target sound source and one or more noise sources;

selecting a first array transfer function for the target sound source from the list of array transfer functions, wherein selecting the first array transfer function comprises comparing the list of array transfer functions to the estimated array transfer function to determine that the first array transfer function is more closely associated with the location of the target sound source than other array transfer functions in the list of array transfer functions;

generating, using the first array transfer function, an audio signal for the target sound source; and

transmitting, by the first device, the audio signal to the second device.

16. The method of claim 15 , further comprising transmitting the first array transfer function to the second device.

17. The method of claim 15 , wherein comparing the list of array transfer functions to the estimated array transfer function comprises determining, for each array transfer function in the list of array transfer functions, a corresponding degree of correlation between the array transfer function and the estimated array transfer function, and wherein the first array transfer function is selected based on having the highest correlation among the list of array transfer functions.

18. The method of claim 15 , further comprising forming a beam at the target sound source using the first array transfer function.

19. The method of claim 15 , wherein the second device generates audio content based on the audio signal.

20. The method of claim 15 , wherein the target sound source is not a dominant sound source for the first device, the dominant sound source for the first device being one of the one or more noise sources, and wherein the dominant sound source for the first device is different than a dominant sound source for the second device.

Assignments (2)
CHANGE OF NAME Recorded Jun 8, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060314/0965 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Dec 2, 2021
From: KOTHAPALLY, VINAY KUMAR; DONLEY, JACOB RYAN; XU, BUYE
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 058274/0493 →
Continuity (1)
Provisional Application 63167748 · Mar 30, 2021