IP Library Granted Patent US 11,245,984
Granted Patent B1
US 11,245,984 · App. 16/945,388 · Granted Feb 8, 2022

Audio system using individualized sound profiles

Inventors: William Owen Brimijoin, II (Kirkland, WA); Nils Thomas Fritiof Lunner (Redmond, WA); Philip Robinson (Seattle, WA); Ravish Mehra (Tacoma, WA)
Assignee: Facebook Technologies, LLC
H04R3/04G02B27/0172G06F3/013G06F3/017G06T19/006H04R1/1016H04R5/033H04R5/04H04S1/005H04S7/304G02B2027/0178
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,245,984
App. No.
16/945,388
Granted
Feb 8, 2022
Kind
B1
Abstract

A system for presenting audio content to a user. The system comprises one or more microphones coupled to a frame of a headset. The one or more microphones capture sound from a local area. The system further comprises an audio controller integrated into the headset and communicatively coupled to an in-ear device worn by a user. The audio controller identifies one or more sound sources in the local area based on the captured sound. The audio controller further determines a target sound source of the one or more sound sources and determines one or more filters to apply to a sound signal associated with the target sound source in the captured sound. The audio controller further generates an augmented sound signal by applying the one or more filters to the sound signal and provides the augmented sound signal to the in-ear device for presentation to a user.

Claims (82)

1. A method comprising:

capturing, by one or more microphones, sound from a local area;

identifying one or more sound sources in the local area based on the captured sound;

determining a target sound source of the one or more sound sources;

determining one or more filters to apply to a sound signal associated with the target sound source in the captured sound, wherein the one or more filters are based in part on a sound profile of the user, the sound profile including at least a user audiogram;

generating an augmented sound signal by applying the one or more filters to the sound signal;

providing the augmented sound signal to a speaker assembly for presentation to a user; and

updating the sound profile of the user based on user feedback.

2. The method of claim 1 , further comprising:

detecting that the target sound source is within a threshold distance of a different sound source in the local area; and

updating at least one of the one or more filters to increase spatial separation between the target sound source and the different sound source to be at least a separation distance that is greater than the threshold distance,

wherein the augmented sound signal presented to the user is such that sound appears to originate from the target sound source and the sound from the different sound source appears to originate from a location in the local area that is at least the separation distance from the target sound source.

3. The method of claim 1 , wherein the one or more filters is based in part on one or more of: a location of the user in the local area, a location of each of the one or more sound sources in the local area, one or more acoustic parameters of the local area, and types of sound sources in the local area.

4. The method of claim 1 , wherein determining the one or more filters to apply to the sound signal associated with the target sound source in the captured sound, comprises:

selecting a wide dynamic range compression filter that corresponds to the target sound source from a plurality of wide dynamic range compression filters, wherein each of the plurality of wide dynamic range compression filters correspond to a different respective sound source of the plurality of sound sources.

5. The method of claim 1 , wherein providing the augmented sound signal to the speaker assembly for presentation to the user, comprises:

providing the augmented sound signal to the speaker assembly of an in-ear device worn by the user.

6. The method of claim 5 , wherein capturing, by one or more microphones, sound from the local area, comprises:

capturing, by one or more microphones coupled to a frame of a headset, sound from the local area, the headset is separate from and communicatively coupled to the in-ear device.

7. The method of claim 5 , further comprising:

capturing, by one or more microphones, a leakage signal from the in-ear device;

determining a mitigation signal to substantially prevent the leakage signal, the mitigation signal is out-of-phase to the leakage signal; and

providing the mitigation signal as an output signal to a separate speaker assembly coupled to a headset.

8. The method of claim 1 , wherein determining the target sound source is based in part on a gaze of the user.

9. The method of claim 1 , wherein determining the target sound source is based in part on a verbal cue provided by the user.

10. The method of claim 1 , wherein determining the target sound source comprises:

capturing an image of a gesture performed by the user;

determining that the gesture corresponds to a selection of a sound source; and

selecting the sound source as the target sound source.

11. A system comprising:

one or more microphones coupled to a frame of a headset, the one or more microphones configured to capture sound from a local area;

an audio controller integrated into the headset and communicatively coupled to an in-ear device worn by a user, the audio controller configured to:

identify one or more sound sources in the local area based on the captured sound;

determine a target sound source of the one or more sound sources;

determine one or more filters to apply to a sound signal associated with the target sound source in the captured sound, wherein the one or more filters are based in part on a sound profile of the user, the sound profile including at least a user audiogram;

generate an augmented sound signal by applying the one or more filters to the sound signal;

provide the augmented sound signal to the in-ear device for presentation to a user; and

update the sound profile of the user based on user feedback.

12. The system of claim 11 , wherein the audio controller is further configured to:

detect that the target sound source is within a threshold distance of a different sound source in the local area; and

update at least one of the one or more filters to increase spatial separation between the target sound source and the different sound source to be at least a separation distance that is greater than the threshold distance,

wherein the augmented sound signal presented to the user is such that sound appears to originate from the target sound source and the sound from the different sound source appears to originate from a location in the local area that is at least the separation distance from the target sound source.

13. The system of claim 11 , wherein determining the one or more to apply to the sound signal associated with the target sound source in the captured sound comprises selecting a wide dynamic range compression filter that corresponds to the target sound source from a plurality of wide dynamic range compression filters, wherein each of the plurality of wide dynamic range compression filters correspond to a different respective sound source of the plurality of sound sources.

14. The system of claim 11 , wherein the one or more microphones capture a leakage signal from the in-ear device, and the audio controller is further configured to:

determine a mitigation signal to substantially prevent the leakage signal, the mitigation signal is out-of-phase to the leakage signal; and

provide the mitigation signal as an output signal to a separate speaker assembly coupled to the headset.

15. The system of claim 11 , wherein the one or more filters is based in part on one or more of: a location of the user in the local area, a location of each of the one or more sound sources in the local area, one or more acoustic parameters of the local area, and types of sound sources in the local area.

16. The system of claim 11 , wherein the target sound source is determined by the audio controller based in part on a gaze of the user.

17. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

capturing, by one or more microphones, sound from a local area;

identifying one or more sound sources in the local area based on the captured sound;

determining a target sound source of the one or more sound sources;

determining one or more filters to apply to a sound signal associated with the target sound source in the captured sound, wherein the one or more filters are based in part on a sound profile of the user, the sound profile including at least a user audiogram;

generating an augmented sound signal by applying the one or more filters to the sound signal;

providing the augmented sound signal to a speaker assembly for presentation to a user; and

updating the sound profile of the user based on user feedback.

18. The non-transitory computer-readable medium of claim 17 , the instructions further cause the one or more processors to perform operations further comprising:

detecting that the target sound source is within a threshold distance of a different sound source in the local area; and

updating at least one of the one or more filters to increase spatial separation between the target sound source and the different sound source to be at least a separation distance that is greater than the threshold distance,

wherein the augmented sound signal presented to the user is such that sound appears to originate from the target sound source and the sound from the different sound source appears to originate from a location in the local area that is at least the separation distance from the target sound source.

19. The non-transitory computer-readable medium of claim 17 , wherein determining the one or more filters to apply to the sound signal associated with the target sound source in the captured sound, comprises:

selecting a wide dynamic range compression filter that corresponds to the target sound source from a plurality of wide dynamic range compression filters, wherein each of the plurality of wide dynamic range compression filters correspond to a different respective sound source of the plurality of sound sources.

20. A method comprising:

capturing, by one or more microphones, sound from a local area;

identifying one or more sound sources in the local area based on the captured sound;

determining a target sound source of the one or more sound sources;

determining one or more filters to apply to a sound signal associated with the target sound source in the captured sound;

generating an augmented sound signal by applying the one or more filters to the sound signal;

providing the augmented sound signal to a speaker assembly for presentation to a user;

detecting that the target sound source is within a threshold distance of a different sound source in the local area; and

updating at least one of the one or more filters to increase spatial separation between the target sound source and the different sound source to be at least a separation distance that is greater than the threshold distance,

wherein the augmented sound signal presented to the user is such that sound appears to originate from the target sound source and the sound from the different sound source appears to originate from a location in the local area that is at least the separation distance from the target sound source.

21. A method comprising:

capturing, by one or more microphones coupled to a frame of a headset, sound from a local area, the headset is separate from and communicatively coupled to an in-ear device;

identifying one or more sound sources in the local area based on the captured sound;

determining a target sound source of the one or more sound sources;

determining one or more filters to apply to a sound signal associated with the target sound source in the captured sound;

generating an augmented sound signal by applying the one or more filters to the sound signal;

providing the augmented sound signal to a speaker assembly of the in-ear device worn by the user;

capturing, by one or more microphones, a leakage signal from the in-ear device;

determining a mitigation signal to substantially prevent the leakage signal, the mitigation signal is out-of-phase to the leakage signal; and

providing the mitigation signal as an output signal to a separate speaker assembly coupled to a headset.

Assignments (3)
CHANGE OF NAME Recorded Jun 8, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060315/0224 →
CORRECTIVE ASSIGNMENT TO CORRECT THE SPELLING OF THE SECOND ASSIGNOR'S NAME LUNNER, THOMAS PREVIOUSLY RECORDED ON REEL 053417 FRAME 0643. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Nov 10, 2020
From: BRIMIJOIN, WILLIAM OWEN, II; LUNNER, NILS THOMAS FRITIOF; ROBINSON, PHILIP; MEHRA, RAVISH
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 054374/0570 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 6, 2020
From: BRIMIJOIN, WILLIAM OWEN, II; LUNNER, THOMAS; ROBINSON, PHILIP; MEHRA, RAVISH
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 053417/0643 →
Continuity (1)
Provisional Application 63052016 · Jul 15, 2020
Cited By (1)
US 12,219,331