IP Library Granted Patent US 11,778,366
Granted Patent B2
US 11,778,366 · App. 17/521,293 · Granted Oct 3, 2023

Gaze-guided audio

Inventors: Sebastian Sztuk (Menlo Park, CA); Salvael Ortega Estrada (Redmond, WA)
Assignee: Meta Platforms Technologies, LLC
H04R1/222G06F3/011H04R1/08H04R1/10H04R2499/15
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,778,366
App. No.
17/521,293
Granted
Oct 3, 2023
Kind
B2
Abstract

A gaze direction of a user is determined by an eye-tracking system of a head mounted device. Audio data generated by at least one microphone is captured. Gaze-guided audio is generated from the audio data based on the gaze direction of the user.

Claims (39)

1. A head mounted device comprising:

an eye-tracking system including one or more sensors configured to determine a gaze direction of an eye in an eyebox region of the head mounted device;

a first microphone configured to record first audio data from first sound waves, wherein the first microphone is oriented to receive the first sound waves originating from a first audio zone of an external environment of the head mounted device;

a second microphone configured to record second audio data from second sound waves, wherein the second microphone is oriented to receive the second sound waves originating from a second audio zone of an external environment of the head mounted device;

and

processing logic configured to:

receive the gaze direction from the eye-tracking system; and

select a primary microphone between the first microphone and the second microphone to generate gaze-guided audio, wherein the primary microphone is selected based on the gaze direction received from the eye-tracking system, wherein the primary microphone is selected based in part on a gaze vector representative of the gaze direction being closest to a middle of a selected audio zone of the primary microphone.

2. The head mounted device of claim 1 , wherein the processing logic is further configured to:

receive a subsequent-gaze direction from the eye-tracking system; and

select a subsequent-primary microphone that is different from the primary microphone when a subsequent-gaze vector representative of the subsequent-gaze direction becomes closer to a subsequent-selected audio zone of the subsequent-primary microphone that is different from an audio zone of the primary microphone.

3. The head mounted device of claim 2 , wherein the first microphone is the primary microphone and the second microphone is the subsequent-primary microphone.

4. The head mounted device of claim 1 further comprising audio transmission devices, wherein generating the gaze-guided audio includes:

generating amplified audio from selected audio data, wherein the first audio data is the selected audio data when the first microphone is selected as the primary microphone, and wherein the second audio data is the selected audio data when the second microphone is selected as the primary microphone; and

driving the amplified audio onto the audio transmission devices as the gaze-guided audio.

5. The head mounted device of claim 4 , wherein generating the amplified audio from the selected audio data includes subtracting unselected audio data from the amplified audio.

6. The head mounted device of claim 1 further comprising audio transmission devices, wherein generating the gaze-guided audio includes:

generating amplified audio from selected audio data, wherein the first audio data is the selected audio data when the first microphone is selected as the primary microphone, and wherein the second audio data is the selected audio data when the second microphone is selected as the primary microphone; and

storing the amplified audio to memory as the gaze-guided audio.

7. The head mounted device of claim 6 , wherein the amplified audio stored to memory is stored as an audio portion of a video file.

8. A head mounted device comprising:

an eye-tracking system including one or more sensors configured to determine a gaze direction of an eye in an eyebox region of the head mounted device;

a first microphone configured to record first audio data from first sound waves, wherein the first microphone is oriented to receive the first sound waves originating from a first audio zone of an external environment of the head mounted device;

a second microphone configured to record second audio data from second sound waves, wherein the second microphone is oriented to receive the second sound waves originating from a second audio zone of an external environment of the head mounted device; and

processing logic configured to:

receive the gaze direction from the eye-tracking system;

select a primary microphone between the first microphone and the second microphone to generate gaze-guided audio, wherein the primary microphone is selected based on the gaze direction received from the eye-tracking system;

receive a subsequent-gaze direction from the eye-tracking system; and

select a subsequent-primary microphone that is different from the primary microphone when a subsequent-gaze vector representative of the subsequent-gaze direction becomes closer to a subsequent-selected audio zone of the subsequent-primary microphone that is different from an audio zone of the primary microphone.

9. The head mounted device of claim 8 , wherein the first microphone is the primary microphone and the second microphone is the subsequent-primary microphone.

10. The head mounted device of claim 8 , wherein the primary microphone is selected based in part on a gaze vector representative of the gaze direction being closest to a middle of a selected audio zone of the primary microphone.

11. The head mounted device of claim 8 further comprising:

speakers, wherein the processing logic is further configured to drive the gaze-guided audio onto the speakers.

12. The head mounted device of claim 8 further comprising:

memory, wherein the processing logic is further configured to store the gaze-guided audio to the memory as an audio portion of a video file that was captured by the head mounted device contemporaneously with the audio data.

13. The head mounted device of claim 8 , wherein generating the gaze-guided audio includes amplifying audio from one or more microphones that are oriented to receive sound waves from audio zones that correspond with a gaze vector representative of the gaze direction.

14. The head mounted device of claim 8 , wherein generating the gaze-guided audio includes:

generating amplified audio by amplifying near-audio data received from a nearest microphone in a plurality of microphones; and

subtracting remaining audio from the amplified audio, wherein the remaining audio is received from remaining microphones in the plurality of microphones that are not included in the nearest microphones.

Assignments (2)
CHANGE OF NAME Recorded Jun 1, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060246/0845 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 6, 2022
From: SZTUK, SEBASTIAN; ESTRADA, SALVAEL ORTEGA
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 058579/0934 →
Continuity (1)
Related Publication 20230148241A1 · May 11, 2023