IP Library Granted Patent US 11,197,118
Granted Patent B2
US 11,197,118 · App. 16/593,950 · Granted Dec 7, 2021

Interaural time difference crossfader for binaural audio rendering

Inventors: Samuel Charles Dicker (San Francisco, CA); Harsh Mayur Barbhaiya (San Jose, CA)
Assignee: Magic Leap, Inc.
H04S7/304H04R3/04H04R5/033H04R5/04H04S3/008H04S7/307H04S2400/01H04S2400/11H04S2400/13H04S2420/01
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,197,118
App. No.
16/593,950
Granted
Dec 7, 2021
Kind
B2
Abstract

Examples of the disclosure describe systems and methods for presenting an audio signal to a user of a wearable head device. According to an example method, a first input audio signal is received, the first input audio signal corresponding to a source location in a virtual environment presented to the user via the wearable head device. The first input audio signal is processed to generate a left output audio signal and a right output audio signal. The left output audio signal is presented to the left ear of the user via a left speaker associated with the wearable head device. The right output audio signal is presented to the right ear of the user via a right speaker associated with the wearable head device. Processing the first input audio signal comprises applying a delay process to the first input audio signal to generate a left audio signal and a right audio signal; adjusting a gain of the left audio signal; adjusting a gain of the right audio signal; applying a first head-related transfer function (HRTF) to the left audio signal to generate the left output audio signal; and applying a second HRTF to the right audio signal to generate the right output audio signal. Applying the delay process to the first input audio signal comprises applying an interaural time delay (ITD) to the first input audio signal, the ITD determined based on the source location.

Claims (91)

1. A method of presenting an audio signal to a user of a wearable head device, the method comprising:

receiving a first input audio signal, the first input audio signal corresponding to a first source location in a virtual environment presented to the user via the wearable head device, wherein the first source location corresponds to a first location of a virtual object in the virtual environment at a first time;

processing the first input audio signal to generate a left output audio signal and a right output audio signal, wherein processing the first input audio signal comprises:

applying a delay process to the first input audio signal to generate a left audio signal and a right audio signal;

adjusting a gain of the left audio signal;

adjusting a gain of the right audio signal;

applying a first head-related transfer function (HRTF) to the left audio signal to generate the left output audio signal; and

applying a second HRTF to the right audio signal to generate the right output audio signal;

presenting the left output audio signal to the left ear of the user via a left speaker associated with the wearable head device;

presenting the right output audio signal to the right ear of the user via a right speaker associated with the wearable head device, and

determining a second source location corresponding to a second location of the virtual object in the virtual environment at a second time, wherein:

the virtual object is at the first location in the virtual environment at the first time and the virtual object is at the second location in the virtual environment at the second time, and

the first location in the virtual environment and the second location in the virtual environment are different locations in the virtual environment relative to a position of the user;

determining the first ear delay comprising determining a prior first ear delay corresponding to the first time and a subsequent first ear delay corresponding to the second time and cross-fading between the prior first ear delay and the subsequent first ear delay to generate the first ear delay; and

determining the second ear delay comprising determining a prior second ear delay corresponding to the first time and a subsequent second ear delay corresponding to the second time and cross-fading between the prior second ear delay and the subsequent second ear delay to generate the second ear delay,

wherein:

applying the delay process to the first input audio signal comprises applying an interaural time delay (ITD) to the first input audio signal, the ITD determined based on the first source location, the first ear delay, and the second ear delay.

2. The method of claim 1 , wherein the first ear delay is zero.

3. The method of claim 1 , wherein applying the delay process comprises applying a filter to the first input audio signal.

4. The method of claim 1 , further comprising applying a filter to the left audio signal.

5. The method of claim 1 , further comprising applying a filter to the right audio signal.

6. The method of claim 1 , wherein applying the delay process comprises transitioning from a first delay module at the first time to a second delay module at the second time, the second delay module different from the first delay module.

7. The method of claim 6 , wherein the first delay module is associated with applying a first one or more filters to a first one or more of the first input audio signal, the left audio signal, and the right audio signal, and the second delay module is associated with applying a second one or more filters to a second one or more of the first input audio signal, the left audio signal, and the right audio signal.

8. The method of claim 1 , wherein the first ear corresponds to the user's left ear and the second ear corresponds to the user's right ear.

9. The method of claim 1 , wherein the first ear corresponds to the user's right ear and the second ear corresponds to the user's left ear.

10. The method of claim 1 , wherein the first source location is closer to the first ear than to the second ear and the second source location is closer to the second ear than to the first ear.

11. The method of claim 1 , wherein the second source location is closer to the first ear than to the second ear and the first source location is closer to the second ear than to the first ear.

12. The method of claim 1 , wherein the first source location is closer than the second source location to the first ear, the first source location is closer to the first ear than to the second ear, and the second source location is closer to the first ear than to the second ear.

13. The method of claim 1 , wherein the second source location is closer than the first source location to the first ear, the first source location is closer to the first ear than to the second ear, and the second source location is closer to the first ear than to the second ear.

14. A system comprising:

a wearable head device;

a left speaker associated with the wearable head device;

a right speaker associated with the wearable head device;

one or more processors configured to execute a method comprising:

receiving a first input audio signal, the first input audio signal corresponding to a first source location in a virtual environment presented to a user via the wearable head device, wherein the first source location corresponds to a first location of a virtual object in the virtual environment at a first time;

processing the first input audio signal to generate a left output audio signal and a right output audio signal, wherein processing the first input audio signal comprises:

applying a delay process to the first input audio signal to generate a left audio signal and a right audio signal;

adjusting a gain of the left audio signal;

adjusting a gain of the right audio signal;

applying a first head-related transfer function (HRTF) to the left audio signal to generate the left output audio signal; and

applying a second HRTF to the right audio signal to generate the right output audio signal;

presenting the left output audio signal to the left ear of the user via the left speaker;

presenting the right output audio signal to the right ear of the user via the right speaker,

determining a second source location corresponding to a second location of the virtual object in the virtual environment at a second time, wherein:

the virtual object is at the first location in the virtual environment at the first time and the virtual object is at the second location in the virtual environment at the second time, and

the first location in the virtual environment and the second location in the virtual environment are different locations in the virtual environment relative to a position of the user;

determining the first ear delay comprising determining a prior first ear delay corresponding to the first time and a subsequent first ear delay corresponding to the second time and cross-fading between the prior first ear delay and the subsequent first ear delay to generate the first ear delay; and

determining the second ear delay comprising determining a prior second ear delay corresponding to the first time and a subsequent second ear delay corresponding to the second time and cross-fading between the prior second ear delay and the subsequent second ear delay to generate the second ear delay,

wherein:

applying the delay process to the first input audio signal comprises applying an interaural time delay (ITD) to the first input audio signal, the ITD determined based on the first source location, the first ear delay, and the second ear delay.

15. The system of claim 14 , wherein the first ear delay is zero.

16. The system of claim 14 , wherein applying the delay process comprises applying a filter to the first input audio signal.

17. The system of claim 14 , the method further comprising applying a filter to the left audio signal.

18. The system of claim 14 , the method further comprising applying a filter to the right audio signal.

19. The system of claim 14 , wherein applying the delay process comprises transitioning from a first delay module at the first time to a second delay module at the second time, the second delay module different from the first delay module.

20. The system of claim 19 , wherein the first delay module is associated with applying a first one or more filters to a first one or more of the first input audio signal, the left audio signal, and the right audio signal, and the second delay module is associated with applying a second one or more filters to a second one or more of the first input audio signal, the left audio signal, and the right audio signal.

21. The system of claim 14 , wherein the first ear corresponds to the user's left ear and the second ear corresponds to the user's right ear.

22. The system of claim 14 , wherein the first ear corresponds to the user's right ear and the second ear corresponds to the user's left ear.

23. The system of claim 14 , wherein the first source location is closer to the first ear than to the second ear and the second source location is closer to the second ear than to the first ear.

24. The system of claim 14 , wherein the second source location is closer to the first ear than to the second ear and the first source location is closer to the second ear than to the first ear.

25. The system of claim 14 , wherein the first source location is closer than the second source location to the first ear, the first source location is closer to the first ear than to the second ear, and the second source location is closer to the first ear than to the second ear.

26. The system of claim 14 , wherein the second source location is closer than the first source location to the first ear, the first source location is closer to the first ear than to the second ear, and the second source location is closer to the first ear than to the second ear.

27. A non-transitory computer-readable medium containing instructions which when executed by one or more processors cause the one or more processors to perform a method of presenting an audio signal to a user of a wearable head device, the method comprising:

receiving a first input audio signal, the first input audio signal corresponding to a first source location in a virtual environment presented to the user via the wearable head device, wherein the first source location corresponds to a first location of a virtual object in the virtual environment at a first time;

processing the first input audio signal to generate a left output audio signal and a right output audio signal, wherein processing the first input audio signal comprises:

applying a delay process to the first input audio signal to generate a left audio signal and a right audio signal;

adjusting a gain of the left audio signal;

adjusting a gain of the right audio signal;

applying a first head-related transfer function (HRTF) to the left audio signal to generate the left output audio signal; and

applying a second HRTF to the right audio signal to generate the right output audio signal;

presenting the left output audio signal to the left ear of the user via a left speaker associated with the wearable head device;

presenting the right output audio signal to the right ear of the user via a right speaker associated with the wearable head device,

determining a second source location corresponding to a second location of the virtual object in the virtual environment at a second time, wherein:

the virtual object is at the first location in the virtual environment at the first time and the virtual object is at the second location in the virtual environment at the second time, and

the first location in the virtual environment and the second location in the virtual environment are different locations in the virtual environment relative to a position of the user;

determining the first ear delay comprising determining a prior first ear delay corresponding to the first time and a subsequent first ear delay corresponding to the second time and cross-fading between the prior first ear delay and the subsequent first ear delay to generate the first ear delay; and

determining the second ear delay comprising determining a prior second ear delay corresponding to the first time and a subsequent second ear delay corresponding to the second time and cross-fading between the prior second ear delay and the subsequent second ear delay to generate the second ear delay,

wherein:

applying the delay process to the first input audio signal comprises applying an interaural time delay (ITD) to the first input audio signal, the ITD determined based on the first source location, the first ear delay, and the second ear delay.

28. The non-transitory computer-readable medium of claim 27 , wherein the first ear delay is zero.

29. The non-transitory computer-readable medium of claim 27 , wherein applying the delay process comprises applying a filter to the first input audio signal.

30. The non-transitory computer-readable medium of claim 27 , the method further comprising applying a filter to the left audio signal.

31. The non-transitory computer-readable medium of claim 27 , the method further comprising applying a filter to the right audio signal.

32. The non-transitory computer-readable medium of claim 27 , wherein applying the delay process comprises transitioning from a first delay module at the first time to a second delay module at the second time, the second delay module different from the first delay module.

33. The non-transitory computer-readable medium of claim 32 , wherein the first delay module is associated with applying a first one or more filters to a first one or more of the first input audio signal, the left audio signal, and the right audio signal, and the second delay module is associated with applying a second one or more filters to a second one or more of the first input audio signal, the left audio signal, and the right audio signal.

34. The non-transitory computer-readable medium of claim 27 , wherein the first ear corresponds to the user's left ear and the second ear corresponds to the user's right ear.

35. The non-transitory computer-readable medium of claim 27 , wherein the first ear corresponds to the user's right ear and the second ear corresponds to the user's left ear.

36. The non-transitory computer-readable medium of claim 27 , wherein the first source location is closer to the first ear than to the second ear and the second source location is closer to the second ear than to the first ear.

37. The non-transitory computer-readable medium of claim 27 , wherein the second source location is closer to the first ear than to the second ear and the first source location is closer to the second ear than to the first ear.

38. The non-transitory computer-readable medium of claim 27 , wherein the first source location is closer than the second source location to the first ear, the first source location is closer to the first ear than to the second ear, and the second source location is closer to the first ear than to the second ear.

39. The non-transitory computer-readable medium of claim 27 , wherein the second source location is closer than the first source location to the first ear, the first source location is closer to the first ear than to the second ear, and the second source location is closer to the first ear than to the second ear.

Assignments (4)
SECURITY INTEREST Recorded Oct 15, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073109/0476 →
SECURITY INTEREST Recorded May 24, 2022
From: MOLECULAR IMPRINTS, INC.; MENTOR ACQUISITION ONE, LLC; MAGIC LEAP, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 060338/0665 →
SECURITY INTEREST Recorded May 21, 2020
From: MAGIC LEAP, INC.; MOLECULAR IMPRINTS, INC.; MENTOR ACQUISITION ONE, LLC
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 052729/0791 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 6, 2020
From: DICKER, SAMUEL CHARLES; BARBHAIYA, HARSH MAYUR
To: MAGIC LEAP, INC.
Reel/Frame 051427/0367 →