IP Library › Granted Patent US 12,212,948
Granted Patent B2
US 12,212,948 · App. 18/455,585 · Granted Jan 28, 2025

Methods and systems for audio signal filtering

Inventors: Remi Samuel Audfray (San Francisco, CA); Jean-Marc Jot (Aptos, CA); Samuel Charles Dicker (San Francisco, CA)
Assignee: Magic Leap, Inc.
H04S1/007H04S1/005H04S7/304H04R5/033H04S2400/11H04S2420/01
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,212,948
App. No.
18/455,585
Granted
Jan 28, 2025
Kind
B2
Abstract

Systems and methods for rendering audio signals are disclosed. In some embodiments, a method may receive an input signal including a first portion and the second portion. A first processing stage comprising a first filter is applied to the first portion to generate a first filtered signal. A second processing stage comprising a second filter is applied to the first portion to generate a second filtered signal. A third processing stage comprising a third filter is applied to the second portion to generate a third filtered signal. A fourth processing stage comprising a fourth filter is applied to the second portion to generate a fourth filtered signal. A first output signal is determined based on a sum of the first filtered signal and the third filtered signal. A second output signal is determined based on a sum of the second filtered signal and the fourth filtered signal. The first output signal is presented to a first ear of a user of a virtual environment, and the second output signal is presented to the second ear of the user. The first portion of the input signal corresponds to a first location in the virtual environment, and the second portion of the input signal corresponds to a second location in the virtual environment.

Claims (68)

1. A method comprising:

receiving, at an amplitude panning stage, an audio input signal;

processing the audio input signal to generate a first processed signal and a second processed signal, wherein the processing comprises:

applying a first filter to the audio input signal,

aligning the first filter and a second filter, and

applying the second filter to the audio input signal;

adjusting, with respect to a first virtual speaker, a gain of the first processed signal to generate a first virtual speaker output;

adjusting, with respect to a second virtual speaker, based on the gain of the first processed signal, a gain of the second processed signal to generate a second virtual speaker output;

determining, based on a sum of the first virtual speaker output and the second virtual speaker output, a virtual speaker array output associated with a virtual speaker array comprising the first virtual speaker and the second virtual speaker; and

presenting the virtual speaker array output to one or more of a first speaker and a second speaker,

wherein the first speaker and the second speaker are associated with a wearable head device.

2. The method of claim 1 , wherein said aligning the first filter and the second filter comprises determining an inter-filter delay (IFD) and applying the IFD to one of the first filter and the second filter.

3. The method of claim 2 , further comprising modifying the IFD based on an anthropometric feature of a user of the wearable head device.

4. The method of claim 2 , further comprising adjusting the IFD based on an output of a sensor of the wearable head device.

5. The method of claim 1 , wherein said aligning the first filter and the second filter comprises:

determining a transfer function associated with the virtual speaker array; and

selecting an alignment based on the transfer function to minimize an amount of distortion.

6. The method of claim 5 , wherein said determining a transfer function associated with the virtual speaker array comprises empirically measuring the transfer function.

7. The method of claim 1 , wherein said aligning the first filter and the second filter comprises:

applying an audio signal to the virtual speaker array;

detecting a timbral artifact associated with the application of the audio signal to the virtual speaker array; and

selecting an alignment to reduce the timbral artifact.

8. A system comprising:

a wearable head device including a first speaker and a second speaker; and

one or more processors configured to perform a method comprising:

receiving, at an amplitude panning stage, an audio input signal;

processing the audio input signal to generate a first processed signal and a second processed signal, wherein the processing comprises:

applying a first filter to the audio input signal,

aligning the first filter and a second filter, and

applying the second filter to the audio input signal;

adjusting, with respect to a first virtual speaker, a gain of the first processed signal to generate a first virtual speaker output;

adjusting, with respect to a second virtual speaker, based on the gain of the first processed signal, a gain of the second processed signal to generate a second virtual speaker output;

determining, based on a sum of the first virtual speaker output and the second virtual speaker output, a virtual speaker array output associated with a virtual speaker array comprising the first virtual speaker and the second virtual speaker; and

presenting the virtual speaker array output to one or more of the first speaker and the second speaker.

9. The system of claim 8 , wherein said aligning the first filter and the second filter comprises determining an IFD and applying the IFD to one of the first filter and the second filter.

10. The system of claim 9 , wherein the method further comprises modifying the IFD based on an anthropometric feature of a user of the wearable head device.

11. The system of claim 9 , wherein:

the wearable head device comprises one or more sensors; and

the method further comprises adjusting the IFD based on an output of the one or more sensors.

12. The system of claim 8 , wherein said aligning the first filter and the second filter comprises:

determining a transfer function associated with the virtual speaker array; and

selecting an alignment based on the transfer function to minimize an amount of distortion.

13. The system of claim 12 , wherein said determining a transfer function associated with the virtual speaker array comprises empirically measuring the transfer function.

14. The system of claim 8 , wherein said aligning the first filter and the second filter comprises:

applying an audio signal to the virtual speaker array;

detecting a timbral artifact associated with the application of the audio signal to the virtual speaker array; and

selecting an alignment to reduce the timbral artifact.

15. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors, cause the one or more processors to perform a method comprising:

receiving, at an amplitude panning stage, an audio input signal;

processing the audio input signal to generate a first processed signal and a second processed signal, wherein the processing comprises:

applying a first filter to the audio input signal,

aligning the first filter and a second filter, and

applying the second filter to the audio input signal;

adjusting, with respect to a first virtual speaker, a gain of the first processed signal to generate a first virtual speaker output;

adjusting, with respect to a second virtual speaker, based on the gain of the first processed signal, a gain of the second processed signal to generate a second virtual speaker output;

determining, based on a sum of the first virtual speaker output and the second virtual speaker output, a virtual speaker array output associated with a virtual speaker array comprising the first virtual speaker and the second virtual speaker; and

presenting the virtual speaker array output to one or more of a first speaker and a second speaker,

wherein the first speaker and the second speaker are associated with a wearable head device.

16. The non-transitory computer-readable medium of claim 15 , wherein said aligning the first filter and the second filter comprises determining an IFD and applying the IFD to one of the first filter and the second filter.

17. The non-transitory computer-readable medium of claim 16 , wherein the method further comprises modifying the IFD based on an anthropometric feature of a user of the wearable head device.

18. The non-transitory computer-readable medium of claim 16 , wherein the method further comprises adjusting the IFD based on an output of a sensor of the wearable head device.

19. The non-transitory computer-readable medium of claim 15 , wherein said aligning the first filter and the second filter comprises:

determining a transfer function associated with the virtual speaker array; and

selecting an alignment based on the transfer function to minimize an amount of distortion.

20. The non-transitory computer-readable medium of claim 15 , wherein said aligning the first filter and the second filter comprises:

applying an audio signal to the virtual speaker array;

detecting a timbral artifact associated with the application of the audio signal to the virtual speaker array; and

selecting an alignment to reduce the timbral artifact.

Assignments (3)
SECURITY INTEREST Recorded Oct 31, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073422/0549 →
SECURITY INTEREST Recorded Oct 20, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073008/0696 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Aug 7, 2024
From: AUDFRAY, REMI SAMUEL; JOT, JEAN-MARC; DICKER, SAMUEL CHARLES
To: MAGIC LEAP, INC.
Reel/Frame 068214/0505 →
Continuity (6)
Continuation 17897036 · Aug 26, 2022
Continuation 16987079 · Aug 6, 2020
Continuation 16789201 · Feb 12, 2020
Continuation 16442258 · Jun 14, 2019
Provisional Application 62685258 · Jun 14, 2018
Related Publication 20230412999A1 · Dec 21, 2023
References Cited (82)
US 4852988A · Velez · 1989 [cited by applicant]
US 6433760B1 · Vaissie · 2002 [cited by applicant]
US 6491391B1 · Blum et al. · 2002 [cited by applicant]
US 6847336B1 · Lemelson · 2005 [cited by applicant]
US 6943754B2 · Aughey · 2005 [cited by applicant]
US 6977776B2 · Volkenandt et al. · 2005 [cited by applicant]
US 7347551B2 · Fergason et al. · 2008 [cited by applicant]
US 7488294B2 · Torch · 2009 [cited by applicant]
US 8235529B1 · Raffle · 2012 [cited by applicant]
US 8428269B1 · Brungart · 2013 [cited by applicant]
US 8488819B2 · Pang · 2013 [cited by applicant]
US 8611015B2 · Wheeler · 2013 [cited by applicant]
US 8638498B2 · Bohn et al. · 2014 [cited by applicant]
US 8696113B2 · Lewis · 2014 [cited by applicant]
US 8929589B2 · Publicover et al. · 2015 [cited by applicant]
US 9010929B2 · Lewis · 2015 [cited by applicant]
US 9226090B1 · Norris · 2015 [cited by applicant]
US 9274338B2 · Robbins et al. · 2016 [cited by applicant]
US 9292973B2 · Bar-zeev et al. · 2016 [cited by applicant]
US 9323325B2 · Perez et al. · 2016 [cited by applicant]
US 9538307B2 · Araki · 2017 [cited by applicant]
US 9720505B2 · Gribetz et al. · 2017 [cited by applicant]
US 10013053B2 · Cederlund et al. · 2018 [cited by applicant]
US 10025379B2 · Drake et al. · 2018 [cited by applicant]
US 10602292B2 · Audfray · 2020 [cited by applicant]
US 10779103B2 · Audfray et al. · 2020 [cited by applicant]
US 10979843B2 · Thagadur Shivappa · 2021 [cited by applicant]
US 11477592B2 · Audfray et al. · 2022 [cited by applicant]
US 11778400B2 · Audfray et al. · 2023 [cited by applicant]
US 20030030597A1 · Geist · 2003 [cited by applicant]
US 20060023158A1 · Howell et al. · 2006 [cited by applicant]
US 20060115090A1 · Kirkeby · 2006 [cited by examiner]
US 20070019812A1 · Kim · 2007 [cited by examiner]
US 20070172086A1 · Dickins · 2007 [cited by applicant]
US 20080056503A1 · Mcgrath · 2008 [cited by applicant]
US 20090214045A1 · Fukui · 2009 [cited by applicant]
US 20100303246A1 · Walsh · 2010 [cited by applicant]
US 20110211056A1 · Publicover et al. · 2011 [cited by applicant]
US 20110213664A1 · Osterhout · 2011 [cited by applicant]
US 20120021806A1 · Maltz · 2012 [cited by applicant]
US 20120215519A1 · Park · 2012 [cited by applicant]
US 20130077147A1 · Efimov · 2013 [cited by applicant]
US 20140195918A1 · Friedlander · 2014 [cited by applicant]
US 20150168731A1 · Robbins · 2015 [cited by applicant]
US 20160026253A1 · Bradski · 2016 [cited by examiner]
US 20160373877A1 · Laitinen · 2016 [cited by applicant]
US 20170094440A1 · Brown · 2017 [cited by applicant]
US 20170245082A1 · Boland · 2017 [cited by applicant]
US 20190116448A1 · Schmidt · 2019 [cited by applicant]
US 20220417686A1 · Audfray et al. · 2022 [cited by applicant]
CA 2316473A1 · 2001 [cited by applicant]
CA 2362895A1 · 2002 [cited by applicant]
CA 2388766A1 · 2003 [cited by applicant]
WO 2014090277A1 · 2014 [cited by applicant]
WO 2019241760A1 · 2019 [cited by applicant]
Grijalva et al. (2014). “Anthropometric-based customization of head-related transfer functions using Isomap in the horizontal plane”, 2014 IEEE, International Conference on Acoustic, Speech and Signal Processing (ICASSP… [cited by applicant]
International Preliminary Report dated Dec. 15, 2020, for PCT Application No. PCT/US2019/037390, filed Jun. 14, 2019, seven pages. [cited by applicant]
International Search Report mailed Sep. 3, 2019, for PCT Application No. PCT/US2019/037390, filed Jun. 14, 2019, three pages. [cited by applicant]
Jacob, R. “Eye Tracking in Advanced Interface Design”, Virtual Environments and Advanced Interface Design, Oxford University Press, Inc. (Jun. 1995). [cited by applicant]
Non-Final Office Action mailed Feb. 10, 2023, for U.S. Appl. No. 17/897,036, filed Aug. 26, 2022, nine pages. [cited by applicant]
Non-Final Office Action mailed Feb. 8, 2022, for U.S. Appl. No. 16/987,079, filed Aug. 6, 2020, eight pages. [cited by applicant]
Non-Final Office Action mailed Mar. 9, 2020, for U.S. Appl. No. 16/789,201, filed Feb. 12, 2020, eight pages. [cited by applicant]
Notice of Allowance (corrected) mailed Jan. 16, 2020, for U.S. Appl. No. 16/442,258, filed Jun. 14, 2019, four pages. [cited by applicant]
Notice of Allowance (corrected) mailed Mar. 6, 2020, for U.S. Appl. No. 16/442,258, filed Jun. 14, 2019, four pages. [cited by applicant]
Notice of Allowance mailed Jul. 22, 2022, for U.S. Appl. No. 16/987,079, filed Aug. 6, 2020, seven pages. [cited by applicant]
Notice of Allowance mailed Jul. 24, 2023, for U.S. Appl. No. 17/897,036, filed Aug. 26, 2022, seven pages. [cited by applicant]
Notice of Allowance malled May 6, 2020, for U.S. Appl. No. 16/789,201, filed Feb. 12, 2020, seven pages. [cited by applicant]
Notice of Allowance mailed Nov. 12, 2019, for U.S. Appl. No. 16/442,258, filed Jun. 14, 2019, nine pages. [cited by applicant]
Rolland, J. et al., “High-resolution inset head-mounted display”, Optical Society of America, vol. 37, No. 19, Applied Optics, (Jul. 1, 1998). [cited by applicant]
Tanriverdi, V. et al. (Apr. 2000). “Interacting With Eye Movements in Virtual Environments,” Department of Electrical Engineering and Computer Science, Tufts University, Medford, MA 02155, USA, Proceedings of the SIGCHI… [cited by applicant]
Yoshida, A. et al., “Design and Applications of a High Resolution Insert Head Mounted Display”, (Jun. 1994). [cited by applicant]
Carmigniani, J., et al., “Augmented Reality Technologies, Systems and Applications,” Multimedia Tools and Applications, 51(1):341-77 (Dec. 2010) (“Augmented Reality Technologies”). [cited by applicant]
Declaration of Dr. Stephen C. Thompson Under 37 C.F.R. § 1.132 including Appendix A (List of Materials Considered) and Appendix B (Curriculum Vitae) and Incorporated Claim Charts CC-1—CC-9. [cited by applicant]
Gerzon, M., “Stereo shuffling: new approach-old technique,” Studio Sound and Broadcast Engineering, pp. 122-130 (Jul. 1986) (“Gerzon”). [cited by applicant]
Mattana, A., “The history of binaural audio, Part 1: The first experiments, 1881-1939” accessed at https://www.linkedin.com/pulse/history-binaural-audio-part-1-anthony-mattana/ (Jul. 2017). [cited by applicant]
Milgram, P. and Kishino, F., “A Taxonomy of Mixed Reality Visual Displays,” IEICE Transactions on Information & Systems, E77-D(12):1321-29 (Dec. 1994) (“Taxonomy of Mixed Reality”). [cited by applicant]
Prosecution File History of U.S. Pat. No. 10,779,103 (“'103 File History”). [cited by applicant]
Zotkin, D.N., et al., “Rendering Localized Spatial Audio in a Virtual Auditory Space,” IEEE Transactions on Multimedia, 6(4): 553-564 (Aug. 2004) (“Zotkin-IEEE”). [cited by applicant]
Request for Ex Parte Reexamination of U.S. Pat. No. 10,779,103 B2 under 35 U.S.C. § 302 and 37 C.F.R. §1.510 with US1-US7, FP1, and NPL1-NPL7. [cited by applicant]
Prosecution File History of U.S. Pat. No. 10,602,292 (″292 File History). [cited by applicant]
Request for Ex Parte Reexamination of U.S. Pat. No. 10,602,292 B2 under 35 U.S.C. § 302 and 37 C.F.R. §1.510 with US1-US7, FP1, and NPL1-NPL8. (submitted in 4 parts). [cited by applicant]
Xie, Bosun, “Head-related Transfer Function and Virtual Auditory Display,” J. Ross Publishing, 2013. [cited by applicant]