IP Library Granted Patent US 10,616,705
Granted Patent B2
US 10,616,705 · App. 16/163,529 · Granted Apr 7, 2020

Mixed reality spatial audio

Inventors: Brian Lloyd Schmidt (Bellevue, WA); Jehangir Tajik (Fort Lauderdale, FL); Jean-Marc Jot (Aptos, CA)
Assignee: Magic Leap, Inc.
H04S7/304G06F3/01H04S3/008H04S7/306H04R5/033H04S7/301H04S2400/01H04S2400/11H04S2400/15H04S2420/01
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,616,705
App. No.
16/163,529
Granted
Apr 7, 2020
Kind
B2
Abstract

A method of presenting an audio signal to a user of a mixed reality environment is disclosed. According to examples of the method, an audio event associated with the mixed reality environment is detected. The audio event is associated with a first audio signal. A location of the user with respect to the mixed reality environment is determined. An acoustic region associated with the location of the user is identified. A first acoustic parameter associated with the first acoustic region is determined. A transfer function is determined using the first acoustic parameter. The transfer function is applied to the first audio signal to produce a second audio signal, which is then presented to the user.

Claims (66)

1. A method of presenting an audio signal to a user of a mixed reality environment, the method comprising:

detecting an audio event associated with the mixed reality environment, where the audio event is associated with a first audio signal;

determining a location of the user with respect to the mixed reality environment;

detecting a spatial property of an environment of the user;

identifying, based on the detected spatial property, a predefined first acoustic region;

determining a first acoustic parameter associated with the predefined first acoustic region;

determining, using the first acoustic parameter, a transfer function;

applying the transfer function to the first audio signal to produce a second audio signal; and

presenting, to the user, the second audio signal.

2. The method of claim 1 , wherein the first audio signal comprises a waveform audio file.

3. The method of claim 1 , wherein the first audio signal comprises a live audio stream.

4. The method of claim 1 , wherein:

the user is associated with a wearable system comprising one or more sensors and a display configured to present a view of the mixed reality environment, and

identifying the predefined first acoustic region comprises detecting first sensor input from the one or more sensors and identifying the predefined first acoustic region is based on the first sensor input.

5. The method of claim 4 , wherein determining the first acoustic parameter comprises detecting second sensor input from the one or more sensors and determining the first acoustic parameter is based on the second sensor input.

6. The method of claim 5 , wherein determining the first acoustic parameter further comprises identifying, based on the second sensor input, a geometric characteristic of the predefined first acoustic region and determining the first acoustic parameter is based on the geometric characteristic.

7. The method of claim 5 , wherein determining the first acoustic parameter further comprises identifying, based on the second sensor input, a material associated with the predefined first acoustic region and determining the first acoustic parameter is based on the material.

8. The method of claim 4 , wherein the wearable system further comprises a microphone, the microphone is located in the predefined first acoustic region, and the first acoustic parameter is determined based on a signal detected by the microphone.

9. The method of claim 1 , wherein the first acoustic parameter corresponds to a reverberation parameter.

10. The method of claim 1 , wherein the first acoustic parameter corresponds to a filtering parameter.

11. The method of claim 1 , further comprising:

detecting a second spatial property of the environment of the user;

identifying, based on the detected second spatial property, a predefined second acoustic region, the predefined second acoustic region acoustically coupled to the predefined first acoustic region; and

determining a second acoustic parameter associated with the predefined second acoustic region,

wherein the transfer function is determined using the second acoustic parameter.

12. A system comprising:

a wearable headgear unit including:

a display configured to present a view of a mixed reality environment;

a speaker;

one or more sensors; and

circuitry configured to perform a method comprising:

detecting an audio event associated with the mixed reality environment, where the audio event is associated with a first audio signal;

determining, based on the one or more sensors, a location of the wearable headgear unit with respect to the mixed reality environment;

detecting a spatial property of an environment of the user;

identifying, based on the detected spatial property, a predefined first acoustic region;

determining a first acoustic parameter associated with the predefined first acoustic region;

determining, using the first acoustic parameter, a transfer function;

applying the transfer function to the first audio signal to produce a second audio signal; and

presenting, to the user via the speaker, the second audio signal.

13. The system of claim 12 , wherein the first audio signal comprises a waveform audio file.

14. The system of claim 12 , wherein the first audio signal comprises a live audio stream.

15. The system of claim 12 , wherein:

identifying the predefined first acoustic region comprises detecting first sensor input from the one or more sensors and identifying the first acoustic region is based on the first sensor input.

16. The system of claim 15 , wherein determining the first acoustic parameter comprises detecting second sensor input from the one or more sensors and determining the first acoustic parameter is based on the second sensor input.

17. The system of claim 16 , wherein determining the first acoustic parameter further comprises identifying, based on the second sensor input, a geometric characteristic of the predefined first acoustic region and determining the first acoustic parameter is based on the geometric characteristic.

18. The system of claim 16 , wherein determining the first acoustic parameter further comprises identifying, based on the second sensor input, a material associated with the predefined first acoustic region and determining the first acoustic parameter is based on the material.

19. The system of claim 15 , wherein the first acoustic parameter is determined based on a signal detected by the microphone.

20. The system of claim 12 , wherein the first acoustic parameter corresponds to a reverberation parameter.

21. The system of claim 12 , wherein the first acoustic parameter corresponds to a filtering parameter.

22. The system of claim 12 , further comprising:

detecting a second spatial property of the environment of the user;

identifying, based on the detected second spatial property, a predefined second acoustic region, the predefined second acoustic region acoustically coupled to the predefined first acoustic region; and

determining a second acoustic parameter associated with the predefined second acoustic region,

wherein the transfer function is determined using the second acoustic parameter.

23. An augmented reality system comprising:

a localization subsystem configured to detect a spatial property of an environment of the user, and determine, based on the detected spatial property, an identity of a predefined first space in which the augmented reality system is located;

a communication subsystem configured to communicate the identity of the first space in which the augmented reality system is located and further configured to receive an audio parameter associated with the first space; and

an audio output subsystem configured to process an audio segment based on the audio parameter and further configured to output the audio segment.

24. An augmented reality system comprising:

a sensor subsystem configured to detect a spatial property of an environment of the user, and determine, based on the detected spatial property, information associated with an acoustic property of a predefined first space corresponding to a location of the augmented reality system;

an audio processing subsystem configured to process an audio segment based on the information, the audio processing subsystem communicatively coupled to the sensor subsystem; and

a speaker for presenting the audio segment, the speaker coupled to the audio processing subsystem.

25. The augmented reality system of claim 24 wherein the sensor subsystem is further configured to determine geometry information for the predefined first space.

26. The augmented reality system of claim 24 wherein the sensor subsystem comprises a camera.

27. The augmented reality system of claim 24 wherein the sensor subsystem comprises an object recognizer configured to recognize objects having distinct acoustic absorption properties.

28. The augmented reality system of claim 24 wherein the sensor subsystem comprises a microphone.

Assignments (3)
ASSIGNMENT OF SECURITY INTEREST IN PATENTS Recorded Nov 7, 2019
From: JPMORGAN CHASE BANK, N.A.
To: CITIBANK, N.A.
Reel/Frame 050967/0138 →
PATENT SECURITY AGREEMENT Recorded Aug 22, 2019
From: MAGIC LEAP, INC.; MOLECULAR IMPRINTS, INC.; MENTOR ACQUISITION ONE, LLC
To: JP MORGAN CHASE BANK, N.A.
Reel/Frame 050138/0287 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 14, 2018
From: SCHMIDT, BRIAN LLOYD; TAJIK, JEHANGIR; JOT, JEAN-MARC
To: MAGIC LEAP, INC.
Reel/Frame 047500/0076 →
Cited By (2)
US 12,317,064 US 12,615,407