IP Library Granted Patent US 11,039,267
Granted Patent B2
US 11,039,267 · App. 16/861,057 · Granted Jun 15, 2021

Head pose mixing of audio files

Inventors: Stephen Vincent Mangiat (Santa Monica, CA); Michael Benson Tucker (Brooklyn, NY); Anastasia Andreyevna Tajik (Ft. Lauderdale, FL)
Assignee: Magic Leap, Inc.
H04S7/40G06F1/163G06F3/011G06F3/012G06F3/013G06F3/017G06F3/165G06F3/167H04S7/304G10H2210/131
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,039,267
App. No.
16/861,057
Granted
Jun 15, 2021
Kind
B2
Abstract

Examples of wearable devices that can present to a user of the display device an audible or visual representation of an audio file comprising a plurality of stem tracks that represent different audio content of the audio file are described. Systems and methods are described that determine the pose of the user; generate, based on the pose of the user, an audio mix of at least one of the plurality of stem tracks of the audio file; generate, based on the pose of the user and the audio mix, a visualization of the audio mix; communicate an audio signal representative of the audio mix to the speaker; and communicate a visual signal representative of the visualization of the audio mix to the display.

Claims (43)

1. A computing device comprising:

one or more sensors configured to provide sensor data usable to determine a pose of a user;

a computer processor configured to execute software instructions to:

determine, based at least on the sensor data, a pose of the user in a three-dimensional (3D) environment;

render on a display device an object comprising a plurality of surfaces associated with different tracks of an audio file;

in response to identifying a change in an orientation of the object relative to the pose of the user:

identify one or more surfaces of the object along a pose direction of the user in the 3D environment;

select one or more tracks associated with the identified one or more surfaces; and

determine an audio mix of the selected one or more tracks; and

communicate an audio signal representative of the audio mix to a speaker.

2. The computing device of claim 1 , wherein tracks of the plurality of tracks are associated with different musical instruments or a vocal track.

3. The computing device of claim 1 , wherein tracks of the plurality of tracks comprise information associated with a spatial location and a spatial directionality of the track.

4. The computing device of claim 1 , wherein the one or more sensors comprise at least one of: a gyroscope, an accelerometer, an inertial measurement unit, an eye tracking camera, or a camera configured to image an environment of the user.

5. The computing device of claim 1 , wherein the pose comprises at least one of: a head pose, an eye pose, a body pose, or a hand gesture.

6. The computing device of claim 1 , wherein the object comprises a graphical representation of an instrument or a person associated with performance of the track.

7. The computing device of claim 1 , wherein at least some of the object is presented at different depth planes relative to the user.

8. The computing device of claim 1 , wherein the computer processor is further configured to cause the computing device to recognize a physical object in an environment of the user, and to present a graphical representation of a track of the plurality of tracks as being associated with the physical object.

9. The computing device of claim 1 , wherein the computer processor is further configured to cause the computing device to generate the audio mix based at least in part on an input from the user.

10. A computerized method, performed by a computing system having one or more hardware computer processors and one or more non-transitory computer readable storage device storing software instructions executable by the computing system to perform the computerized method comprising:

determining, based at least on sensor data, a pose of a user in a three-dimensional (3D) environment;

rendering on a display device an object comprising a plurality of surfaces associated with different tracks of an audio file;

in response to identifying a change in an orientation of the object relative to the pose of the user:

identifying one or more surfaces of the object along a pose direction of the user in the 3D environment;

selecting one or more tracks associated with the identified one or more surfaces; and

determining an audio mix of the selected one or more tracks; and

communicating an audio signal representative of the audio mix to a speaker.

11. The computerized method of claim 10 , wherein tracks of the plurality of tracks are associated with different musical instruments or a vocal track.

12. The computerized method of claim 10 , wherein tracks of the plurality of tracks comprise information associated with a spatial location and a spatial directionality of the track.

13. The computerized method of claim 10 , wherein the sensor data is received from one or more sensors comprising at least one of: a gyroscope, an accelerometer, an inertial measurement unit, an eye tracking camera, or a camera configured to image an environment of the user.

14. The computerized method of claim 10 , wherein the pose comprises at least one of: a head pose, an eye pose, a body pose, or a hand gesture.

15. The computerized method of claim 10 , wherein the object comprises a graphical representation of an instrument or a person associated with performance of the track.

16. The computerized method of claim 10 , wherein at least some of the object is presented at different depth planes relative to the user.

17. The computerized method of claim 10 , wherein the computer processor is further configured to cause the computing device to recognize a physical object in an environment of the user, and to present a graphical representation of a track of the plurality of tracks as being associated with the physical object.

18. The computerized method of claim 10 , wherein the computer processor is further configured to cause the computing device to generate the audio mix based at least in part on an input from the user.

19. A non-transitory computer readable storage device storing software instructions executable by a computing system to cause the computing system to:

determine, based at least on sensor data, a pose of a user in a three-dimensional (3D) environment;

render on a display device an object comprising a plurality of surfaces associated with different tracks of an audio file;

in response to identifying a change in an orientation of the object relative to the pose of the user:

identify one or more surfaces of the object along a pose direction of the user in the 3D environment;

select one or more tracks associated with the identified one or more surfaces; and

determine an audio mix of the selected one or more tracks; and

communicate an audio signal representative of the audio mix to a speaker.

20. The non-transitory computer readable storage device of claim 19 , wherein the object comprises a graphical representation of an instrument or a person associated with performance of the track.

Assignments (4)
SECURITY INTEREST Recorded Oct 15, 2025
From: MAGIC LEAP, INC.; MENTOR ACQUISITION ONE, LLC; MOLECULAR IMPRINTS, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 073109/0476 →
SECURITY INTEREST Recorded May 24, 2022
From: MOLECULAR IMPRINTS, INC.; MENTOR ACQUISITION ONE, LLC; MAGIC LEAP, INC.
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 060338/0665 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 22, 2020
From: MANGIAT, STEPHEN VINCENT; TUCKER, MICHAEL BENSON; TAJIK, ANASTASIA ANDREYEVNA
To: MAGIC LEAP, INC.
Reel/Frame 053847/0254 →
SECURITY INTEREST Recorded May 21, 2020
From: MAGIC LEAP, INC.; MOLECULAR IMPRINTS, INC.; MENTOR ACQUISITION ONE, LLC
To: CITIBANK, N.A., AS COLLATERAL AGENT
Reel/Frame 052729/0791 →
Continuity (4)
Continuation 15264164 · Sep 13, 2016
Provisional Application 62289085 · Jan 29, 2016
Provisional Application 62219604 · Sep 16, 2015
Related Publication 20200260213A1 · Aug 13, 2020