IP Library Granted Patent US 10,812,928
Granted Patent B1
US 10,812,928 · App. 16/538,638 · Granted Oct 20, 2020

Audio service design for operating systems

Inventors: Marc Ihaab Salem (Sunnyvale, CA); Sumit Kamath (Kirkland, WA)
Assignee: Facebook Technologies, LLC
H04S7/305H04R5/02H04S7/301H04S7/303H04S2420/01
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,812,928
App. No.
16/538,638
Granted
Oct 20, 2020
Kind
B1
Abstract

In one embodiment, an audio service executing on a computing device performs a method including receiving, from one or more applications, a number of world-locked audio objects, where each of the world-locked audio objects is associated with a coordinate within a three-dimensional environment; identifying a first subset of the world-locked audio objects and a second subset of the world-locked audio objects, where the first subset includes a number of world-locked audio objects and the second subset includes one or more of the world-locked audio objects that are excluded from the first subset; generating a spherical harmonic object based on the second subset of world-locked audio objects and their associated coordinates; and sending data associated with the first subset of world-locked audio objects and the spherical harmonic object to an audio driver configured to render audio for simultaneous output.

Claims (47)

1. A method comprising, by an audio service executing on a computing device:

receiving, from one or more applications, world-locked audio objects, wherein each of the world-locked audio objects is associated with a coordinate within a three-dimensional environment;

identifying a first subset of the world-locked audio objects and a second subset of the world-locked audio objects, wherein the first subset includes a predetermined number of world-locked audio objects permitted by the audio service and the second subset includes one or more of the world-locked audio objects that are excluded from the first subset;

generating, based on the second subset of the world-locked audio objects and their associated coordinates, a spherical harmonic object to represent the one or more of the world-locked audio objects in the second subset; and

sending data associated with the first subset of the world-locked audio objects and the spherical harmonic object to an audio driver configured to render audio for the world-locked audio objects for simultaneous output.

2. The method of claim 1 , wherein the generating the spherical harmonic object is further based on at least one of a headset-related transfer function, location, or pose of a user relative to the three-dimensional environment.

3. The method of claim 1 , wherein the coordinate within the three-dimensional environment defines an audio source location.

4. The method of claim 1 , wherein the data associated with the first subset of world-locked audio objects comprises direct propagation impulse responses, reverb impulse responses, and mono audio streams.

5. The method of claim 4 , further comprising:

generating the direct propagation impulse responses and reverb impulse responses based on at least one of a headset-related transfer function, location, or pose of a user relative to the three-dimensional environment.

6. The method of claim 5 , wherein the generating the direct propagation impulse responses and reverb impulse responses is further based on a three-dimensional model of at least a portion of the three-dimensional environment.

7. The method of claim 1 , further comprising:

receiving one or more additional spherical harmonic objects;

wherein the generating the spherical harmonic object is further based on the one or more additional spherical harmonic objects.

8. The method of claim 1 , further comprising:

receiving one or more head-locked audio objects; and

wherein the data associated with the first subset of world-locked audio objects and the spherical harmonic object is further associated with the one or more head-locked audio objects.

9. The method of claim 1 , wherein an operating system of the computing device comprises the audio service.

10. The method of claim 1 , wherein the one or more applications and the audio driver are each coupled to a synchronized clock reference.

11. One or more computer-readable non-transitory storage media embodying software that is operable when executed by a client device to:

receive, from one or more applications, world-locked audio objects, wherein each of the world-locked audio objects is associated with a coordinate within a three-dimensional environment;

identify a first subset of the world-locked audio objects and a second subset of the world-locked audio objects, wherein the first subset includes a predetermined number of world-locked audio objects permitted by an audio service of the client device and the second subset includes one or more of the world-locked audio objects that are excluded from the first subset;

generate, based on the second subset of the world-locked audio objects and their associated coordinates, a spherical harmonic object to represent the one or more of the world-locked audio objects in the second subset; and

send data associated with the first subset of the world-locked audio objects and the spherical harmonic object to an audio driver configured to render audio for the world-locked audio objects for simultaneous output.

12. The media of claim 11 , wherein the coordinate within the three-dimensional environment defines an audio source location.

13. The media of claim 11 , wherein the data associated with the first subset of world-locked audio objects comprises direct propagation impulse responses, reverb impulse responses, and mono audio streams.

14. The media of claim 11 , wherein the software is further operable when executed by the client device to:

receive one or more additional spherical harmonic objects; and

wherein the generating the spherical harmonic object is further based on the one or more additional spherical harmonic objects.

15. The media of claim 11 , wherein the software is further operable when executed by the client device to:

receive one or more head-locked audio objects to be simultaneously output with the world-locked audio objects; and

wherein the data associated with the first subset of world-locked audio objects and the spherical harmonic object is further associated with the one or more head-locked audio objects.

16. A client device comprising:

one or more processors; and

one or more computer-readable non-transitory storage media coupled to one or more of the one or more processors and comprising instructions operable when executed by one or more of the one or more processors to cause the client device to:

receive, from one or more applications, world-locked audio objects, wherein each of the world-locked audio objects is associated with a coordinate within a three-dimensional environment;

identify a first subset of the world-locked audio objects and a second subset of the world-locked audio objects, wherein the first subset includes a predetermined number of world-locked audio objects permitted by an audio service of the client device and the second subset includes one or more of the world-locked audio objects that are excluded from the first subset;

generate, based on the second subset of the world-locked audio objects and their associated coordinates, a spherical harmonic object to represent the one or more of the world-locked audio objects in the second subset; and

send data associated with the first subset of the world-locked audio objects and the spherical harmonic object to an audio driver configured to render audio for the world-locked audio objects for simultaneous output.

17. The client device of claim 16 , wherein the coordinate within the three-dimensional environment associated with each of the world-locked audio objects defines an audio source location within the three-dimensional environment.

18. The client device of claim 16 , wherein the data associated with the first subset of world-locked audio objects comprises direct propagation impulse responses, reverb impulse responses, and mono audio streams.

19. The client device of claim 16 , wherein the instructions are further operable when executed by the one or more processors to:

receive one or more additional spherical harmonic objects;

wherein the generating the spherical harmonic object is further based on the one or more additional spherical harmonic objects.

20. The client device of claim 16 , wherein the instructions are further operable when executed by the one or more processors to:

receive one or more head-locked audio objects; and

wherein the data associated with the first subset of world-locked audio objects and the spherical harmonic object is further associated with the one or more head-locked audio objects.

Assignments (2)
CHANGE OF NAME Recorded Jul 6, 2022
From: FACEBOOK TECHNOLOGIES, LLC
To: META PLATFORMS TECHNOLOGIES, LLC
Reel/Frame 060591/0848 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 9, 2019
From: SALEM, MARC IHAAB; KAMATH, SUMIT
To: FACEBOOK TECHNOLOGIES, LLC
Reel/Frame 050318/0615 →