IP Library › Granted Patent US 12,231,869
Granted Patent B2
US 12,231,869 · App. 17/976,417 · Granted Feb 18, 2025

Spatial audio in virtual conference mingling

Inventors: Zhaofeng Jia (Saratoga, CA); Qiyong Liu (Singapore, SG); Mengfan Zhang (Mountain View, CA)
Assignee: Zoom Video Communications, Inc.
H04S7/303H04L65/403H04S3/008H04S2400/01H04S2400/11H04S2420/01
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,231,869
App. No.
17/976,417
Granted
Feb 18, 2025
Kind
B2
Abstract

One example method includes presenting, by a client device, a view of a virtual conference hosted by a virtual conference provider, the virtual conference including a plurality of participants, the client device associated with a participant of the plurality of participants, the view including a plurality of groupings of participants within a virtual conference area, each grouping associated with a different meeting or sub-meeting of the virtual conference; assign a location within the virtual conference area to the participant; receiving, at the client device from the conference provider, one or more audio streams associated with one or more audio sources within the plurality of groupings, the one or more audio streams provided by one or more remote client devices; determining a first location within the virtual conference area of a first audio source of the one or more audio sources; generating a plurality of spatialized audio streams based on the first location of the first audio source, the location of the indicator, and a first audio stream associated with the first audio source; and outputting the spatialized audio streams.

Claims (86)

1. A method comprising:

presenting, by a client device, a view of a virtual conference hosted by a virtual conference provider, the virtual conference including a plurality of participants, the client device associated with a participant of the plurality of participants, the view including a plurality of groupings of participants within a virtual conference area, each grouping associated with a different meeting or sub-meeting of the virtual conference;

assigning a location within the virtual conference area to the participant;

receiving, at the client device from the conference provider, one or more audio streams associated with one or more audio sources within the plurality of groupings, the one or more audio streams provided by one or more remote client devices;

determining a first location within the virtual conference area of a first audio source of the one or more audio sources;

generating a plurality of spatialized audio streams based on the first location of the first audio source, the location of the participant, and a first audio stream associated with the first audio source; and

outputting the spatialized audio streams.

2. The method of claim 1 , further comprising:

selecting a head-related transfer function (“HRTF”) from a set of HRTFs based on the first location of the first audio source and the location of the participant, and

wherein generating the plurality of spatialized audio streams is further based on the selected HRTF.

3. The method of claim 1 , wherein the first audio source is associated with a first participant within a first grouping of the plurality of groupings, and further comprising:

determining a second location within the virtual conference area of a second audio source of the one or more audio sources, the second audio source associated with a second participant within a second grouping of the plurality of groupings; and

wherein generating the plurality of spatialized audio streams is further based on the location of the second audio source and the second audio stream associated with the second audio source.

4. The method of claim 3 , further comprising:

selecting a first head-related transfer function (“HRTF”) from a set of HRTFs based on the first location of the first audio source and the location of the participant,

selecting a second HRTF from a set of HRTFs based on the second location of the second audio source and the location of the participant, and

generating the plurality of spatialized audio streams is further based on the first and second HRTFs.

5. The method of claim 1 , further comprising:

receiving, by the client device, an input changing the location of the participant;

updating the location of the participant based on the input;

generating a plurality of spatialized audio streams based on the first location of the first audio source, the updated location of the participant, and a first audio stream associated with the first audio source; and

outputting the spatialized audio streams.

6. The method of claim 1 , wherein the participant is a member of a first grouping of the plurality of groupings and the first audio source is associated with a first participant within a second grouping of the plurality of groupings, and further comprising:

selecting a first head-related transfer function (“HRTF”) from a set of HRTFs based on the first location of the first audio source and the location of the participant;

receiving a second audio stream from a second participant within the first grouping;

determining a second location with respect to the participant of a second audio source associated with the second participant within the first grouping;

selecting a second HRTF from a set of HRTFs based on the second location of the second audio source with respect to the participant; and

generating the plurality of spatialized audio streams is further based on the first and second HRTFs.

7. The method of claim 6 , wherein the first HRTF is a far-field HRTF and the second HRTF is a near-field HRTF.

8. The method of claim 6 , further comprising:

receiving, from a camera connected to the client device, a video stream;

determining a pose of the participant's head in the video stream; and

wherein selecting the first or the second HRTF is based on the pose of the participant's head.

9. A system comprising:

a communications interface;

a non-transitory computer-readable medium; and

one or more processors communicatively coupled to the communications interface and the non-transitory computer-readable medium, the one or more processors configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to:

present, by a client device, a view of a virtual conference hosted by a virtual conference provider, the virtual conference including a plurality of participants, the client device associated with a participant of the plurality of participants, the view including a plurality of groupings of participants within a virtual conference area, each grouping associated with a different meeting or sub-meeting of the virtual conference;

assign a location within the virtual conference area to the participant;

receive, from the conference provider, one or more audio streams associated with one or more audio sources within the plurality of groupings, the one or more audio streams provided by one or more remote client device;

determine a first location within the virtual conference area of a first audio source of the one or more audio sources;

generate a plurality of spatialized audio streams based on the first location of the first audio source, the location of the participant, and a first audio stream associated with the first audio source; and

output the spatialized audio streams.

10. The system of claim 9 , wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to:

select a head-related transfer function (“HRTF”) from a set of HRTFs based on the first location of the first audio source and the location of the participant, and

generate the plurality of spatialized audio streams further based on the selected HRTF.

11. The system of claim 9 , wherein the first audio source is associated with a first participant within a first grouping of the plurality of groupings, and wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to:

determine a second location within the virtual conference area of a second audio source of the one or more audio sources, the second audio source associated with a second participant within a second grouping of the plurality of groupings; and

generate the plurality of spatialized audio streams further based on the location of the second audio source and the second audio stream associated with the second audio source.

12. The system of claim 11 , wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to:

select a first head-related transfer function (“HRTF”) from a set of HRTFs based on the first location of the first audio source and the location of the participant,

select a second HRTF from a set of HRTFs based on the second location of the second audio source and the location of the participant, and

generate the plurality of spatialized audio streams is further based on the first and second HRTFs.

13. The system of claim 9 , wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to:

receive an input changing the location of the participant;

update the location of the participant based on the input;

generate a plurality of spatialized audio streams based on the first location of the first audio source, the updated location of the participant, and a first audio stream associated with the first audio source; and

output the spatialized audio streams.

14. The system of claim 9 , wherein the participant is a member of a first grouping of the plurality of groupings and the first audio source is associated with a first participant within a second grouping of the plurality of groupings, and wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to:

select a first head-related transfer function (“HRTF”) from a set of HRTFs based on the first location of the first audio source and the location of the participant;

receive a second audio stream from a second participant within the first grouping;

determine a second location with respect to the participant of a second audio source associated with the second participant within the first grouping;

select a second HRTF from a set of HRTFs based on the second location of the second audio source with respect to the participant; and

generate the plurality of spatialized audio streams is further based on the first and second HRTFs.

15. The system of claim 14 , wherein the first HRTF is a far-field HRTF and the second HRTF is a near-field HRTF.

16. The system of claim 14 , wherein the system further comprises a camera, and wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to:

receive, from the camera, a video stream;

determining a pose of the participant's head in the video stream; and

select the first or the second HRTF further based on the pose of the participant's head.

17. A non-transitory computer-readable medium comprising processor-executable instructions configured to cause one or more processors to:

present, by a client device, a view of a virtual conference hosted by a virtual conference provider, the virtual conference including a plurality of participants, the client device associated with a participant of the plurality of participants, the view including a plurality of groupings of participants within a virtual conference area, each grouping associated with a different meeting or sub-meeting of the virtual conference;

assign a location within the virtual conference area to the participant;

receive, from the conference provider, one or more audio streams associated with one or more audio sources within the plurality of groupings, the one or more audio streams provided by one or more remote client device;

determine a first location within the virtual conference area of a first audio source of the one or more audio sources;

generate a plurality of spatialized audio streams based on the first location of the first audio source, the location of the participant, and a first audio stream associated with the first audio source; and

output the spatialized audio streams.

18. The non-transitory computer-readable medium of claim 17 , further comprising processor-executable instructions configured to cause one or more processors to:

select a head-related transfer function (“HRTF”) from a set of HRTFs based on the first location of the first audio source and the location of the participant, and

generate the plurality of spatialized audio streams further based on the selected HRTF.

19. The non-transitory computer-readable medium of claim 17 , wherein the first audio source is associated with a first participant within a first grouping of the plurality of groupings, and further comprising processor-executable instructions configured to cause one or more processors to:

determine a second location within the virtual conference area of a second audio source of the one or more audio sources, the second audio source associated with a second participant within a second grouping of the plurality of groupings; and

generate the plurality of spatialized audio streams further based on the location of the second audio source and the second audio stream associated with the second audio source.

20. The non-transitory computer-readable medium of claim 19 , further comprising processor-executable instructions configured to cause one or more processors to:

select a first head-related transfer function (“HRTF”) from a set of HRTFs based on the first location of the first audio source and the location of the participant,

select a second HRTF from a set of HRTFs based on the second location of the second audio source and the location of the participant, and

generate the plurality of spatialized audio streams further based on the first and second HRTFs.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 20, 2024
From: ZHANG, MENGFAN; JIA, ZHAOFENG; LIU, QIYONG
To: ZOOM VIDEO COMMUNICATIONS, INC.
Reel/Frame 066845/0730 →
Continuity (1)
Related Publication 20240147177A1 · May 2, 2024
References Cited (3)
US 11671272B1 · Farina · 2023 [cited by examiner]
US 20110271192A1 · Jones · 2011 [cited by examiner]
US 20230133265A1 · Springer · 2023 [cited by examiner]