Granted Patent B2

US 12,342,100 · App. 18/175,698 · Granted Jun 24, 2025

Inventors: Jeffrey William Smith (Layton, UT); Chi-chian Yu (San Ramon, CA)

Assignee: Zoom Communications, Inc.

H04N7/15G06T7/70G06V40/10G10L15/22G10L25/57H04N5/2624H04N5/268H04N7/142H04N23/90H04R1/326G06N20/00G06T2207/10016G06T2207/30196G10L2015/227G10L2015/228

Abstract

Claims (55)

1. A method, comprising:

determining, based on a direction of audio captured within a conference room and an input video stream depicting conference participants in the conference room, a conversational context within the conference room;

selecting a first region of interest of the input video stream for use in a primary view of conferencing software, wherein the first region of interest is selected based on a determination that one or more of the conference participants depicted in the first region of interest are actively participating in a conversation corresponding to the conversational context;

selecting a second region of interest of the input video stream for use in a secondary view of the conferencing software, wherein the second region of interest is selected based on a determination that one or more of the conference participants depicted in the second region of interest are not actively participating in the conversation;

outputting, based on the conversational context, the first region of interest within a primary view of the conferencing software and the second region of interest within a secondary view of the conferencing software; and

changing, based on a second conversational context within the conference room, the output within the secondary view of the conferencing software to a third region of interest of the input video stream, wherein the third region of interest of the input video stream depicts one or more of the conference participants who are actively participating in a second conversation corresponding to the second conversational context,

wherein the primary view of the conferencing software and the secondary view of the conferencing software are simultaneously rendered within a gallery view layout user interface of the conferencing software.

2. The method of claim 1 , wherein the conversational context corresponds to one or both of a context or a length of a conversation involving one or more of the conference participants.

3. The method of claim 1 , wherein the input video stream is captured using a video capture device located within the conference room and the direction of audio is determined using an input audio stream captured using a multi-directional audio capture device located within the conference room.

4. The method of claim 1 , comprising:

changing, based on a change in the conversational context, an output within the primary view of the conferencing software to a region of interest of the input video stream other than the first region of interest.

5. The method of claim 1 , wherein the first region of interest is determined using a machine learning model trained to determine regions of interest of video streams based on conversational dynamic processing using recordings of past video conferences.

6. The method of claim 1 , wherein changing the output within the secondary view of the conferencing software to the third region of interest of the input video stream comprises:

determining the second conversational context; and

selecting the third region of interest of the input video stream for use in the secondary view of the conferencing software based on a determination that the one or more of the conference participants depicted in the third region of interest are actively participating in the second conversation.

7. The method of claim 1 , wherein the primary view of the conferencing software is a largest view of a gallery layout of the conferencing software and the secondary view of the conferencing software is smaller than the primary view.

8. The method of claim 1 , wherein the secondary view of the conferencing software switches between regions of interest depicting one or more of the conference participants that are not actively participating in the conversation while the primary view depicts the first region of interest.

9. The method of claim 1 , comprising:

detecting the conference participants within the input video stream by:

using facial detection to identify one or more humans within the input video stream; and

segmenting the one or more humans from a background identified within the input video stream.

10. The method of claim 1 , comprising:

determining the direction of audio by:

processing an input audio stream to detect voice activity within the conference room; and

determining the direction of audio based on the voice activity.

11. A non-transitory computer readable medium storing instructions operable to cause one or more processors to perform operations comprising:

determining, based on a direction of audio captured within a conference room and an input video stream depicting conference participants in the conference room, a conversational context within the conference room;

selecting a first region of interest of the input video stream for use in a primary view of conferencing software, wherein the first region of interest is selected based on a determination that one or more of the conference participants depicted in the first region of interest are actively participating in a conversation corresponding to the conversational context;

selecting a second region of interest of the input video stream for use in a secondary view of the conferencing software, wherein the second region of interest is selected based on a determination that one or more of the conference participants depicted in the second region of interest are not actively participating in the conversation;

outputting, based on the conversational context, the first region of interest within a primary view of the conferencing software and the second region of interest within a secondary view of the conferencing software; and

changing, based on a second conversational context within the conference room, the output within the secondary view of the conferencing software to a third region of interest of the input video stream, wherein the third region of interest of the input video stream depicts one or more of the conference participants who are actively participating in a second conversation corresponding to the second conversational context,

wherein the primary view of the conferencing software and the secondary view of the conferencing software are simultaneously rendered within a gallery view layout user interface of the conferencing software.

12. The non-transitory computer readable medium of claim 11 , the operations comprising:

changing, based on a change in the conversational context, an output with the primary view of the conferencing software to a region of interest of the input video stream other than the first region of interest.

13. The non-transitory computer readable medium of claim 11 , wherein changing the output within the secondary view of the conferencing software to the third region of interest of the input video stream comprises:

determining the second conversational context within the conference room; and

selecting the third region of interest of the input video stream.

14. The non-transitory computer readable medium of claim 11 , the operations comprising:

detecting the conference participants within the input video stream; and

determining the direction of audio.

15. An apparatus, comprising:

a memory; and

a processor configured to execute instructions stored in the memory to:

determine, based on a direction of audio captured within a conference room and an input video stream depicting conference participants in the conference room, a conversational context within the conference room;

select a first region of interest of the input video stream for use in a primary view of conferencing software, wherein the first region of interest is selected based on a determination that one or more of the conference participants depicted in the first region of interest are actively participating in a conversation corresponding to the conversational context;

select a second region of interest of the input video stream for use in a secondary view of the conferencing software, wherein the second region of interest is selected based on a determination that one or more of the conference participants depicted in the second region of interest are not actively participating in the conversation;

output, based on the conversational context, the first region of interest within a primary view of the conferencing software and the second region of interest within a secondary view of the conferencing software; and

change, based on a second conversational context within the conference room, the output within the secondary view of the conferencing software to a third region of interest of the input video stream, wherein the third region of interest of the input video stream depicts one or more of the conference participants who are actively participating in a second conversation corresponding to the second conversational context,

wherein the primary view of the conferencing software and the secondary view of the conferencing software are simultaneously rendered within a gallery view layout user interface of the conferencing software.

16. The apparatus of claim 15 , wherein the conversational context is determined using a machine learning model trained to determine regions of interest within the input video stream.

17. The apparatus of claim 15 , wherein the processor is configured to execute the instructions to:

determine to output the first region of interest within the primary view of the conferencing software based on an evaluation of the conversational context and one or more other conversational contexts.

18. The apparatus of claim 15 , wherein the output within the secondary view of the conferencing software is further changed based on a change to the conversational context while the first region of interest remains output within the primary view of the conferencing software.

19. The apparatus of claim 15 , wherein the conversational context corresponds to a presentation led by a conference participant included in the one or more conference participants depicted in the first region of interest.

20. The apparatus of claim 15 , wherein the conversational context corresponds to a dialogue between two or more conference participants included in the one or more conference participants depicted in the first region of interest.

Assignments (2)

Continuity (2)

References Cited (78)