Granted Patent B2

US 11,785,180 · App. 17/531,636 · Granted Oct 10, 2023

Inventors: Michael H Peters (Washington, DC); Alexander M. Stufflebeam (Indianapolis, IN)

Assignee: Reelay Meetings, Inc.

H04N7/152G06V20/41

Abstract

Claims (57)

1. A method performed by one or more computers, the method comprising:

identifying, by the one or more computers, multiple sub-sessions of a network-based communication session in which multiple remote endpoint devices each provide media streams over a communication network, wherein the sub-sessions occur concurrently and include different subsets of the endpoint devices, and wherein each sub-session involves sharing media streams among the subset of endpoint devices included in the sub-session;

for each of the sub-sessions, identifying, by the one or more computers, the endpoint devices included in the sub-session;

obtaining, by the one or more computers, user state data for each of the endpoint devices, the user state data for each endpoint device being generated based on analysis of face images of the user of the endpoint device captured by the endpoint device during the communication session;

aggregating, by the one or more computers, the user state data to determine a sub-session state for each of the sub-sessions, wherein the sub-session state for each sub-session is determined based on the user state data for the subset of endpoint devices identified as being included in the sub-session; and

during the communication session, communicating, by the one or more computers, over the communication network with a remote device associated with the communication session to cause a user interface of the remote device to indicate the sub-session states determined for one or more of the multiple sub-sessions.

2. The method of claim 1 , wherein the sub-session states are scores or classifications assigned for the sub-sessions.

3. The method of claim 1 , wherein the user state data for the endpoint devices comprises one or more measures of emotional or cognitive attributes of the users of the endpoint devices, the one or more measures of emotional or cognitive attributes being derived based on properties of the face images of the users.

4. The method of claim 1 , wherein the user state data is further based on characteristics of speech of users of the endpoint devices during the communication session.

5. The method of claim 1 , comprising:

evaluating the sub-session states determined for the sub-sessions;

determining, based on the evaluation, that a predetermined condition has occurred for a particular sub-session based on the sub-session state for the particular sub-session; and

causing a notification for the predetermined condition to be provided on the user interface of the remote device in response to the determination.

6. The method of claim 1 , comprising:

ranking or filtering the sub-sessions based on the determined sub-session states; and

sending data to the remote device that causes the user interface to present user interface elements corresponding to the sub-sessions arranged based on the ranking or filtering.

7. The method of claim 1 , wherein the user interface comprises a plurality of areas each corresponding to a different sub-session of the multiple sub-sessions, wherein the method comprises:

selecting one or more actions to adjust or interact with a particular sub-session based on the sub-session state for the sub-session; and

causing interactive user interface elements configured to initiate the one or more actions to be provided in the area of the user interface corresponding to the particular sub-session.

8. The method of claim 1 , wherein the sub-sessions are virtual breakout rooms that endpoint devices join or are assigned to after joining the communication session.

9. The method of claim 1 , wherein the network-based communication session is a primary video conference session, and the sub-sessions are secondary video conference sessions that occur concurrently with each other are each associated with the primary video conference session.

10. The method of claim 1 , wherein the user state data is generated by the respective endpoint devices each processing video data that the respective endpoint devices captured during the communication session.

11. A system comprising:

one or more computers; and

one or more computer-readable media storing instructions that are operable, when executed, to perform operations comprising:

identifying, by the one or more computers, multiple sub-sessions of a network-based communication session in which multiple remote endpoint devices each provide media streams over a communication network, wherein the sub-sessions occur concurrently and include different subsets of the endpoint devices, and wherein each sub-session involves sharing media streams among the subset of endpoint devices included in the sub-session;

for each of the sub-sessions, identifying, by the one or more computers, the endpoint devices included in the sub-session;

obtaining, by the one or more computers, user state data for each of the endpoint devices, the user state data for each endpoint device being generated based on analysis of face images of the user of the endpoint device captured by the endpoint device during the communication session;

aggregating, by the one or more computers, the user state data to determine a sub-session state for each of the sub-sessions, wherein the sub-session state for each sub-session is determined based on the user state data for the subset of endpoint devices identified as being included in the sub-session; and

during the communication session, communicating, by the one or more computers, over the communication network with a remote device associated with the communication session to cause a user interface of the remote device to indicate the sub-session states determined for one or more of the multiple sub-sessions.

12. The system of claim 11 , wherein the sub-session states are scores or classifications assigned for the sub-sessions.

13. The system of claim 11 , wherein the user state data for the endpoint devices comprises one or more measures of emotional or cognitive attributes of the users of the endpoint devices, the one or more measures of emotional or cognitive attributes being derived based on properties of the face images of the users.

14. The system of claim 11 , wherein the user state data is further based on characteristics of speech of users of the endpoint devices during the communication session.

15. A method performed by one or more computers, the method comprising:

identifying, by the one or more computers, multiple sub-sessions of a network-based communication session in which multiple remote endpoint devices each provide media streams over a communication network, wherein the sub-sessions occur concurrently and include different subsets of the endpoint devices, and wherein each sub-session involves sharing media streams among the subset of endpoint devices included in the sub-session;

for each of the sub-sessions, identifying, by the one or more computers, the endpoint devices included in the sub-session;

obtaining, by the one or more computers, user state data for each of the endpoint devices, the user state data for each endpoint device being generated based on analysis of face images of the user of the endpoint device captured by the endpoint device during the communication session;

aggregating, by the one or more computers, the user state data to determine a sub-session state for each of the sub-sessions, wherein the sub-session state for each sub-session is determined based on the user state data for the subset of endpoint devices identified as being included in the sub-session;

based on the sub-session state determined for a particular sub-session, selecting, by the one or more computers, an action to adjust or interact with the particular sub-session;

generating, by the one or more computers, user interface data configured to cause a user interface of a remote device to present an interactive control configured to initiate the selected action for the particular sub-session in response to interaction with the interactive control; and

during the communication session, communicating, by the one or more computers, over the communication network with a remote device associated with the communication session to cause a user interface of the remote device to (i) indicate the sub-session states determined one or more of the multiple sub-sessions and (ii) present the interactive control configured to initiate the selected action for the particular sub-session that was determined based on the sub-session state determined for the particular sub-session.

16. The method of claim 15 , wherein the action selected comprises at least one of:

removing a participant from the particular sub-session;

muting audio for a participant in the particular sub-session;

pausing the particular sub-session;

ending the particular sub-session;

playing a pre-recorded media segment in the particular sub-session;

sending an alert to one or more participants in the particular sub-session;

introducing the user of the remote device into the particular sub-session; or

setting or altering a speaking time limit for one or more participants in the particular sub-session.

17. The method of claim 15 , comprising:

storing (i) rules or thresholds defining different conditions to be detected in the sub-sessions, and (ii) mapping data identifying different actions corresponding to the different conditions; and

determining, based on the sub-session state determined for the particular sub-session, that a particular condition of the different conditions has occurred;

wherein selecting the action to adjust or interact with the particular sub-session comprises selecting a particular action that the mapping data identifies as corresponding to the particular condition that is determined to occur.

18. The method of claim 17 , wherein the different conditions comprise conditions defined to be present when corresponding values or ranges of values for cognitive or emotional attributes of participants in a sub-session are detected.

19. The method of claim 17 , wherein the different conditions comprise conditions defined to be present when a predetermined combination, predetermined pattern, or predetermined trend of cognitive or emotional attributes are detected.

20. The method of claim 15 , wherein the user interface is configured to present media content from one or more of the endpoint devices in the communication concurrently with a monitoring region that indicates the sub-session states for multiple sub-sessions and the interactive control, wherein the monitoring region is periodically updated in response to changes in the sub-session states to (i) alter a set of interactive controls provided in the monitoring region or (ii) change a ranking or prioritization of sub-session state information for multiple sub-sessions.

Assignments (3)

Continuity (21)

Cited By (5)

US 12,341,831 US 12,368,819 US 12,494,293 US 12,641,194 US 12,701,199