IP Library › Granted Patent US 12,495,083
Granted Patent B2
US 12,495,083 · App. 18/517,949 · Granted Dec 9, 2025

Technologies for audiovisual communication using interestingness algorithms

Inventors: Sigal Louchheim (El Dorado Hills, CA); Omer Ben-Shalom (Rishon Le-Tzion, IL)
Assignee: Intel Corporation
H04L65/403G06F3/013G06V20/40G06V40/193G06V40/20H04L65/1083H04N5/265H04N7/147H04N7/15H04N7/152
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,495,083
App. No.
18/517,949
Granted
Dec 9, 2025
Kind
B2
Abstract

Technologies for audiovisual communication include an audiovisual server and a number of audiovisual client devices, including a presenter device and a number of audience devices. Each audience device captures an audiovisual stream and transmits the audiovisual stream to the audiovisual server. Each audience device also captures sensor input data such as eye tracking data or facial expression data and transmits abstracted sensor input data to the audiovisual server. The abstracted sensor input data may be based on the captured audiovisual stream. The audiovisual server determines an interestingness rating associated with each audience device based on the sensor input data, and selects one or more audiovisual streams based on the interestingness ratings. The audiovisual server transmits the selected audiovisual streams to the presenter device. The audiovisual server may update the interestingness rating algorithm based on feedback from the presenter device or the audience devices. Other embodiments are described and claimed.

Claims (49)

1 . An apparatus comprising:

interface circuitry;

instructions; and

at least one processor circuit to be programmed based on the instructions to:

obtain a plurality of audiovisual streams of a video conference, the plurality of audiovisual streams from a plurality of first client devices;

determine respective values for corresponding ones of the audiovisual streams based on at least one of facial expression data or eye tracking data associated with participants in the audiovisual streams

select, based on the values, a first one or more of the audiovisual streams to transmit to a second client device;

cause transmission of the first one or more of the audiovisual streams to the second client device, unselected ones of the audiovisual streams not to be transmitted to the second client device;

update the respective values for the corresponding ones of the audiovisual streams based on at least one of updated facial expression data or updated eye tracking data associated with the participants in the audiovisual streams; and

select, based on the updated values, a second one or more of the audiovisual streams to transmit to the second client device, at least one of the second one or more of the audiovisual streams different from the first one or more of the audiovisual streams.

2 . The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to:

select the second one or more of the audiovisual streams based on feedback from the second client device and the updated values; and

cause transmission of the second one or more of the audiovisual streams to the second client device.

3 . The apparatus of claim 2 , wherein the feedback is based on eye tracking data associated with presentation of the first one or more of the audiovisual streams by the second client device.

4 . The apparatus of claim 1 , wherein one or more of the at least one processor circuit is to:

determine ranks for the audiovisual streams based on the values; and

select the first one or more of the audiovisual streams based on the ranks.

5 . The apparatus of claim 4 , wherein the selected first one or more of the audiovisual streams includes:

a first group of the audiovisual streams having highest ranks among the audiovisual streams; and

a second group of the audiovisual streams having lowest ranks among the audiovisual streams.

6 . The apparatus of claim 4 , wherein one or more of the at least one processor circuit is to determine the ranks for the audiovisual streams based on rating data from the first client devices, the rating data associated with a presentation from the second client device.

7 . The apparatus of claim 1 , wherein a first audiovisual stream of the selected first one or more audiovisual streams is from a first one of the first client devices, and one or more of the at least one processor circuit is to cause transmission of an instruction to the first one of the first client devices, the instruction to cause an adjustment to a sensor of the first one of the first client devices, the sensor to obtain first eye tracking data associated with the first one of the audiovisual streams.

8 . At least one memory comprising computer readable instruction to cause at least one processor circuit to at least:

obtain a plurality of audiovisual streams from a plurality of client devices, the plurality of audiovisual streams including first audiovisual stream from a first client device and a second audiovisual stream from a second client device;

determine respective values for corresponding ones of the audiovisual streams based on at least one of (i) first facial expression data associated with the first audiovisual stream, (ii) first eye tracking data associated with the first audiovisual stream, (iii) second facial expression data associated with the second audiovisual stream, or (iv) second eye tracking data associated with the second audiovisual stream;

select, based on the values, a first one or more of the audiovisual streams for transmission;

cause transmission of the first one or more of the audiovisual streams to a third client device, unselected ones of the audiovisual streams to not be transmitted to the third client device;

update the respective values for the corresponding ones of the audiovisual streams based on at least one of (i) updated first facial expression data associated with the first audiovisual stream, (ii) updated first eye tracking data associated with the first audiovisual stream, (iii) updated second facial expression data associated with the second audiovisual stream, or (iv) updated second eye tracking data associated with the second audiovisual stream; and

select, based on the updated values, a second one or more of the audiovisual streams to transmit to the third client device, at least one of the second one or more of the audiovisual streams different from the first one or more of the audiovisual streams.

9 . The at least one memory of claim 8 , wherein the instructions are to cause one or more of the at least one processor circuit to select the second one or more of the audiovisual streams based on feedback from the third client device.

10 . The at least one memory of claim 9 , wherein the feedback is based on presentation of the selected first one or more of the audiovisual streams by the third client device.

11 . The at least one memory of claim 8 , wherein the instructions are to cause one or more of the at least one processor circuit to select the first one or more of the audiovisual streams based on first rating data from the first client device and second rating data from the second client device, the first rating data and the second rating data associated with a presentation from the third client device.

12 . The at least one memory of claim 8 , wherein the instructions are to cause one or more of the at least one processor circuit to cause transmission of a third audiovisual stream to the first client device and to the second client device, the third audiovisual stream from the third client device.

13 . The at least one memory of claim 8 , wherein the first client device, the second client device and the third client device are in a video conference.

14 . The at least one memory of claim 8 , wherein the instructions are to cause one or more of the at least one processor circuit to cause transmission of an instruction to the first client device, the instruction to cause an adjustment to at least one of a focus or a zoom of a sensor of the first client device, the sensor to obtain the first eye tracking data.

15 . A method comprising:

obtaining, by at least one processor circuit programmed based on at least one instruction, user data associated with a plurality of audiovisual streams, the audiovisual streams from first client devices, the user data including at least one of facial expression data or eye tracking data associated with the audiovisual streams;

determining, by one or more of the at least one processor circuit, respective values for corresponding ones of the audiovisual streams based on the at least one of the facial expression data or eye tracking data associated with the audiovisual streams;

selecting, based on the values, a first one or more of the audiovisual streams to transmit to a second client device;

transmitting the first one or more of the audiovisual streams to the second client device, unselected ones of the audiovisual streams not to be transmitted to the second client device;

updating, by one or more of the at least one processor circuit, the respective values for the corresponding ones of the audiovisual streams based on at least one of updated facial expression data or updated eye tracking data associated with the audiovisual streams; and

selecting, based on the updated values, a second one or more of the audiovisual streams to transmit to the second client device, at least one of the second one or more of the audiovisual streams different from the first one or more of the audiovisual streams.

16 . The method of claim 15 , including determining ranks for the plurality of audiovisual streams based on the updated values, and wherein the selected second one or more of the audiovisual streams includes:

a first group of the audiovisual streams having highest ranks among the audiovisual streams; and

a second group of having lowest ranks among the audiovisual streams.

17 . The method of claim 16 , wherein the determining of the ranks for the plurality of audiovisual streams is based on feedback from the second client device.

18 . The method of claim 17 , wherein the feedback is based on eye tracking data associated with presentation of the selected first one or more of the audiovisual streams by the second client device.

19 . The method of claim 16 , wherein the determining of the ranks for the plurality of audiovisual streams is based on rating data from the first client devices, the rating data associated with a presentation from the second client device.

20 . The method of claim 15 , wherein a first one of the audiovisual streams is from a first one of the first client devices, and including transmitting an instruction to the first one of the first client devices, the instruction to cause an adjustment to at least one of a focus or a zoom of a sensor of the first one of the first client devices, the sensor to obtain first eye tracking data associated with the first one of the audiovisual streams.

Continuity (6)
Continuation 17850457 · Jun 27, 2022
Continuation 17222713 · Apr 5, 2021
Continuation 16550569 · Aug 26, 2019
Continuation 15197937 · Jun 30, 2016
Continuation 14317607 · Jun 27, 2014
Related Publication 20240171622A1 · May 23, 2024
References Cited (83)
US 6356303B1 · Shibata · 2002 [cited by applicant]
US 8670018B2 · Cunnington · 2014 [cited by applicant]
US 8866943B2 · Park · 2014 [cited by applicant]
US 8873851B2 · Fedorovskaya · 2014 [cited by applicant]
US 9131053B1 · Tan · 2015 [cited by applicant]
US 9386272B2 · Louchheim · 2016 [cited by applicant]
US 9491374B1 · Avrahami · 2016 [cited by applicant]
US 10440071B2 · Louchheim · 2019 [cited by applicant]
US 10972518B2 · Louchheim · 2021 [cited by applicant]
US 11374991B2 · Louchheim et al. · 2022 [cited by applicant]
US 20020101505A1 · Gutta · 2002 [cited by applicant]
US 20040230651A1 · Ivashin · 2004 [cited by applicant]
US 20050231588A1 · Yang · 2005 [cited by examiner]
US 20060287070A1 · Walker · 2006 [cited by applicant]
US 20070271580A1 · Tischer · 2007 [cited by examiner]
US 20070282948A1 · Praino · 2007 [cited by examiner]
US 20080091512A1 · Marci · 2008 [cited by applicant]
US 20090146775A1 · Bonnaud · 2009 [cited by examiner]
US 20100207874A1 · Yuxin · 2010 [cited by applicant]
US 20110052155A1 · Desmarais · 2011 [cited by applicant]
US 20110072037A1 · Lotzer · 2011 [cited by applicant]
US 20110205330A1 · Liu · 2011 [cited by applicant]
US 20110292162A1 · Byun · 2011 [cited by examiner]
US 20110295392A1 · Cunnington · 2011 [cited by applicant]
US 20120109714A1 · Azar · 2012 [cited by applicant]
US 20120122476A1 · Lee · 2012 [cited by applicant]
US 20120151344A1 · Humphrey · 2012 [cited by applicant]
US 20120194631A1 · Venolia · 2012 [cited by applicant]
US 20120221960A1 · Robinson · 2012 [cited by applicant]
US 20120290950A1 · Rapaport · 2012 [cited by applicant]
US 20120296914A1 · Romanov · 2012 [cited by applicant]
US 20120306993A1 · Sellers-Blais · 2012 [cited by applicant]
US 20130046738A1 · Kuo · 2013 [cited by applicant]
US 20130162900A1 · Suzuki · 2013 [cited by applicant]
US 20130198629A1 · Tandon · 2013 [cited by applicant]
US 20130205311A1 · Ramaswamy · 2013 [cited by applicant]
US 20130268954A1 · Hulten · 2013 [cited by applicant]
US 20130290434A1 · Bank · 2013 [cited by examiner]
US 20130330055A1 · Zimmermann · 2013 [cited by applicant]
US 20140003716A1 · Fedorovskaya · 2014 [cited by applicant]
US 20140091897A1 · Lemmey · 2014 [cited by applicant]
US 20140106318A1 · Wright · 2014 [cited by applicant]
US 20150154445A1 · Biswas · 2015 [cited by applicant]
US 20150381932A1 · Louchheim · 2015 [cited by applicant]
US 20160381095A1 · Louchheim · 2016 [cited by applicant]
US 20200059499A1 · Louchheim · 2020 [cited by applicant]
US 20230051931A1 · Louchheim et al. · 2023 [cited by applicant]
CN 1551632A · 2004 [cited by applicant]
CN 102170552A · 2011 [cited by applicant]
CN 103548339A · 2014 [cited by applicant]
JP 2003304337A · 2003 [cited by applicant]
JP 2006338493A · 2006 [cited by applicant]
JP 2010141843A · 2010 [cited by applicant]
JP 4585380B2 · 2010 [cited by applicant]
JP 2011228817A · 2011 [cited by applicant]
JP 2014050018A · 2014 [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 14/317,607, mailed on Jul. 15, 2015, 6 pages. [cited by applicant]
International Searching Authority, “Written Opinion,” issued in connection with International Patent Application No. PCT/US2015/032567, mailed on Sep. 7, 2015, 10 pages. [cited by applicant]
International Searching Authority, “International Search Report,” issued in connection with International Patent Application No. PCT/US2015/032567, on Sep. 7, 2015, 3 pages. [cited by applicant]
United States Patent and Trademark Office, “Final Office Action,” issued in connection with U.S. Appl. No. 14/317,607, mailed on Oct. 9, 2015, 7 pages. [cited by applicant]
United States Patent and Trademark Office, “Advisory Action,” issued in connection with U.S. Appl. No. 14/317,607, mailed on Jan. 25, 2016, 1 page. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 14/317,607, mailed on Feb. 24, 2016, 7 pages. [cited by applicant]
Korean Intellectual Property Office, “Notice of Preliminary Rejection,” issued in connection with Korean Patent Application No. 2016-7032912, dated May 10, 2017, 11 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 15/197,937, mailed on Sep. 26, 2017, 10 pages. [cited by applicant]
European Patent Office, “Extended European Search Report,” issued in connection with European Patent Application No. 15811325.8, dated Feb. 8, 2018, 12 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 15/197,937, mailed on Mar. 8, 2018, 10 pages. [cited by applicant]
United States Patent and Trademark Office, “Final Office Action,” issued in connection with U.S. Appl. No. 15/197,937, mailed on Jul. 10, 2018, 6 pages. [cited by applicant]
United States Patent and Trademark Office, “Advisory Action,” issued in connection with U.S. Appl. No. 15/197,937, mailed on Oct. 25, 2018, 2 pages. [cited by applicant]
China National Intellectual Property Administration, “First Office Action,” issued in connection with Chinese Patent Application No. 201580027953.X, dated Nov. 1, 2018, 34 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 15/197,937, mailed on Feb. 14, 2019, 9 pages. [cited by applicant]
China National Intellectual Property Administration, “Second Office Action,” issued in connection with Chinese Patent Application No. 201580027953.X, dated May 17, 2019, 31 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 15/197,937, mailed on May 24, 2019, 6 pages. [cited by applicant]
China National Intellectual Property Administration, “Third Office Action,” issued in connection with Chinese Patent Application No. 201580027953.X, dated Aug. 30, 2019, 11 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 16/550,569, mailed on Jan. 16, 2020, 17 pages. [cited by applicant]
United States Patent and Trademark Office, “Final Office Action” issued in connection with U.S. Appl. No. 16/550,569, mailed on Mar. 6, 2020, 14 pages. [cited by applicant]
United States Patent and Trademark Office, “Advisory Action,” issued in connection with U.S. Appl. No. 16/550,569, mailed on May 14, 2020, 3 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 16/550,569, mailed on Jun. 1, 2020, 10 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 16/550,569, mailed on Aug. 11, 2020, 15 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 16/550,569, mailed on Dec. 2, 2020, 12 pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 17/222,713, mailed on Feb. 14, 2022, 9 pages. [cited by applicant]
United States Patent and Trademark Office, “Corrected Notice of Allowability,” issued in connection with U.S. Appl. No. 17/222,713, mailed on May 18, 2022, 2 pages. [cited by applicant]
United States Patent and Trademark Office, “Non-Final Office Action,” issued in connection with U.S. Appl. No. 17/850,457, mailed on May 9, 2023, 20 Pages. [cited by applicant]
United States Patent and Trademark Office, “Notice of Allowance and Fee(s) Due,” issued in connection with U.S. Appl. No. 17/850,457, mailed on Aug. 22, 2023, 8 Pages. [cited by applicant]