IP Library Granted Patent US 10,375,354
Granted Patent B2
US 10,375,354 · App. 15/863,713 · Granted Aug 6, 2019

Video communication using subtractive filtering

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,375,354
App. No.
15/863,713
Granted
Aug 6, 2019
Kind
B2
Abstract

In one embodiment, a method includes receiving a request to establish an audiovisual streaming session between a first device and a second device; determining a level of network connectivity for a network associated with the first device or the second device; receiving data from the first device. The data may the data represent facial expressions of a user in a video captured by the first device. The data may be generated by the first device using the video based on: detections of a face and facial features of the user in the video and application of one or more subtractive filters using the detected face and facial features. The subtractive filters may be selected based on the level of network connectivity. The method may also include sending the data to the second device with instructions to process the data and display a representation of the user's facial expressions.

Claims (40)

1. A method comprising, by one or more computing devices:

receiving a request to establish a live audiovisual streaming session between a first device and a second device;

determining a level of network connectivity for a network associated with the first device or the second device;

receiving data generated by the first device, wherein the data represents facial expressions of a user in a video captured by the first device, wherein the data is generated by the first device using the video based on:

a detection of a face or a facial feature of the user in the video; and

an application of one or more subtractive filters on the video using the detected face or facial feature, wherein the one or more subtractive filters are selected based on the level of network connectivity, wherein the data, relative to the video, is reduced in size based on the application of the one or more subtractive filters; and

sending the data to the second device with instructions to process the data and display a representation of the user's facial expressions on a display screen associated with the second device.

2. The method of claim 1 , wherein one of the subtractive filters is a facial-landmark filter, wherein the facial-landmark filter removes substantially all data from the video except for a set of facial landmarks for the face, and wherein the facial-landmark filter is applied based on a determination that the level of network connectivity is below a threshold level.

3. The method of claim 1 , wherein the representation of the user's facial expressions comprises one or more animated features that have been added to the data.

4. The method of claim 1 , wherein one of the subtractive filters is a dominant-feature filter, wherein the dominant-feature filter removes substantially all video data from the original data stream except for one or more dominant features of each of the isolated faces, and wherein the dominant-feature filter is applied based on a determination that the level of network connectivity is moderately poor.

5. The method of claim 4 , wherein the dominant features comprise eyes and a mouth of the first user.

6. The method of claim 1 , wherein the representation of the user's facial expressions comprises one or more animated features that have been added to the data, and wherein the animated features are generated at least in part based on a machine-learning model that takes as input the data received from the first device and audio data and outputs the representation of the user's facial expressions.

7. The method of claim 6 , wherein the machine-learning model is trained with an audiovisual recording of the user.

8. The method of claim 6 , wherein the machine-learning model is trained with one or more images or audiovisual recordings stored in association with a social-networking system.

9. The method of claim 1 , wherein the detection of the face or the facial feature is accomplished using Haar Cascading.

10. One or more computer-readable non-transitory storage media embodying software that is operable when executed to:

receive a request to establish a live audiovisual streaming session between a first device and a second device;

determine a level of network connectivity for a network associated with the first device or the second device;

receive data generated by the first device, wherein the data represents facial expressions of a user in a video captured by the first device, wherein the data is generated by the first device using the video based on:

a detection of a face or a facial feature of the user in the video; and

an application of one or more subtractive filters on the video using the detected face or facial feature, wherein the one or more subtractive filters are selected based on the level of network connectivity, wherein the data, relative to the video, is reduced in size based on the application of the one or more subtractive filters; and

send the data to the second device with instructions to process the data and display a representation of the user's facial expressions on a display screen associated with the second device.

11. The media of claim 10 , wherein one of the subtractive filters is a facial-landmark filter, wherein the facial-landmark filter removes substantially all data from the video except for a set of facial landmarks for the face, and wherein the facial-landmark filter is applied based on a determination that the level of network connectivity is below a threshold level.

12. The media of claim 10 , wherein the representation of the user's facial expressions comprises one or more animated features that have been added to the data.

13. The media of claim 10 , wherein one of the subtractive filters is a dominant-feature filter, wherein the dominant-feature filter removes substantially all video data from the original data stream except for one or more dominant features of each of the isolated faces, and wherein the dominant-feature filter is applied based on a determination that the level of network connectivity is moderately poor.

14. The media of claim 13 , wherein the dominant features comprise eyes and a mouth of the first user.

15. The media of claim 10 , wherein the representation of the user's facial expressions comprises one or more animated features that have been added to the data, and wherein the animated features are generated at least in part based on a machine-learning model that takes as input the data received from the first device and audio data and outputs the representation of the user's facial expressions.

16. A system comprising:

one or more processors; and

one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:

receive a request to establish a live audiovisual streaming session between a first device and a second device;

determine a level of network connectivity for a network associated with the first device or the second device;

receive data generated by the first device, wherein the data represents facial expressions of a user in a video captured by the first device, wherein the data is generated by the first device using the video based on:

a detection of a face or a facial feature of the user in the video; and

an application of one or more subtractive filters on the video using the detected face or facial feature, wherein the one or more subtractive filters are selected based on the level of network connectivity, wherein the data, relative to the video, is reduced in size based on the application of the one or more subtractive filters; and

send the data to the second device with instructions to process the data and display a representation of the user's facial expressions on a display screen associated with the second device.

17. The system of claim 16 , wherein one of the subtractive filters is a facial-landmark filter, wherein the facial-landmark filter removes substantially all data from the video except for a set of facial landmarks for the face, and wherein the facial-landmark filter is applied based on a determination that the level of network connectivity is below a threshold level.

18. The system of claim 16 , wherein the representation of the user's facial expressions comprises one or more animated features that have been added to the data.

19. The system of claim 16 , wherein one of the subtractive filters is a dominant-feature filter, wherein the dominant-feature filter removes substantially all video data from the original data stream except for one or more dominant features of each of the isolated faces, and wherein the dominant-feature filter is applied based on a determination that the level of network connectivity is moderately poor.

20. The system of claim 19 , wherein the dominant features comprise eyes and a mouth of the first user.

Assignments (5)
CHANGE OF NAME Recorded Dec 20, 2021
From: FACEBOOK, INC.
To: META PLATFORMS, INC.
Reel/Frame 058553/0802 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 3, 2019
From: INGAWALE, MYSHKIN
To: FACEBOOK, INC.
Reel/Frame 048781/0140 →
CORRECTIVE ASSIGNMENT TO CORRECT THE THE EXECUTION DATE PREVIOUSLY RECORDED AT REEL: 048712 FRAME: 0661. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT . Recorded Apr 2, 2019
From: SANKHE, PRITESH
To: FACEBOOK, INC.
Reel/Frame 049259/0262 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 27, 2019
From: SANKHE, PRITESH
To: FACEBOOK, INC.
Reel/Frame 048712/0661 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Feb 2, 2018
From: SATHYA, SAI SRI
To: FACEBOOK, INC.
Reel/Frame 044815/0126 →