IP Library Granted Patent US 11,849,258
Granted Patent B2
US 11,849,258 · App. 17/545,318 · Granted Dec 19, 2023

Systems and methods for offloading processing-intensive video conferencing task to edge computing device

Inventor: Serhad Doken (Bryn Mawr, PA)
Assignee: Rovi Guides, Inc.
H04N7/157G06F11/3013G06F11/3041H04N7/142
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,849,258
App. No.
17/545,318
Filed
Dec 8, 2021
Granted
Dec 19, 2023
Kind
B2
Art Unit
2691
USPC
348/14.08
Abstract

Systems and methods are described for determining that a conferencing device participating in a video conference via a conferencing server is a mobile device. The mobile device may be performing virtual background processing to generate for display a virtual background during the video conference. Based on a monitored parameter of the mobile device, a determination may be made whether the virtual background processing being performed by the mobile device should be offloaded to an edge computing device. In response to determining the virtual background processing should be offloaded, at least a portion of the virtual background processing being performed by the mobile device may be offloaded to the edge computing device, and the edge computing device may be configured to receive a video feed from the conferencing device, modify the video feed by performing the virtual background processing, and transmit the modified video feed to the conferencing server.

Claims (71)

1. A method comprising:

determining that a conferencing device participating in a video conference via a conferencing server is a mobile device, wherein the mobile device is performing virtual background processing to generate for display a virtual background during the video conference;

monitoring a parameter of the mobile device during the video conference;

determining, based on the monitored parameter, whether the virtual background processing being performed by the mobile device should be offloaded to an edge computing device;

in response to determining the virtual background processing should be offloaded, offloading at least a portion of the virtual background processing being performed by the mobile device to the edge computing device, wherein the edge computing device is configured to receive a video feed from the conferencing device, modify the video feed by performing the virtual background processing, and transmit the modified video feed to the conferencing server;

determining, subsequent to offloading at least a portion of the virtual background processing being performed by the mobile device to the edge computing device, and based on the monitored parameter, that the mobile device should resume performance of the virtual background processing;

transmitting an indication to the edge computing device to cease performing the virtual background processing; and

transmitting an indication to the mobile device to resume performing the virtual background processing.

2. The method of claim 1 , further comprising:

in response to determining the virtual background processing should be offloaded:

causing the conferencing device to transmit a partial video feed, instead of transmitting a full video feed;

wherein the partial video feed comprises a depiction of a human figure.

3. The method of claim 2 , wherein the partial video feed is a polygonal portion of the video feed defined by identifying a polygon that comprises the depiction of the human figure.

4. The method of claim 1 , wherein:

monitoring the parameter of the mobile device during the video conference comprises:

determining a current battery level of the mobile device; and

determining whether the current battery level exceeds a threshold battery level; and

determining, based on the monitored parameter, whether the virtual background processing being performed by the mobile device should be offloaded to an edge computing device is performed based on whether the current battery level exceeds the threshold battery level.

5. The method of claim 1 , wherein:

monitoring the parameter of the mobile device during the video conference comprises:

determining a current battery level of the mobile device;

determining an amount of time remaining in the video conference; and

determining a rate of battery consumption based on video conferencing processing tasks being performing by the mobile device; and

determining, based on the monitored parameter, whether the virtual background processing being performed by the mobile device should be offloaded to an edge computing device comprises determining whether the battery of the mobile device will fall below a threshold level of charge before the conclusion of the video conference if virtual background processing continues to be performed by the mobile device.

6. The method of claim 1 ; wherein determining that the mobile device should resume performance of the virtual background processing is performed based on a current battery level of the mobile device, an amount of time remaining in the video conference, and a rate of battery consumption.

7. The method of claim 1 , wherein:

an additional conferencing device is participating in the video conference; and

the edge computing device is configured to:

receive, from the additional conferencing device, a video feed intended for the mobile device;

process the video feed intended for the mobile device; and

transmit the processed video feed to the mobile device.

8. The method of claim 1 , wherein:

monitoring the parameter of the mobile device during the video conference comprises determining, based on sensor data received from the mobile device, that a video frame captured by the mobile device during the video conference is unstable, wherein the sensor data is indicative of movement of the mobile device; and

offloading at least a portion of the virtual background processing being performed by the mobile device to the edge computing device comprises causing the unstable video frame to be transmitted to the edge computing device, wherein the edge computing device stabilizes the video frame based at least in part on the sensor data.

9. The method of claim 1 , wherein the mobile device is a smartphone, and the edge computing device is a mobile edge computing device.

10. A system comprising: memory; control circuitry configured to:

determine that a conferencing device participating in a video conference via a conferencing server is a mobile device, wherein the mobile device is performing virtual background processing to generate for display a virtual background during the video conference;

monitor a parameter of the mobile device during the video conference, wherein the parameter is stored in the memory;

determine, based on the monitored parameter, whether the virtual background processing being performed by the mobile device should be offloaded to an edge computing device;

in response to determining the virtual background processing should be offloaded, offload at least a portion of the virtual background processing being performed by the mobile device to the edge computing device, wherein the edge computing device is configured to receive a video feed from the conferencing device, modify the video feed by performing the virtual background processing, and transmit the modified video feed to the conferencing server;

determine, subsequent to offloading at least a portion of the virtual background processing being performed by the mobile device to the edge computing device, and based on the monitored parameter, that the mobile device should resume performance of the virtual background processing;

transmit an indication to the edge computing device to cease performing the virtual background processing; and

transmit an indication to the mobile device to resume performing the virtual background processing.

11. The system of claim 10 , wherein the control circuitry is further configured to:

in response to determining the virtual background processing should be offloaded:

cause the conferencing device to transmit a partial video feed, instead of transmitting a full video feed;

wherein the partial video feed comprises a depiction of a human figure.

12. The system of claim 11 :

wherein the partial video feed is a polygonal portion of the video feed defined by identifying a polygon that comprises the depiction of the human figure.

13. The system of claim 10 , wherein the control circuitry is further configured to:

monitor the parameter of the mobile device during the video conference by:

determining a current battery level of the mobile device; and

determining whether the current battery level exceeds a threshold battery level; and

determine, based on the monitored parameter, whether the virtual background processing being performed by the mobile device should be offloaded to an edge computing device based on whether the current battery level exceeds the threshold battery level.

14. The system of claim 10 , wherein the control circuitry is further configured to:

monitor the parameter of the mobile device during the video conference by:

determining a current battery level of the mobile device;

determining an amount of time remaining in the video conference; and

determining a rate of battery consumption based on video conferencing processing tasks being performing by the mobile device; and

determine, based on the monitored parameter, whether the virtual background processing being performed by the mobile device should be offloaded to an edge computing device by determining whether the battery of the mobile device will fall below a threshold level of charge before the conclusion of the video conference if virtual background processing continues to be performed by the mobile device.

15. The system of claim 10 , wherein the control circuitry is configured to determine that the mobile device should resume performance of the virtual background processing is performed based on a current battery level of the mobile device, an amount of time remaining in the video conference, and a rate of battery consumption.

16. The system of claim 10 , wherein:

an additional conferencing device is participating in the video conference; and

the edge computing device is configured to:

receive, from the additional conferencing device, a video feed intended for the mobile device;

process the video feed intended for the mobile device; and

transmit the processed video feed to the mobile device.

17. The system of claim 10 , wherein the control circuitry is further configured to:

monitor the parameter of the mobile device during the video conference by determining, based on sensor data received from the mobile device, that a video frame captured by the mobile device during the video conference is unstable, wherein the sensor data is indicative of movement of the mobile device; and

offload at least a portion of the virtual background processing being performed by the mobile device to the edge computing device by causing the unstable video frame to be transmitted to the edge computing device, wherein the edge computing device stabilizes the video frame based on the sensor data.

18. The system of claim 10 , wherein the mobile device is a smartphone, and the edge computing device is a mobile edge computing device.