System and method for video performance optimizations during a video conference session
An information handling system executing a multimedia multi-user collaboration application (MMCA) may include a memory; a power management unit; a video camera to capture video of a user participating in a video conference session conducted by a processor executing code instructions of the MMCA via a first network link on a network interface device; a time-of-flight (TOF) sensor to provide distance data descriptive of the distance between a user's body and the video camera; the processor configured to execute a video frame user presence detection system to: slice a video frame into a plurality of video frame slices; based on the distance data, create a human object presence (HOP) heat map to assign a probability of the presence of the user's body within each video frame slice of the video frame; the processor configured to select, among the plurality of video frame slices, those video frame slices with an assigned probability of presence of the user's body that exceeds a threshold probability as a HOP video frame slice group where the user's body appears within the video frame; and the network interface device configured to send data defining the HOP video frame slice group to a sink information handling system participating in the video conference session over the network to assist decoding of the video frame at the sink information handling system.
1. An information handling system executing a multimedia multi-user collaboration application (MMCA), comprising:
a memory;
a power management unit;
a video camera to capture video of a user participating in a video conference session conducted by a processor executing code instructions of the MMCA via a first network link on a network interface device;
a time-of-flight (TOF) sensor to provide distance data descriptive of the distance between a user's body and the video camera;
the processor configured to execute a video frame user presence detection system to:
slice a video frame into a plurality of video frame slices;
based on the distance data, create a human object presence (HOP) heat map to assign a probability of the presence of the user's body within each video frame slice of the video frame;
the processor configured to select, among the plurality of video frame slices, those video frame slices with an assigned probability of presence of the user's body that exceeds a threshold probability as a HOP video frame slice group where the user's body appears within the video frame; and
the network interface device configured to send data defining the HOP video frame slice group to a sink information handling system participating in the video conference session over the network to assist decoding of the video frame at the sink information handling system.
2. The information handling system of claim 1 further comprising:
the user presence detection system providing data descriptive of a delta HOP heat map descriptive of changes in position of the human image within the plurality of video frame slices over a plurality of subsequent video frames captured by the video camera; and
the MMCA to send the data of the delta HOP heat map to the sink information handling system for decoding of the subsequent video frame.
3. The information handling system of claim 1 further comprising:
the MMCA to detect the actuation of a virtual background; and
data descriptive of the video frame slices where the user's body is found to execute a matting algorithm used to reduce a frame area of computation necessary to detect the outline of the user for use with the virtual background application.
4. The information handling system of claim 1 further comprising:
the processor to execute a first encoding algorithm to encode the data within the HOP video frame slice group for transmission by the MMCA to the sink information handling system;
the processor to execute a second encoding algorithm having a lower computational burden than the first encoding algorithm to encode data outside of the HOP video frame slice group for transmission by the MMCA to the sink information handling system;
the MMCA sending data to the sink information handling system to define and use the first encoding algorithms to use on the video frame slices in the HOP video frame slice group and the second encoding algorithm designation for the video frame slices outside of the HOP video frame slice group.
5. The information handling system of claim 1 further comprising:
the processor to execute an AV processing instruction manager to offload execution of the user presence detection system to an alternative processor in the information handling system.
6. The information handling system of claim 1 , wherein the network interface device sends data defining the HOP video frame slice group to a sink information handling system for decoding of the video frame using an out-of-band (OOB) communication network link.
7. The information handling system of claim 1 further comprising:
the processor configured to execute an unusual motion detection processing instruction module to calculate a delta HOP heat map descriptive of changes in position of the human image within the plurality of subsequent video frame slices over a plurality of video frames captured by the video camera after a first captured video frame by comparing historic HOP heat maps with a currently created HOP heat map.
8. The information handling system of claim 1 further comprising:
the processor to execute a decoding processing instruction module to receive, from a remote information handling system, instructions describing how to decode a received video frame from the remote information handling system based on a HOP heat map produced at the remote information handling system.
9. A method of reducing resource consumption at an information handling system during a video conferencing session, comprising:
receiving, from a time-of-flight (TOF) sensor, distance data descriptive of the distance between a user's body and the video camera of the information handling system;
executing a multimedia multi-user collaboration application (MMCA) to establish the video conferencing session via a network interface device;
receiving, from the video camera, a video frame;
with a processor of the information handling system, executing a user presence detection system for:
slicing the video frame into a plurality of video frame slices;
based on the distance data, creating a human object presence (HOP) heat map assigning a probability of the presence of the user's body within each video frame slice of the video frame;
selecting, among the plurality of video frame slices, those video frame slices with an assigned probability of presence of the user's body appearing in the video frame slices that exceeds a threshold and probabilities; and
creating a HOP video frame slice group where the user's body is found within the video frame;
with a network interface device, sending data defining the HOP video frame slice group to a sink information handling system over the network for decoding of the video frame; and
with the network interface device, the MMCA to transmit encoded video frames for the video conference session.
10. The method of claim 9 further comprising:
executing the user presence detection system to provide data descriptive of a delta HOP heat map describing changes in position of the human image within the plurality of subsequent video frame slices over a plurality of subsequent video frames captured by the video camera after a first video frame; and
the MMCA to send the data of the delta HOP heat map to the sink information handling system over the network for decoding of the video frame.
11. The method of claim 9 further comprising:
with the execution of the MMCA, detecting the actuation of a virtual background and, with the data describing the HOP video frame slice group, providing the data descriptive of the video frame slices where the user's body is found to a processor executing a matting algorithm of an outline detection code used to reduce an area of computation to detect the outline of the user.
12. The method of claim 9 further comprising:
executing a first encoding algorithm to encode the data within the HOP video frame slice group for transmission by the MMCA to the sink information handling system;
executing a second encoding algorithm having lower computational requirements than the first encoding algorithm to encode data outside of the HOP video frame slice group for transmission by the MMCA to the sink information handling system; and
sending data to the sink information handling system defining which of the first and second encoding algorithms to use on the video frame slices of the HOP video frame slice group and the video frame slices outside of the HOP video frame slice group.
13. The method of claim 9 further comprising:
sending data defining the HOP video frame slice group to a sink information handling system over the network for decoding of the video frame using an out-of-band (OOB) communication on a network link separate from the encoded video frames.
14. The method of claim 9 further comprising:
executing an unusual motion detection processing instruction module to calculate a delta HOP heat map descriptive of changes in position of the human image within the plurality of video frame slices over a plurality of video frames captured by the video camera and comparing historic HOP heat maps with a currently created HOP heat map.
15. An information handling system executing a multimedia multi-user collaboration application (MMCA), comprising:
a memory;
a power management unit;
a processor executing code instruction of the MMCA to establish the video conference session via a network interface device;
a video camera to capture video of a user participating in a video conference session;
a time-of-flight (TOF) sensor to provide distance data descriptive of the distance between a user's body and the video camera;
the processor configured to execute a user presence detection system to:
slice a video frame into a plurality of video frame slices; and
based on the distance data, create a human object presence (HOP) heat map assigning a probability of the presence of the user's body appears within each video frame slice of the video frame; and
select, among the plurality of video frame slices, those video frame slices with an assigned probability of presence of the user's body that exceeds a threshold probability and create a HOP video frame slice group in which the user's body is found within the video frame;
the network interface device to send data defining the HOP video frame slice group to a sink information handling system participating in the video conference session over the network for decoding of the video frame; and
an unusual motion detection system configured to generate data descriptive of a delta HOP heat map descriptive of changes in position of the human image within the plurality of video frame slices over a plurality of video frames captured by the video camera and to send the data of the delta HOP heat map to the sink information handling system over the network for decoding of the streamed video frames.
16. The information handling system of claim 15 further comprising:
the processor executing the MMCA to detect the actuation of a virtual background; and
the processor executing a matting algorithm of a boundary detection processing instruction module with the data describing the HOP video frame slice group to reduce computation area of the video frame necessary to detect the outline of the user.
17. The information handling system of claim 15 further comprising:
the processor to execute a first encoding algorithm to encode the data within the HOP video frame slice group for transmission by the MMCA to the sink information handling system;
the processor to execute a second encoding algorithm having a lower computational burden than the first encoding algorithm to encode data outside of the HOP video frame slice group for transmission by the MMCA to the sink information handling system; and
the MMCA sending data to the sink information handling system to use the first encoding algorithm on the video frame slices of the HOP video frame slice group and to use the second encoding algorithm on the video frame slices outside of the HOP video frame slice group.
18. The information handling system of claim 15 further comprising:
the network interface device to send data defining the HOP video frame slice group to a sink information handling system over the network for decoding of the video frame using an out-of-band (OOB) communication, the OOB communication sending the data defining the HOP video frame slice group on a different network link than used to send the encoded video frame.
19. The information handling system of claim 15 further comprising:
the processor to execute an AV processing instruction manager to offload execution of the user presence detection system to an alternative processor in the information handling system.
20. The information handling system of claim 15 further comprising:
a decoding processing instruction module to receive, from a remote information handling system, instructions describing how to decode a received video frame based on a HOP heat map produced at the remote information handling system.